Towards Sketch-based Motion Queries in Sports Videos

Ihab Al Kabary and Heiko Schuldt

In Proceedings

2013/12

Proceedings of the 15th IEEE International Symposium on Multimedia (ISM 2013)

Anaheim, CA, USA

IEEE

The advent of pen-based user interfaces has facilitated several natural ways for human-computer interaction. One example is sketch-based retrieval, i.e., the search for (multimedia) objects on the basis of sketches as query input. So far, work has focused mainly on sketch-based image retrieval. However, more and more application domains also benefit from sketches as query input for searching in video collections. Enabling spatial search in videos, in the form of sketch-based motion queries, is increasingly demanded by coaches and analysts in team sports as a novel and innovative tool for game analysis. Even though game analysis is already a major activity in this domain, it is still mostly based on manual selection of video sequences. In this paper, we present SportSense, a first approach to enabling intuitive and efficient video retrieval using sketch-based motion queries. This is accomplished by using videos of games in team sports, together with an overlay of meta data that incorporates spatio-temporal information about various events. The creation of meta data significantly benefits from recent advances in tracking technologies that make use of light-weight wireless sensor devices and that are able to capture a more accurate and broader collection of data about the ball and the players without the need for manual tracking or placing specialized cameras within stadiums. SportSense exploits spatio-temporal databases to store, index, and retrieve the tracked information at interactive response times. Moreover, it provides first intuitive user input interfaces for sketches representing motion paths. A particular challenge is to convert the users' sketches into spatial queries and to execute these queries in a flexible way that allows for some controlled deviation between the sketched path and the actual movement of the players and/or the ball. The evaluation results of SportSense show that this approach to sketch-based retrieval in sports videos is both very effective and efficient.