SportSense - Sketch-based spatio-temporal Query Support for Sports Videos (Ongoing)

Game analysis is an important task in sports, especially in team sports. The objective is to analyze the behavior (strengths, weaknesses) of other teams. Currently, this is mainly a manual task, done by coaches or dedicated game analysts based on videos of the teams to be analyzed.

The goal of the SportSense project is to apply novel sketch-based and content-based approaches to video retrieval to support the tasks of coaches and game analysts. The objective is to facilitate the search for scenes in sport videos. In most cases, queries are formulated on the basis of specific motion characteristics the coach is interested in or remembers from previous views of the video. Providing sketching interfaces for graphically specifying query input is thus a very natural user interaction for a retrieval application. However, the quality of the query (the sketch) heavily depends on the memory of the user and her ability to accurately formulate the intended search query by transforming this 3D memory of the known item(s) into 2D sketch query. Therefore, appropriate user interfaces have to be provided that are easy to use and that allow for the specification of rough sketches. The retrieval back-end then needs to execute these queries by allowing a certain degree of tolerance between the query sketch and the actual motion.

So far, SportSense automatically links event data (such as those procuded by StreamTeam) to a video of a game and provides coaches and game analysts the possibility to search within the video on the basis of sketches on the motion of players or the ball as well as on the basis of chains of events, both in backward and forward direction, by specifying spatio-temporal information. The latter is supported by user-friendly interfaces such as tablets, interactive paper, or novel gesture-based interactions.

As a next step, SportSense will be extended towards supporting i.) more sports disciplines, ii.) sketch-based queries for complex events without clear spatial reference, and iii.) jointly analyzing position data and physiological data of the individuals.

The SportSense project will continue focussing on team sports, and in particular on football, based on the existing collaboration between the DBIS group at UNIBAS and the BFH Centre for Technologies in Sports and Medicine. As the Swiss Federal Institute of Sport Magglingen (SFISM) is part of the BFH centre, the latter acts as interface between technology and especially computer science and sports. The BFH centre closely collaborates with the Swiss Football Association (SFV) and in particular with the coaches of the youth national teams of the SFV. In addition, the BFH centre has also a close collaboration with the International Ice Hockey Federation (IIHF). Therefore, ice hockey will be considered as a second use case for the evaluation of the project results.

A video showing the user interface of SportSense is published on Youtube:


The whole code of SportSense is published on GitHub under the GNU Affero General Public License v3.0.




BFH Centre for Technologies in Sports and Medicine (Martin Rumo)


Funded since September 2017 by the Hasler Foundation in the Cyber-Human Sytems program (contract no. 16074).


Research Topics