IMOTION - Intelligent Multi-Modal Augmented Video Motion Retrieval System (Finished)
Video is increasingly gaining importance as medium to capture and disseminate information. This is not only the case for personal use but also (and most importantly) for professional and educational applications. With the enormous growth of video collections, effective yet efficient content-based retrieval of (parts of) videos is becoming more and more essential. Conventionally, video retrieval relies on metadata such as manual annotations, or inherent features extracted from the video. However, the most decisive information that distinguishes video content from static content, the movement of individual objects across subsequent frames, so far is largely ignored. This is particularly the case for so-called augmented video where additional spatio-temporal data on the movement of objects (e.g., captured by dedicated sensors systems) is available in addition to the actual video content. The IMOTION project will develop and evaluate innovative multi-modal user interfaces for interacting with augmented videos. Starting with an extension of existing query paradigms (keyword search in manual annotations), image search (query by example in key frames), IMOTION will consider novel sketch- and speech-based user interfaces. In particular, novel types of motion queries will be supported where users can specify motion paths of objects, via sketches, gestures, natural language interfaces, or combinations thereof. Several types of user interfaces (voice, tablets, multi-touch tables, interactive paper) will be supported and seamlessly combined so as to smoothly migrate a session from one type of user interface to another during the process of specifying and refining a query. This will be based on novel approaches to representation learning and the extraction of high-level motion descriptors from augmented videos, based on a motion ontology. In addition, IMOTION will develop novel index structures that jointly support traditional video features and the additional motion metadata.
A major contribution will be the quantitative and qualitative evaluation and user studies of the intelligent multi-modal interfaces and query paradigms developed in two concrete use cases – sample applications from which the project will select include, but are not limited to, augmented sports videos where users search on the basis of trajectories of player or ball movements, educational videos from the natural sciences where users search for animal movements inside a horde or a swarm, or sketch-based searches for currents in the sea captured by sensors integrated into buys. The IMOTION consortium will openly publish the augmented video collections and the motion metadata created in the course of the project's evaluation activities.
Awards
Demonstration
Parts of IMOTION have been released within the vitrivr project as open source projects and have participated to Google Summer of Code.
In the Media
- 03.12.2016: Innovationspreis für Pilz-Zucht auf Kaffeesatz (bz Basel) on the City-Stories project and its use of vitrivr
- 31.05.2016: Bild- und Videosuche per Handskizze (informatik.ch) on vitrivr
- 29.07.2016: Lauter bekannte Gesichter (Hannoversche Allgemeine) on vitrivr
- 06.07.2016: Suchmaschine für Skizzen (sem-seo.at) on vitrivr
- 20.06.2016: Ein Bild sucht mehr als tausend Worte (Deutschlandfunk) report and interview on IMOTION/vitrivr
- 03.06.2016: Online-Suche mittels Skizzen (20min) on IMOTION/vitrivr
- 31.05.2016: In a new method for searching image databases, a hand-drawn sketch is all it takes (phys.org) on vitrivr
- 04.11.2014: Videos via Suchanfrage durchsuchen (netzwoche) on IMOTION
- 05.09.2014: Big data tamed with the cloud (Microsoft Research Blog) on ADAM/ADAMpro
Start / End Dates
01.01.2014 - 31.05.2018
Partners
- Prof. Dr. Heiko Schuldt, Department of Mathematics and Computer Science, University of Basel (UNIBAS) - Coordinator
- Prof. T. Metin Sezgin, Dept. of Computer Engineering, Koç University, Turkey (UKoç)
- Dr. Stéphane Dupont, TCTS Lab, Université de Mons, Belgium (UMons)
Funding Agencies
IMOTION is a CHIST-ERA project, funded by
- Swiss National Science Foundation, SNF (for UNIBAS)
- TÜRKİYE BİLİMSEL ve TEKNOLOJİK ARAŞTIRMA KURUMU, TÜBITAK (for UKoç)
- Fonds de la Recherche Scientifique, FNRS (for UMons)
Funding
Total project funding: 770'065.- EUR Project funding UNIBAS: 558'934.- CHF Infrastructure support provided by Microsoft Azure (a blog post on the IMOTION project can be found here).
Project extension until May 31, 2018 (additional funds at UNIBAS: 126'092.- CHF).
Staff
Research Topics
Publications
2019
- Ralph Gasser, Luca Rossetto, Heiko Schuldt
Towards an All-Purpose Content-Based Multimedia Information Retrieval System
arXiv cs.MM, 2019/2 - Luca Rossetto, Heiko Schuldt, George Awad, Asad A. Butt
V3C — a Research Video Collection
Proceedings of the 25th International Conference on MultiMedia Modeling, Thessaloniki, Greece 2019/1
2018
- Luca Rossetto
Multi-Modal Video Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel 2018/9 - Luca Rossetto, Ivan Giangreco, Ralph Gasser, Heiko Schuldt
Competitive Video Retrieval with vitrivr at the Video Browser Showdown 2018 - Final Notes
arXiv cs.MM, 2018/5 - Ivan Giangreco
Database Support for Large-Scale Multimedia Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel 2018/3 - Luca Rossetto, Ivan Giangreco, Ralph Gasser, Heiko Schuldt
Competitive Video Retrieval with vitrivr
Proceedings of the International Conference on Multimedia Modeling (MMM), Bangkok, Thailand 2018/2 - Luca Rossetto, Heiko Schuldt
The Long Tail of Web Video
Proceedings of the 24th International Conference on Multimedia Modeling, Bangkok, Thailand 2018/2 - Stéphane Dupont, Ozan Can Altiok, Aysegül Bumin, Ceren Dikmen, Ivan Giangreco, Silvan Heller, Emre Külah, Gueorgui Pironkov, Luca Rossetto, Yusuf Sahillioglu, Heiko Schuldt, Omar Seddati, Yusuf Setinkaya, Metin Sezgin, Claudiu Tanase, Emre Toyan, Sean Wood, Doguhan Yeke
VideoSketcher: Innovative Query Modes for Searching Videos through Sketches, Motion and Sound
Proceedings of the eNTERFACE 2015 Workshop on Intelligent Interfaces, 2018/1
2017
- Luca Rossetto, Heiko Schuldt
Web Video in Numbers - An Analysis of Web-Video Metadata
arXiv cs.MM, 2017/7 - Luca Rossetto, Ivan Giangreco, Claudiu Tănase, Heiko Schuldt
Multimodal Video Retrieval with the 2017 IMOTION System
Proceedings of International Conference on Multimedia Retrieval (ICMR), Bucharest, Romania 2017/6 - Ivan Giangreco, Luca Rossetto, Claudiu Tănase, Heiko Schuldt
vitrivr: A Multimedia Search System supporting Multimodal Interactions
Proceedings of the International Conference on Multimodal Communication: Developing New Theories and Methods, Osnabrück, Germany 2017/6 - Prateek Goel, Ivan Giangreco, Luca Rossetto, Claudiu Tănase, Heiko Schuldt
Hey, vitrivr! - A Multimodal UI for Video Retrieval
Proceedings of the European Conference on Information Retrieval (ECIR), Aberdeen, Scotland, UK 2017/4 - Luca Rossetto, Ivan Giangreco, Claudiu Tănase, Heiko Schuldt, Stéphane Dupont, Omar Seddati
Enhanced Retrieval and Browsing in the IMOTION System
Proceedings of the International Conference on Multimedia Modeling (MMM), Reykjavik, Iceland 2017/1
2016
- Claudiu Tănase, Luca Rossetto, Ivan Giangreco, Heiko Schuldt, Stéphane Dupont, Omar Seddati
The IMOTION System at TRECVID 2016: The Ad-Hoc Video Search Task
Proceedings of the 2016 TRECVID Ad-Hoc Video Search Task, 2016/10 - Claudiu Tănase, Luca Rossetto, Ivan Giangreco, Heiko Schuldt
The vitrivr System at TRECVID 2016: The Ad-Hoc Video Search Task
Proceedings of the 2016 TRECVID Ad-Hoc Video Search Task, 2016/10 - Luca Rossetto, Ivan Giangreco, Claudiu Tănase, Heiko Schuldt
vitrivr - A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections
Proceedings of the International ACM Conference on Multimedia (ACM MM), Amsterdam, NL 2016/10 - Claudiu Cobârzan, Klaus Schoeffmann, Werner Bailer, Wolfgang Hürst, Adam Blažek, Jakub Lokoč, Stefanos Vrochidis, Kai Uwe Barthel, Luca Rossetto
Interactive video search tools: a detailed analysis of the video browser showdown 2015
Multimedia Tools and Applications, 2016/7 - Ivan Giangreco, Heiko Schuldt
ADAMpro: Database Support for Big Multimedia Retrieval
Datenbank-Spektrum, Schwerpunktthema “Big Data & IR”, 2016/3 - Claudiu Tănase, Ivan Giangreco, Luca Rossetto, Heiko Schuldt, Omar Seddati, Stéphane Dupont, Ozan Can Altiok, Metin Sezgin
Semantic Sketch-Based Video Retrieval with Autocompletion
Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI), Sonoma, CA, USA 2016/3 - Luca Rossetto, Claudiu Tănase, Heiko Schuldt
Dealing with ambiguous Queries in Multimodal Video Retrieval
Proceedings of the 22nd International Conference on Multimedia Modeling (MMM 2016), Miami, FL, USA 2016/1 - Luca Rossetto, Ivan Giangreco, Claudiu Tănase, Heiko Schuldt, Stéphane Dupont, Omar Seddati, Metin Sezgin, Yusuf Sahillioğlu
iAutoMotion - an Autonomous Content-based Video Retrieval Engine
Proceedings of the International Conference on Multimedia Modeling (MMM) - Video Browser Showdown, Miami, FL, USA 2016/1 - Luca Rossetto, Ivan Giangreco, Silvan Heller, Claudiu Tănase, Heiko Schuldt, Stéphane Dupont, Omar Seddati, Metin Sezgin, Ozan Can Altıok, Yusuf Sahillioğlu
IMOTION - Searching for Video Sequences using Multi-Shot Sketch Queries
Proceedings of the International Conference on Multimedia Modeling (MMM) - Video Browser Showdown, Miami, FL, USA 2016/1 - Luca Rossetto, Ivan Giangreco, Silvan Heller, Claudiu Tănase, Heiko Schuldt
Searching in Video Collections using Sketches and Sample Images - The Cineast System
Proceedings of the International Conference on Multimedia Modeling (MMM), Miami, FL, USA 2016/1
2015
- Luca Rossetto, Ivan Giangreco, Heiko Schuldt
OSVC - Open Short Video Collection 1.0
Technical Report CS-2015-002, Department of Mathematics and Computer Science, University of Basel, 2015/10 - Luca Rossetto, Ivan Giangreco, Heiko Schuldt, Stéphane Dupont, Omar Seddati, Metin Sezgin, Yusuf Sahillioğlu
IMOTION - a Content-based Video Retrieval Engine
Proceedings of the 21st MultiMedia Modelling Conference (MMM2015) - Video Search Showcase Track, Sydney, Australia 2015/1
2014
- Luca Rossetto, Ivan Giangreco, Heiko Schuldt
Cineast: A Multi-Feature Sketch-Based Video Retrieval Engine
Proceedings of the IEEE International Symposium on Multimedia (ISM), Taichung, Taiwan 2014/12 - Ivan Giangreco, Ihab Al Kabary, Heiko Schuldt
ADAM — A System for Jointly Providing IR and Database Queries in Large-Scale Multimedia Retrieval
Proceedings of the International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR), Gold Coast, Australia 2014/7 - Ivan Giangreco, Ihab Al Kabary, Heiko Schuldt
ADAM — A Database and Information Retrieval System for Big Multimedia Collections
Proceedings of the International Congress on Big Data (BigData), Anchorage, AK, USA 2014/6