vitrivr (Ongoing)

vitrivr is an open source full-stack content-based multimedia retrieval system. It started with support for video, but its modular architecture has made it easy to extend it to other types of media as well. Currently, it also supports image, audio and 3D collections. Moreover, vitrivr features a very broad set of query paradigms that are supported: textual queries, query-by-sketch, concept queries, semantic sketches (semantic concepts and their spatial relationship within the object), query-by-gesture (in videos), and temporal as well as spatial queries (the latter particularly on mobile devices) - to just name the most prominent ones. The virtual reality version of vitrivr is vitrivr-VR.

Awards

Google Summer of Code

vitrivr has been part of Google Summer of Code.

In 2016, vitrivr was supported by Google Summer of Code in the projects

Cthulhu - Vitrivr Feature Extraction Pipeline
Motion-based Retrieval: Feature Extraction
Natural Language Query Interface (a demo video and scientific publication have been published)

In 2018, vitrivr was supported in the projects

Subtitles for vitrivr
vitrivr App

In 2021, vitrivr had two exciting projects

IIIF support
Scene Text Recognition

In the Media

03.12.2016: Innovationspreis für Pilz-Zucht auf Kaffeesatz (bz Basel) on the City-Stories project and its use of vitrivr
31.05.2016: Bild- und Videosuche per Handskizze (informatik.ch) on vitrivr
29.07.2016: Lauter bekannte Gesichter (Hannoversche Allgemeine) on vitrivr
06.07.2016: Suchmaschine für Skizzen (sem-seo.at) on vitrivr
20.06.2016: Ein Bild sucht mehr als tausend Worte (Deutschlandfunk) report and interview on IMOTION/vitrivr
03.06.2016: Online-Suche mittels Skizzen (20min) on IMOTION/vitrivr
31.05.2016: In a new method for searching image databases, a hand-drawn sketch is all it takes (phys.org) on vitrivr
04.11.2014: Videos via Suchanfrage durchsuchen (netzwoche) on IMOTION
05.09.2014: Big data tamed with the cloud (Microsoft Research Blog) on ADAM/ADAMpro

Website

https://www.vitrivr.org

Since

01.01.2016

Funding

vitrivr has been supported in 2016, 2018 and 2021 by Google Summer of Code (GSoC).

Staff

Research Topics

Publications

2024

Loris Sauter, Ralph Gasser, Heiko Schuldt, Abraham Bernstein, Luca Rossetto
Performance Evaluation in Multimedia Retrieval
ACM Transactions on Multimedia Computing, Communications and Applications, 2024/10
Florian Spiess, Raphael Waltenspül, Heiko Schuldt
The Sketchfab 3D Creative Commons Collection (S3D3C)
, 2024/7
Loris Sauter, Ralph Gasser, Laura Rettig, Heiko Schuldt, Luca Rossetto
General Purpose Multimedia Retrieval with vitrivr at LSC’24
LSC ‘24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge, Phuket, Thailand 2024/6
Florian Spiess, Heiko Schuldt, Luca Rossetto
Spatiotemporal Lifelog Analytics in Virtual Reality with vitrivr-VR
, 2024/6
Loris Sauter
Flexible Information Retrieval Evaluation
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2024/2
Ralph Gasser, Rahel Arnold, Fynn Faber, Heiko Schuldt, Raphael Waltenspül, Luca Rossetto
A new Retrieval Engine for vitrivr
MMM 2024 - The 30th International Conference on Multimedia Modeling, Amsterdam, The Netherlands 2024/1
Loris Sauter, Tim Bachmann, Heiko Schuldt, Luca Rossetto
Augmented Reality Photo Presentation and Content-Based Image Retrieval on Mobile Devices with AR-Explorer
Proceedings of the 30th International Conference on Multimedia Modeling (MMM 2024), Amsterdam, Netherlands 2024/1
Florian Spiess, Luca Rossetto, Heiko Schuldt
Exploring Multimedia Vector Spaces with vitrivr-VR
, 2024/1

2023

Loris Sauter, Tim Bachmann, Luca Rossetto, Heiko Schuldt
Spatially Localised Immersive Contemporary and Historic Photo Presentation on Mobile Devices in Augmented Reality
Proceedings of the 5th Workshop on analySis, Unerstanding and proMotion of heritAge Contents, Ottowa ON, Canada 2023/11
Loris Sauter, Heiko Schuldt, Raphael Waltenspül, Luca Rossetto
Novice-Friendly Text-based Video Search with vitrivr
Proceedings of the 20th International Conference on Content-based Multimedia Indexing, Orléans, France 2023/9
Florian Spiess, Ralph Gasser, Silvan Heller, Heiko Schuldt, Luca Rossetto
A Comparison of Video Browsing Performance between Desktop and Virtual Reality Interfaces
Proceedings of the International Conference on Multimedia Retrieval (ICMR ‘23), Thessaloniki, Greece 2023/6
Werner Bailer, Rahel Arnold, Vera Benz, Davide Coccomini, Anastasios Gkagkas, Gylfi Þór Guðmundsson, Silvan Heller, Björn Þór Jónsson, Jakub Lokoc, Nicola Messina, Nick Pantelidis, Jiaxin Wu
Improving Query and Assessment Quality in Text-Based Interactive Video Retrieval Evaluation
Proceedings of the International Conference on Multimedia Retrieval (ICMR ‘23), 2023/6
Florian Spiess, Ralph Gasser, Heiko Schuldt, Luca Rossetto
The Best of Both Worlds: Lifelog Retrieval with a Desktop-Virtual Reality Hybrid System
Proceedings of the 6th Annual ACM Lifelog Search Challenge (LSC’23), Thessaloniki, Greece 2023/6
Murielle Cornut, Julien Antoine Raemy, Florian Spiess
Annotations as Knowledge Practices in Image Archives: Application of Linked Open Usable Data and Machine Learning
ACM Journal on Computing and Cultural Heritage, 2023/5
Silvan Heller, Florian Spiess, Heiko Schuldt
A tale of two interfaces: vitrivr at the lifelog search challenge
Multimedia Tools and Applications, 2023/4
Silvan Heller
Temporal Multimodal Video and Lifelog Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2023/3
Ly-Duyen Tran, Manh-Duy Nguyen, Duc-Tien Dang-Nguyen, Silvan Heller, Florian Spiess, Jakub Lokoč, Ladislav Peška, Thao-Nhu Nguyen, Omar Shahbaz Khan, Aaron Duane, Björn Þór Jónsson, Luca Rossetto, An-Zi Yen, Ahmed Alateeq, Naushad Alam, Minh-Triet Tran, Graham Healy, Klaus Schoeffmann, Cathal Gurrin
Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021
IEEE Access, 2023/2
Loris Sauter, Ralph Gasser, Silvan Heller, Luca Rossetto, Colin Saladin, Florian Spiess, Heiko Schuldt
Exploring Effective Interactive Text-based Video Search in vitrivr
Proceedings of the 29th International Conference on Multimedia Modeling (MMM 2023), Bergen, Norway 2023/1
Rahel Arnold, Loris Sauter, Heiko Schuldt
Free-form Multi-Modal Multimedia Retrieval (4MR)
Proceedings of the 29th International Conference on Multimedia Modeling (MMM 2023), Bergen, Norway 2023/1
Florian Spiess, Silvan Heller, Luca Rossetto, Loris Sauter, Philipp Weber, Heiko Schuldt
Traceable Asynchronous Workflows in Video Retrieval with vitrivr-VR
Proceedings of the 29th International Conference on Multimedia Modeling (MMM 2023), Bergen, Norway 2023/1

2022

Florian Spiess, Philipp Weber, Heiko Schuldt
Direct Interaction Word-Gesture Text Input in Virtual Reality
Proceedings of the 5th IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR 2022), held virtually 2022/12
Ralph Gasser
Data Management for Dynamic Multimedia Analytics and Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2022/11
Loris Sauter, Ralph Gasser, Abraham Bernstein, Heiko Schuldt, Luca Rossetto
An Asynchronous Scheme for the Distributed Evaluation of Interactive Multimedia Retrieval
Proceedings of the 2nd International Workshop on Interactive Multimedia Retrieval (IMuR’22), Lisbon, Portugal 2022/10
Florian Spiess, Heiko Schuldt
Multimodal Interactive Lifelog Retrieval with vitrivr-VR
Proceedings of the fifth Lifelog Search Challenge (LSC), Newark, NJ, USA 2022/6
Silvan Heller, Luca Rossetto, Loris Sauter, Heiko Schuldt
vitrivr at the Lifelog Search Challenge 2022
Proceedings of the fifth Lifelog Search Challenge (LSC), Newark, NJ, USA 2022/6
Silvan Heller, Rahel Arnold, Ralph Gasser, Viktor Gsteiger, Mahnaz Parian-Scherb, Luca Rossetto, Loris Sauter, Florian Spiess, Heiko Schuldt
Multi-Modal Interactive Video Retrieval with Temporal Queries
Proceedings of the 28th International Conference on Multimedia Modeling, Qui Nhon, Vietnam 2022/4
Florian Spiess, Ralph Gasser, Silvan Heller, Mahnaz Parian-Scherb, Luca Rossetto, Loris Sauter, Heiko Schuldt
Multi-Modal Video Retrieval in Virtual Reality with vitrivr-VR
Proceedings of the 28th International Conference on Multimedia Modeling, Qui Nhon, Vietnam 2022/4
Jakub Lokoč, Werner Bailer, Kai Uwe Barthel, Cathal Gurrin, Silvan Heller, Björn þór Jónsson, Ladislav Peška, Luca Rossetto, Klaus Schoeffmann, Lucia Vadicamo, Stefanos Vrochidis, Jiaxin Wu
A Task Category Space for User-Centric Comparative Multimedia Search Evaluations
MultiMedia Modeling, 2022/3
Silvan Heller, Viktor Gsteiger, Werner Bailer, Cathal Gurrin, Björn Þór Jónsson, Jakub Lokoč, Andreas Leibetseder, František Mejzlík, Ladislav Peška, Luca Rossetto, Konstantin Schall, Klaus Schoeffmann, Heiko Schuldt, Florian Spiess, Ly-Duyen Tran, Lucia Vadicamo, Patrik Veselý, Stefanos Vrochidis, Jiaxin Wu
Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown
International Journal of Multimedia Information Retrieval, 2022/1

2021

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Parian-Scherb, Loris Sauter, Florian Spiess, Heiko Schuldt, Ladislav Peška, Tomáš Souček, Miroslav Kratochvíl, František Mejzlík, Patrik Veselý, Jakub Lokoč
On the User-centric Comparative Remote Evaluation of Interactive Video Search Systems
IEEE Multimedia, 2021/10
Florian Spiess, Ralph Gasser, Silvan Heller, Luca Rossetto, Loris Sauter, Milan van Zanten, Heiko Schuldt
Exploring Intuitive Lifelog Retrieval and Interaction Modes in Virtual Reality with vitrivr-VR
Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR’21), Taipei, Taiwan (held virtually) 2021/8
Silvan Heller, Ralph Gasser, Mahnaz Parian-Scherb, Sanja Popovic, Luca Rossetto, Loris Sauter, Florian Spiess, Heiko Schuldt
Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021
Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR’21), Taipei, Taiwan (held virtually) 2021/8
Jakub Lokoč, Patrik Veselý, František Mejzlík, Gregor Kovalčík, Tomáš Souček, Luca Rossetto, Klaus Schoeffmann, Werner Bailer, Cathal Gurrin, Loris Sauter, Jaeyub Song, Stefanos Vrochidis, Jiaxin Wu, Björn þóR Jónsson
Is the Reign of Interactive Search Eternal? Findings from the Video Browser Showdown 2020
ACM Transactions on Multimedia Computing, Communications, and Applications, 2021/7
Luca Rossetto, Ralph Gasser, Loris Sauter, Abraham Bernstein, Heiko Schuldt
A System for Interactive Multimedia Retrieval Evaluations
Proceedings of the 27th International Conference on Multimedia Modeling (MMM 2021), Prague, Czech Republic (held virtually) 2021/6
Florian Spiess, Ralph Gasser, Silvan Heller, Luca Rossetto, Loris Sauter, Heiko Schuldt
Competitive Interactive Video Retrieval in Virtual Reality with vitrivr-VR
Proceedings of the 27th International Conference on Multimedia Modeling (MMM 2021), Prague, Czech Republic (held virtually) 2021/6
Silvan Heller, Ralph Gasser, Cristina Illi, Maurizio Pasquinelli, Loris Sauter, Florian Spiess, Heiko Schuldt
Towards Explainable Interactive Multi-Modal Video Retrieval with vitrivr
Proceedings of the 27th International Conference on Multimedia Modeling (MMM 2021), Prague, Czech Republic (held virtually) 2021/6
Ralph Gasser, Luca Rossetto, Silvan Heller, Heiko Schuldt
Multimedia Retrieval and Analysis with Cottontail DB
ACM SIGMM Records, 2021/3

2020

Samuel Börlin, Ralph Gasser, Florian Spiess, Heiko Schuldt
3D Model Retrieval Using Constructive Solid Geometry in Virtual Reality
Proceedings of the 3rd International Conference on Artificial Intelligence & Virtual Reality (AIVR 2020), Utrecht, The Netherlands (held virtually) 2020/12
Silvan Heller, Mahnaz Amiri Parian, Ralph Gasser, Loris Sauter, Heiko Schuldt
Interactive Lifelog Retrieval with vitrivr
Proceedings of the Third Annual Workshop on Lifelog Search Challenge (LSC 2020), Dublin, Ireland (held virtually) 2020/10
Silvan Heller, Mahnaz Parian-Scherb, Maurizio Pasquinelli, Heiko Schuldt
vitrivr-explore: Guided Multimedia Collection Exploration for Ad-hoc Video Search
In Proceedings of the 13th International Conference on Similarity Search and Applications (SISAP 2020), Copenhagen, Denmark (held virtually) 2020/10
Silvan Heller, Loris Sauter, Heiko Schuldt, Luca Rossetto
Multi-Stage Queries and Temporal Scoring in vitrivr
2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), London, UK (held virtually) 2020/7
Mahnaz Amiri Parian, Luca Rossetto, Heiko Schuldt, Stéphane Dupont
vitrivr and Deepmime: Gesture-based search in Video Collections
Proceedings of the International Conference on Multimodal Communication (ICMC): Developing New Theories and Methods, Osnabrück, Germany 2020/5
Luca Rossetto, Ralph Gasser, Jakub Lokoč, Werner Bailer, Klaus Schoeffmann, Bernd Muenzer, Tomás Souček, Phuong Anh Nguyen, Paolo Bolettieri, Andreas Leibetseder, Stefanos Vrochidis
Interactive Video Retrieval in the Age of Deep Learning – Detailed Evaluation of VBS 2019
IEEE Transactions on Multimedia, 2020/3
Loris Sauter, Mahnaz Amiri Parian, Ralph Gasser, Silvan Heller, Luca Rossetto, Heiko Schuldt
Combining Boolean and Multimedia Retrieval in vitrivr for large-scale Video Search
Proceedings of the 26th International Conference on MultiMedia Modeling, Daejeon, Korea 2020/1

2019

Luca Rossetto, Ralph Gasser, Heiko Schuldt
Query by Semantic Sketch
arXiv cs.MM, 2019/9
Jakub Lokoč, Klaus Schöffmann, Werner Bailer, Luca Rossetto, Cathal Gurrin
Interactive video retrieval in the age of deep learning
Proceedings of the International Conference on Multimedia Retrieval (ICMR), Ottawa, ON, Canada 2019/6
Ralph Gasser, Luca Rossetto, Heiko Schuldt
Multimodal Multimedia Retrieval with vitrivr
Proceedings of the International Conference on Multimedia Retrieval (ICMR), Ottawa, ON, Canada 2019/6
Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Amiri Parian, Heiko Schuldt
Retrieval of Structured and Unstructured Data with vitrivr
Proceedings of the International Conference on Multimedia Retrieval (ICMR), Ottawa, ON, Canada 2019/6
Luca Rossetto, Fabian Berns, Klaus Schoeffmann, George Awad, Christian Beecks
The V3C1 Dataset: Advancing the State of the Art in Video Retrieval
ACM SIGMM Records, 2019/6
Luca Rossetto, Mahnaz Amiri Parian, Ralph Gasser, Ivan Giangreco, Silvan Heller, Heiko Schuldt
Deep Learning-based Concept Detection in vitrivr at the Video Browser Showdown 2019 - Final Notes
arXiv cs.MM, 2019/2
Jakub Lokoc, Gregor Kovalcic, Bernd Münzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, Kai Uwe Barthel
Interactive Search or Sequential Browsing? A Detailed Analysis of the VBS 2018
ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2019/2
Ralph Gasser, Luca Rossetto, Heiko Schuldt
Towards an All-Purpose Content-Based Multimedia Information Retrieval System
arXiv cs.MM, 2019/2
Luca Rossetto, Mahnaz Amiri Parian, Ralph Gasser, Ivan Giangreco, Silvan Heller, Heiko Schuldt
Deep Learning-based Concept Detection in vitrivr
Proceedings of the 25th International Conference on MultiMedia Modeling, Thessaloniki, Greece 2019/1

2018

Luca Rossetto
Multi-Modal Video Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel 2018/9
Luca Rossetto, Ivan Giangreco, Ralph Gasser, Heiko Schuldt
Competitive Video Retrieval with vitrivr at the Video Browser Showdown 2018 - Final Notes
arXiv cs.MM, 2018/5
Ivan Giangreco
Database Support for Large-Scale Multimedia Retrieval
PhD Thesis, Department of Mathematics and Computer Science, University of Basel 2018/3
Luca Rossetto, Ivan Giangreco, Ralph Gasser, Heiko Schuldt
Competitive Video Retrieval with vitrivr
Proceedings of the International Conference on Multimedia Modeling (MMM), Bangkok, Thailand 2018/2
Luca Rossetto, Heiko Schuldt
The Long Tail of Web Video
Proceedings of the 24th International Conference on Multimedia Modeling, Bangkok, Thailand 2018/2

2017

Luca Rossetto, Ivan Giangreco, Ralph Gasser, Heiko Schuldt
Content-based Multimedia Retrieval using vitrivr
ACM SIGMM Records, 2017/12
Ivan Giangreco, Luca Rossetto, Claudiu Tănase, Heiko Schuldt
vitrivr: A Multimedia Search System supporting Multimodal Interactions
Proceedings of the International Conference on Multimodal Communication: Developing New Theories and Methods, Osnabrück, Germany 2017/6
Prateek Goel, Ivan Giangreco, Luca Rossetto, Claudiu Tănase, Heiko Schuldt
Hey, vitrivr! - A Multimodal UI for Video Retrieval
Proceedings of the European Conference on Information Retrieval (ECIR), Aberdeen, Scotland, UK 2017/4

2016

Claudiu Tănase, Luca Rossetto, Ivan Giangreco, Heiko Schuldt
The vitrivr System at TRECVID 2016: The Ad-Hoc Video Search Task
Proceedings of the 2016 TRECVID Ad-Hoc Video Search Task, 2016/10
Luca Rossetto, Ivan Giangreco, Claudiu Tănase, Heiko Schuldt
vitrivr - A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections
Proceedings of the International ACM Conference on Multimedia (ACM MM), Amsterdam, NL 2016/10