MM-DocTable: Multimedia Document Engineering Workflows on Tabletop Devices (Finished)
Multi-touch devices have recently enjoyed a surge of popularity by supported by an ever-increasing wealth of applications which is a testimony that digital natural user interfaces (NUI) have gained attraction in the public. While paper is still far from extinct, we observe that habits and interaction patterns are more and more shifting towards those types of digital devices, as they become user-friendlier and more effective at performing regular document tasks. Digital interactive tabletops form an important subcategory, which receives considerable attention in the different HCI-research communities, yet surprisingly has not seen a great number of advanced document engineering applications that take advantage of this type of platform. We postulate that bimanual pen-and-touch-operated tabletops, as enhanced virtual office desks, are very suitable platforms to perform a number of document-engineering tasks and are a natural extension of the paper-digital interface paradigm, the obvious advantage of tabletops being that they provide immediate, interactive feedback to users upon their actions. Currently, though, systems only demonstrate the novel UI concepts in rather isolated, single-document environments and are not yet integrated in any particular document engineering workflow. Very often, documents are engaged in a process involving several interactions with interfaces of sometimes heterogeneous systems and so the need to integrate tabletop UIs with a variety of legacy workflows arises. The advent of NUIs has also strongly influenced the multimedia retrieval community. In particular, novel approaches that make use of the interaction capabilities of NUIs for posing queries, such as query by sketch applied to digital image collections, have become increasingly popular. While the general idea to search for images on the basis of a (rough) user-provided sketch is very appealing, it is currently limited to a particular search task, namely known-item search. With the availability of large digital video collections, query by sketch will also become highly relevant for content-based video retrieval. However, in order to go beyond the comparison of a sketch with a single video frame, support for dedicated gestures to express the motion of objects over several frames are needed. The MM-DocTable project aims at providing a sound architecture and tools to support document engineering tasks performed on digital tabletops in a wider range of contexts, in particular by enabling rapid and seamless access to reference material for multimedia document creation or editing. Most importantly, the proposed architecture will support several querying methods (keyword search, query by example, and query by sketch) with a view to apply them interchangeably in a number of retrieval scenarios. The pen-and-touch-operated UI of the tabletop will be designed so that the different retrieval operations are all seamlessly executable within the main document task, that is, without requiring tedious context switches. To achieve these goals, the project will consist of four main parts. First, we will investigate what scenarios can benefit from complex document engineering workflows on tabletop devices that jointly use different query paradigms. Second, we will develop new algorithms for the detection of prominent objects in sketch-based image retrieval in order to apply query by sketch to other interaction intentions beyond known item search. Third, we will extend sketch-based retrieval to digital video collections, by adding gestures for specifying the motion of objects. Fourth, we will seamlessly integrate all these services into the tabletop UI so that users are able to directly access elements of the multimedia database and use this integrated system in user studies.
Start / End Dates
01.10.2011 - 30.09.2014
Partners
Prof. Moira Norrie, ETH Zürich (globis Group Website)
Funding Agencies
Swiss National Science Foundation (SNF)
Funding
286'873.- CHF
Staff
Research Topics
Publications
2014
- Ihab Al Kabary, Heiko Schuldt
SKETCHify - an Adaptive Prominent Edge Detection Algorithm for Optimized Query-by-Sketch Image Retrieval
Post-Proceedings of the 10th International Workshop on Adaptive Multimedia Retrieval (AMR’12), Copenhagen, Denmark 2014/12 - Ihab Al Kabary, Heiko Schuldt
Enhancing Sketch-based Sport Video Retrieval by Suggesting Relevant Motion Paths
Proceedings of the 37th ACM SIGIR Conference, Gold Coast, Australia 2014/7 - Ihab Al Kabary, Heiko Schuldt
Using Hand Gestures for Specifying Motion Path Queries in Sketch-based Video Retrieval
Proceedings of the 36th European Conference on Information Retrieval (ECIR), Amsterdam, Netherlands 2014/4
2013
- Ihab Al Kabary, Ivan Giangreco, Heiko Schuldt, Fabrice Matulic, Moira Norrie
QUEST – Towards a Multi-Modal CBIR Framework Combining Query-by-Example, Query-by-Sketch, and Text Search
Proceedings of the IEEE International Workshop on Multimedia Information Processing and Retrieval (MIPR), Anaheim, CA, USA 2013/12 - Ihab Al Kabary, Heiko Schuldt
Towards Sketch-based Motion Queries in Sports Videos
Proceedings of the 15th IEEE International Symposium on Multimedia (ISM 2013), Anaheim, CA, USA 2013/12 - Ihab Al Kabary, Heiko Schuldt
SportSense: Using Motion Queries to Find Scenes in Sports Videos
Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), San Francisco, CA, USA 2013/10 - Ihab Al Kabary, Marcel Büchler, Heiko Schuldt
TOUCHify:Bringing Pen-Based Touch Screen Functionality to Flat Panel Display Screens
Proceedings of the International Conference on Information Society (i-Society 2013), Toronto, Canada 2013/6 - Fabrice Matulic, Moira C. Norrie, Ihab Al Kabary, Heiko Schuldt
Gesture-Supported Document Creation on Pen and Touch Tabletops
Proceedings of the 31st ACM Conference on Human Factors in Computing Systems (CHI 2013), Paris, France 2013/4
2012
- Ihab Al Kabary, Heiko Schuldt
SKETCHify - an Adaptive Prominent Edge Detection Algorithm for Optimized Query-by-Sketch Image Retrieval
Proceedings of the 10th International Workshop on Adaptive Multimedia Retrieval (AMR’12), Copenhagen, Denmark 2012/10 - Ihab Al Kabary, Heiko Schuldt
Sketch-based Image Similarity Search with a Pen and Paper Interface
Proceedings of the 2012 International Conference on Research and Development in Information Retrieva, Portland, OR, USA 2012/8 - Ivan Giangreco, Michael Springmann, Ihab Al Kabary, Heiko Schuldt
A User Interface for Query-by-Sketch based Image Retrieval with Color Sketches
Proceedings of the European Conference on Information Retrieval (ECIR), Barcelona, Spain 2012/4 - Roman Kreuzer, Michael Springmann, Ihab Al Kabary, Heiko Schuldt
An Interactive Paper and Digital Pen Interface for Query-by-Sketch Image Retrieval
Proceedings of the 34th European Conference on Information Retrieval, Barcelona, Spain 2012/4 - Michael Springmann
Building Blocks for Adaptable Image Search in Digital Libaries
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2012/4