MM-DocTable: Multimedia Document Engineering Workflows on Tabletop Devices (Finished)

Multi-touch devices have recently enjoyed a surge of popularity by supported by an ever-increasing wealth of applications which is a testimony that digital natural user interfaces (NUI) have gained attraction in the public. While paper is still far from extinct, we observe that habits and interaction patterns are more and more shifting towards those types of digital devices, as they become user-friendlier and more effective at performing regular document tasks. Digital interactive tabletops form an important subcategory, which receives considerable attention in the different HCI-research communities, yet surprisingly has not seen a great number of advanced document engineering applications that take advantage of this type of platform. We postulate that bimanual pen-and-touch-operated tabletops, as enhanced virtual office desks, are very suitable platforms to perform a number of document-engineering tasks and are a natural extension of the paper-digital interface paradigm, the obvious advantage of tabletops being that they provide immediate, interactive feedback to users upon their actions. Currently, though, systems only demonstrate the novel UI concepts in rather isolated, single-document environments and are not yet integrated in any particular document engineering workflow. Very often, documents are engaged in a process involving several interactions with interfaces of sometimes heterogeneous systems and so the need to integrate tabletop UIs with a variety of legacy workflows arises. The advent of NUIs has also strongly influenced the multimedia retrieval community. In particular, novel approaches that make use of the interaction capabilities of NUIs for posing queries, such as query by sketch applied to digital image collections, have become increasingly popular. While the general idea to search for images on the basis of a (rough) user-provided sketch is very appealing, it is currently limited to a particular search task, namely known-item search. With the availability of large digital video collections, query by sketch will also become highly relevant for content-based video retrieval. However, in order to go beyond the comparison of a sketch with a single video frame, support for dedicated gestures to express the motion of objects over several frames are needed. The MM-DocTable project aims at providing a sound architecture and tools to support document engineering tasks performed on digital tabletops in a wider range of contexts, in particular by enabling rapid and seamless access to reference material for multimedia document creation or editing.  Most importantly, the proposed architecture will support several querying methods (keyword search, query by example, and query by sketch) with a view to apply them interchangeably in a number of retrieval scenarios. The pen-and-touch-operated UI of the tabletop will be designed so that the different retrieval operations are all seamlessly executable within the main document task, that is, without requiring tedious context switches. To achieve these goals, the project will consist of four main parts. First, we will investigate what scenarios can benefit from complex document engineering workflows on tabletop devices that jointly use different query paradigms. Second, we will develop new algorithms for the detection of prominent objects in sketch-based image retrieval in order to apply query by sketch to other interaction intentions beyond known item search. Third, we will extend sketch-based retrieval to digital video collections, by adding gestures for specifying the motion of objects. Fourth, we will seamlessly integrate all these services into the tabletop UI so that users are able to directly access elements of the multimedia database and use this integrated system in user studies.

Start / End Dates

01.10.2011 - 30.09.2014

Partners

Prof. Moira Norrie, ETH Zürich (globis Group Website)

Funding Agencies

Swiss National Science Foundation (SNF)

Funding

286'873.- CHF

Staff

Research Topics

Publications

2014
2013
2012