Building Blocks for Adaptable Image Search in Digital Libaries

Michael Springmann

PhD Thesis

2012/4

PhD Thesis, Department of Mathematics and Computer Science

University of Basel, Switzerland

With the availability of easy and inexpensive methods to create and store images in digital formats, the visual information preserved and shared electronically has grown dramatically. As images are important means to archive, express, and communicate human knowledge, experience, and feelings it is desirable or even unavoidable that digital libraries do not contain only textual information, but also such images. One central aspect of digital libraries is the ability, to make its content easily available to its users and therefore also to provide adequate retrieval mechanisms for image-related search tasks. Traditional text- and metadata-based approaches are not sufficient as personal digital libraries as well as automatically acquired image collections commonly lack detailed descriptions that could be used in searches. To better support image search in digital libraries also methods from content-based image retrieval (CBIR) are needed:CBIR provides mechanisms to search for images by using the image content itself and compare the images with (visual) input the user provides and ranking the results based on similarity. The aim of this thesis is to identify, implement, and evaluate building blocks that can be used to build digital libraries with the ability to perform similarity search for images in addition to traditional approaches. This thesis follows a top-down approach and has three main contributions:First, we introduce the Image Task Model (ITM) to characterize the user’s intention in image-related search tasks. This new model integrates and refines pre-existing models into one concise model for interaction intentions. It considers the user’s Task Input  and Aim, Matching Tolerance, and intended Result Usage.Second, we use ITM to identify conceptual building blocks that provide the required functionality in digital libraries to support CBIR and similarity search in general:Content Management, Query Formulation and Execution, and User Interaction. These conceptual building blocks and their interactions are analyzed and a comprehensive survey reviews state-of-the-art approaches to which extent they can support search tasks on the basis of ITM to identify strong and weak spots. Third, we present a detailed discussion of selected building blocks together with our own implementations that extend and improve state-of-the-art approaches to better support similarity searches for images in digital libraries. The key principal that we follow is adjusting the matching tolerance to the needs of a task, such that existing building blocks can be reused and optimized for different application domains. To demonstrate the reusability, we show prototypical implementations of complete digital library systems based on our building blocks for three different domains:automatic classification of medical images, sketch-based search for known images, and retrospective geo-tagging of images. This thesis therefore supports future development of digital libraries with image search functionality from the early stage of understanding the user requirements through characterizing user tasks in ITM over the selection of appropriate conceptual building blocks for providing the required functionality to finally implement entire systems with the potential to reuse existing building blocks.