City-Stories: Spatio-temporal search over crowdsourced content (Finished)
The City Stories project aims to develop a smartphone app that makes Basel's urban history accessible in a new and innovative way. In the GoFind! app, historical pictures, videos and even 3D models are shown at their original location so that users can compare the current view with the historical one. For this, City Stories applies concepts of Augmented Reality (AR) and Virtual Reality (VR) to display historic multimedia content. For example, it is possible to "browse through time" by overlying one or several historic photos with the current camera view of the smartphoneand fading smoothly from one photo to another, thereby highlighting the common parts of a city view and the ones that have changed over time. With this, City Stories provides a window into the past.
In a recent collaboration with the Basel University Library, City Stories got access to historic city maps and in particular to a full 3D reconstructtion of the Basel Merian plan from 1615. In her Bachelor's thesis, Rahel Arnold has integrated both into City Stories' GoFInd! smartphone app. It is now possible to show, at any location in the Basel city center (within the city walls of 1615) to display the city buildings of that time in VR.
General City Stories Project Description
A driving force behind the development of novel services tailored to the needs of citizens is the increasing flexibility of social time. Even though city residents may possess valuable information for other citizens visiting their home area, it is hardly usable with today's information systems because of the inevitably scattered information resources. Considering the explosive growth of information, knowledge sharing can ensure valuable interdisciplinary applications and services development. While many organizations propose relevant data sets, they are hardly accessed, analyzed and reused in an integrated way –despite of the enormous added value such integration would have– because of the heterogeneity of formats, the lack of interoperability, and the inappropriate support for information browsing and visualization. The City-Stories platform for data crowdsourcing and spatio-temporal knowledge visualization will overcome these limitations by jointly addressing three tightly coupled challenges. First, data integration responsible for locating, extracting, and syntactically aligning heterogeneous data. Second, novel approaches to spatio-temporal multimedia search and browsing on top of the integrated data, providing users new perspectives and query types to explore these data. Third, the knowledge crowdsourcing and visualization responsible for gathering users' content and enrich this content with linked data services, in order to offer a novel way of knowledge visualization. From a high-level perspective, City-Stories platform is composed of three main modules:
- Data integrator module responsible of gathering distributed and heterogeneous data and offering data alignment, mapping and cleaning
- Retrieval engine that stores combined and semantically enriched data and metadata in order to serve as basis for advanced spatio-temporal and personalized queries
- Knowledge crowdsourcing and visualization module that offers an adaptive and profile aware crowdsourcing and knowledge visualization and navigation interface
City-Stories successfully participated at Innovation Basel 2016, a challenge of the Basel Guilds and Honorary Societies, and was awarded third place (jointly with the Blickwinkel project). A summary of the City-Stories project (in German) as presented at Innovation Basel 2016 can be found here. The following video shows the midterm report of the joint project of City-Stories with projekt BLICKWINKEL at Innovation Basel 2016:
Start / End Dates
01.01.2016 - 31.12.2019
Partners
- Prof. Maria Sokhn, HES-SO Valais
- Prof. Philippe Cudré-Mauroux, eXascale Infolab, University of Fribourg
Funding Agencies
Funding
Total project funding: 392'672.- CHF, project funding UNIBAS: 128'961.- CHF.
The project has been extended for two more years (City-Stories II) with a total budget of 387'100.- CHF; project funding at UNIBAS: 125'872.- CHF.
Staff
- Heiko Schuldt
- Loris Sauter
- Ivan Giangreco
- Rahel Arnold
- Lukas Beck
- Shaban Shabani
- Luca Rossetto
- Fabrizio Parrillo
- Merwin Mudavamkunnel
Research Topics
Publications
2024
- Loris Sauter
Flexible Information Retrieval Evaluation
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2024/2 - Loris Sauter, Tim Bachmann, Heiko Schuldt, Luca Rossetto
Augmented Reality Photo Presentation and Content-Based Image Retrieval on Mobile Devices with AR-Explorer
Proceedings of the 30th International Conference on Multimedia Modeling (MMM 2024), Amsterdam, Netherlands 2024/1
2023
- Loris Sauter, Tim Bachmann, Luca Rossetto, Heiko Schuldt
Spatially Localised Immersive Contemporary and Historic Photo Presentation on Mobile Devices in Augmented Reality
Proceedings of the 5th Workshop on analySis, Unerstanding and proMotion of heritAge Contents, Ottowa ON, Canada 2023/11 - Shaban Shabani
Hybrid Human-Machine Information Systems for Data Classification
PhD Thesis, Department of Mathematics and Computer Science, University of Basel, Switzerland 2023/6
2021
- Laura Rettig, Shaban Shabani, Loris Sauter, Philippe Cudré-Mauroux, Maria Sokhn, Heiko Schuldt
City-Stories: Combining Entity Linking, Multimedia Retrieval, and Crowdsourcing to Make Historical Data Accessible
Proceedings of the 21st International Conference on Web Engineering (ICWE 2021), Biarritz, France (held virtually) 2021/5 - Shaban Shabani, Zarina Charlesworth, Maria Sokhn, Heiko Schuldt
SAMS: Human-in-the-loop Approach to Combat the Sharing of Digital Misinformation
Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering (AAAI-MAKE 2021), Stanford University, Palo Alto, California, USA (held virtually) 2021/3
2020
- Shaban Shabani, Maria Sokhn, Heiko Schuldt
Hybrid Human-Machine Classification System for Cultural Heritage Data
Proceedings of the 2nd workshop on Structuring and Understanding of Multimedia heritAge Contents (SUMAC 20), Seattle, WA, USA (held virtually) 2020/10
2019
- Ivan Giangreco, Loris Sauter, Mahnaz Amiri Parian, Ralph Gasser, Silvan Heller, Luca Rossetto, Heiko Schuldt
VIRTUE — A Virtual Reality Museum Experience
Proceedings of the 24th ACM International Conference on Intelligent User Interfaces (IUI’19), Los Angeles, CA, USA 2019/3
2018
- Loris Sauter, Luca Rossetto, Heiko Schuldt
Exploring Cultural Heritage in Augmented Reality with GoFind!
Proceedings of the first IEEE International Conference on Artificial Intelligence and Virtual Reality, Taichung, Taiwan 2018/12
2017
- Shaban Shabani, Maria Sokhn, Laura Rettig, Philippe Cudré-Mauroux, Lukas Beck, Claudiu Tănase, Heiko Schuldt
City-Stories: A Multimedia Hybrid Content and Entity Retrieval System for Historical Data
Proceedings of the 4th International Workshop on Computational History (HistoInformatics 2017), Singapore 2017/11
2016
- Lukas Beck, Heiko Schuldt
City-Stories: A Spatio-Temporal Mobile Multimedia Search System
Proceedings of the 2016 IEEE International Symposium on Multimedia (ISM 2016), San Jose, CA, USA 2016/12