Workshop: Usefulness of interactive IR systems

At CHIIR 2016, Chapel Hill, North Carolina, USA 13.3-17.3.2016

Motivation, Goals and Objectives

Information Retrieval (IR) systems aim at helping a user to solve an information seeking problem. There is a long-standing tradition in IR research to measure the quality of an IR system in terms of the relevance of the documents returned (so-called Cranfield paradigm). However, due to the high interactivity of current IR systems it turned out that pure relevance-based measures fail to capture many factors that should be taken into account for IR evaluations, such as learning, task completion and effort. Current IR research therefore started to "take a broader perspective of the information seeking process to explicitly include users, tasks, and contexts in a dynamic setting" (Cole et al. 2009). This novel paradigm in IR evaluation expands the perspective to the entire search process instead of just evaluating single search results (cp. Dumais 2012). In addition, modern digital libraries offer more functionality besides search e.g. – browsing relations, recommender, storing and structuring information, sharing information -- broadening the user-system interaction possibilities. These interactions lead to more valuable data for a better understanding of user needs and contexts and what is useful or not.

The notion of usefulness was first introduced by Cole et al. (2009) as a general criterion evaluating "how well the user is able to achieve their goal". However, there is still a lack of computational usefulness metrics that can be taken to evaluate interactive IR systems.

The main goal of the workshop is to provide an international forum for discussing novel approaches that might contribute to an approximation of usefulness in interactive information retrieval. The workshop aims at bringing together experts from both 'user' and 'system' oriented information retrieval for a fruitful exchange of ideas and discussion how to tackle the evaluation of interactive IR from the perspective of usefulness.

The long-term research goal is to develop and evaluate new approaches for measuring usefulness of interactive IR systems. More specifically, we address questions such as:

  • What is usefulness and how can it be measured?
  • How can logging tools and frameworks look like to better capture usefulness?
  • How can usefulness be evaluated?
  • What can usefulness contribute to the improvement of interactive IR systems?

Workshop Topics

Contributions are solicited on, but not limited to, following topics:

  • Evaluation of interactive information retrieval
  • Information seeking behavior
  • Task based user modelling, interaction and personalization
  • Logging frameworks for sessions and tasks
  • Analyzing user behavior

Format, Structure and Audience

The format of the workshop should reflect a form of grass root movement, meaning to go bottom up in order to identify real measurements, to analyze existing logs and the gap in logs to measure usefulness along several dimensions.

The workshop will be a full-day workshop with a keynote speech and paper sessions laying the ground of understanding as first part. The paper sessions will focus on the following areas:

  • Theoretical measures for usefulness
  • How can usefulness be observed?
    • Tools
    • Case studies
    • Evaluation methods and metrics

For the second part we propose a world cafe format to enable brainstorming of ideas and deeper discussions of the approaches presented in the morning. We intend to have discussion rounds of 15-30 minutes at max. four tables which are moderated by the organizers and/or participants. The participants should move from table to table after each round. In the final session the discussion result will be presented by the moderators.

The workshop is intended for researchers in the field of interactive information retrieval and digital libraries and librarians interested in a better understanding of users and user needs and provide better system support to enhance the digital libraries systems.


Workshop proceedings will be deposited online in the CEUR workshop proceedings publication service (ISSN 1613-0073). This way the proceedings will be permanently available and citable (digital persistent identifiers and long term preservation). All accepted workshop papers will be published in the workshop proceedings.

Program Committee

  • Leif Azzopardi, University of Glasgow, UK
  • Stefano Mizzaro, University of Udine, IT
  • Norbert Fuhr, University of Duisburg-Essen, Germany
  • Claus-Peter Klas, GESIS-Leibniz-Institute for the Social Sciences, Germany
  • Peter Mutschke, GESIS-Leibniz-Institute for the Social Sciences, Germany
  • Vivien Petras, Humboldt-Universität zu Berlin, Germany
  • Barbara Wildermuth, School of Information and Library Science,University of North Carolina at Chapel Hill, USA
  • Chirag Shah, Rutgers University, USA


Short bios of organizers

Claus-Peter Klas is lead of the GESIS-Architecture team in the department Knowledge Technologies for the Social Sciences of GESIS. He received his PhD computer science at the University of Duisburg-Essen and was a postdoctoral researcher in the Department of Multimedia and Internet Applications, Faculty of Mathematics and Computer Science, University of Hagen, Germany. His research focuses on information retrieval, interactive information retrieval, information systems, databases, digital libraries, preservation and grid and cloud architectures. He developed the software Daffodil founded on a nation research project and worked in national and European research projects such as The European Film Gateway, SHAMAN (Sustaining Heritage Access through Multivalent ArchiviNg) and Smart Vortex (Scalable Semantic Product Data Stream Management for Collaboration and Decision Making in Engineering). He is currently responsible for several infrastructure projects within GESIS, such as da|ra, SowiDataNet or Missy, all concerned with providing information and data for social scientists.

Peter Mutschke is senior researcher at GESIS – Leibniz Institute for the Social Sciences (Cologne) and acting head of the GESIS department “Knowledge Technologies for the Social Sciences”. His research focuses on information retrieval, network analysis and Social Web. He worked in a number of national and international research projects such as DAFFODIL (Distributed Agents for User-Friendly Access of Digital Libraries), INFOCONNEX (interdisciplinary information network for Social Sciences, Education Science and Psychology), IRM (value-added search services), the DELOS/NSF Working Group on reference models for digital libraries, and the EU-funded projects WeGov (Where eGovernment meets the eSociety) and SENSE4US (Data insights for policy makers & citizens). Currently, he is involved in major national and European research networks such as the COST Action KNOWeSCAPE (“Analyzing the dynamics of information and knowledge landscapes”) and the research alliance “Science 2.0” of the German Leibniz Association. For both research networks Peter is member of the management committee. Peter is author of a number of research articles, member of a number of international program committees, was co-organizer of the workshop “Combining Bibliometrics and Information Retrieval” at ISSI 2013 and “Bibliometric-enhanced Information Retrieval (BIR)” at ECIR 2014 and main organizer of the “Knowledge Maps and Information Retrieval (KMIR)” workshop at DL 2014.

Preben Hansen is an Associate Professor at the Department of Computer and Systems Sciences, Stockholm University. He earned his Ph.D. from the Department of Information School at Tampere University, Finland (2011), under the supervision of Professor Kalervo Järvelin. He also has a Master Science degree in Information Science from the College of Borås/Gothenburg. Special research interests are: Collaborative Information Seeking; Session-based searching; Task-based information seeking' User and Usage-oriented studies; Technology Enhanced-Learning Design; Participatory design and Ethnography.

Leif Azzopardi is a Senior Lecturer within the Glasgow Information Retrieval Group within the School of Computing Science, at the University of Glasgow. His research focuses on modeling how people interact with information and information systems. Specifically, his work is aimed at understanding the influence and impact of (search) technology on people and society. His work encompasses Information Retrieval, Information Science and Human Computer Interaction, incorporating theory and models from disciplines such as Economics, Psychology, Operations Research, Transportation Planning, Urban Studies and Quantum Mechanics. His key contributions include the development of models for: expertise retrieval, query generation, simulation of interaction, retrievability, search and search behavior. More recently, he has been using Economic Theory to develop models that not only describe, but also predict and explain people’s information behaviors and their decisions.