The STELLA

project aims to create an evaluation infrastructure allowing retrieval and

recommender systems to be run within academic search systems with real users.

Thus, STELLA provides an integrated e-research environment, a so-called living

lab. The experimental setup differs significantly from classical offline TREC

studies as it allows academic researchers to use an evaluation method

previously reserved for industrial research or operators of large online

platforms. The STELLA framework is designed to support academic search

solutions, dealing with heterogeneity and mass of documents (and datasets) that

still introduce new challenges to the disciplines of information retrieval,

recommender systems, digital libraries, and related fields.

In STELLA

I, the three project partners, TH Köln, ZB MED, and GESIS, aimed to strengthen

the concept of user-centric living labs for the domain of academic search. By

allowing researchers to evaluate their retrieval approaches in two real-world

search portals, the methodological gap between real-world and lab-based evaluations

was effectively reduced. As a result, STELLA was the cornerstone of the CLEF

2021 evaluation lab LiLAS and proved to be reliable and extensible.

In STELLA

II, the three partners will extend the existing framework to include the

additional external partners DIPF, TIB, and ZBW and their corresponding search

platforms and evaluation use-cases. Furthermore, STELLA's current ad-hoc

retrieval and recommendation setup will be extended as a continuous evaluation

platform.