Cross-Language Evaluation Forum - CLEF (in cooperation with the Network of Excellence on Digital Libraries)

Staff: Stefan Baerisch
Project leader: Prof. Dr. Jürgen Krause
Contact: Dr. Vivien Petras

Project description:

The CLEF project is now an independent EC-project in cooperation with DELOS Network of Excellence on Digital Libraries. The CLEF project offers users a testbed and evaluation environment for testing multilingual retrieval systems. For European languages the long established evaluation program TREC-CLIR is continued based on the TREC-evaluation method. The goal is for the system to return a number of cross-language documents for any user inquiry submitted in any given European language and to produce relevance ranked results as an answer to the query. The intended focus points of evaluation are as follows: 1. Multilingual information retrieval 2. Bilingual information retrieval, 3. Single language (non-English) information retrieval, 4. Information retrieval in the area-specialized document selection GIRT (German Indexing and Retrieval Test database), already assembled by GESIS-IZ and which contains selections from the SOFIS and SOLIS databases.

Information about the CLEF Domain-Specific Track 2008 is now online (http://www.gesis.org/en/research/information_technology/clef_ds.htm).

Project partner:

• Eurospider Information Technology AG Zürich
• Universidad National de Educación a Distanzia Madrid (UNED)
• National Institute for Standards and Technology (NIST) Gaithersburg (USA)
• Istituto di Scienza e Tecnologie dell'Informazione, Consiglio Nazionale delle Ricerche (ISTI-CNR), Pisa
• The Evaluations and Language resources Distribution Agency (ELDA), Paris
• Swedish Institute of Computer Science (SICS), Kista
• University of Tampere - Department of Information Studies, Tampere
• University of Twente - Department of Computer Science, Enschede
• in cooperation with Universität Hildesheim, Institut für Angewandte Sprachwissenschaft - Informationswissenschaft

Duration: (since January 2000)  October 2001 - March 2004

Funding source: European Community

Publications:

• Kluck, Michael; Womser-Hacker, Christa (2002): Inside the Evaluation Process of the Cross-Language Evaluation Forum (CLEF): Issues of Multilingual Topic Creation and Multilingual Relevance Assessment. S. 573-576. In: Rodríguez, M. G.; Araujo, C.P.S. (Hrsg.): Proceedings of the Third International Conference on Language Resources and Evaluation, LREC 2002, Las Palmas de Gran Canaria, 29-31 May 2002. Paris: ELRA. URL: preprint: http://www.educat.hu-berlin.de/~kluck/lrec-paper-final.doc  or http://www.educat.hu-berlin.de/~kluck/lrec-paper-final.pdf 
• Michael Kluck: Das Cross-Language Evaluation Forum (CLEF) - Evaluationsumgebung und Forschungskontext für mehrsprachiges Information Retrieval (mit einer Skizze der Ergebnisse von CLEF 2002). In: Rainer Hammwöhner, Christian Wolff, Christa Womser.-Hacker (eds.): Information und Mobilität. Optimierung und Vermeidung von Mobilität durch Information. Proceedings des 8. Internationalen Symposiums für Informationswissenschaft (ISI 2002). Konstanz: UVK 2002, 225-237. Preprint at http://www.educat.hu-berlin.de/~kluck/clef-isi2002-2.rtf; slides at http://www.educat.hu-berlin.de/~kluck/isi2002-clef2002.ppt
• Carol Peters, Martin Braschler, Julio Gonzalo, Michael Kluck (eds.): Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001, Darmstadt, Germany, September 3-4, 2001. Revised Papers. Berlin u.a.: Springer 2002 (= Lecture Notes in Computer Science, 2406), http://link.springer.de/link/service/series/0558/tocs/t2406.htm#toc2406
• Michael Kluck, Thomas Mandl, Christa Womser-Hacker: Cross-Language Evaluation Forum (CLEF) : Europäische Initiative zur Bewertung sprachübergreifender Retrievalverfahren. In: Information - Wissenschaft und Praxis (nfd), Jg. 53, 2002, Nr. 2, S. 82-89
• Kluck, Michael; Gey, Frederic C. (2000): The Domain-Specific Task of CLEF: Specific Evaluation Strategies in Cross-Language Information Retrieval. URL: http://www.educat.hu-berlin.de/~kluck/domain-specific.pdf 

• Martin Braschler, Donna Harman, Michael Hess, Michael Kluck, Carol Peters, Peter Schäuble: The Evaluation of Systems for Cross-Language Information Retrieval, in: Maria Gavrilidou et al. (eds.): Second International Conference on Language Resources and Evaluation, Proceedings, Athens, Greece, 31 May - 2 June 2000. Athen: ELRA 2000, p. 1469-1474; (Preprint: http://www.educat.hu-berlin.de/~kluck/lrec2000-paper-70.pdf)

• Peters, Carol (ed.): Cross-Language Information Retrieval and Evaluation, Workshop of the Cross-Language Evaluation Forum, CLEF 2000, Lisbon, Portugal, September 2000, Revised Papers. Berlin et al.: Springer 2001 (= Lecture Notes in Computer Science, LNCS 2069)

• Michael Kluck/Frederic C. Gey: The Domain-Specific Task of CLEF - Specific Evaluation Strategies in Cross-Language Information Retrieval. In: Peters, Carol (ed.): Cross-Language Information Retrieval and Evaluation, Berlin et al.: Springer 2001, S. 48-56 (Preprint: http://www.educat.hu-berlin.de/~kluck/domain-specific.pdf Copyright Springer-Verlag)

• Donna Harman, Martin Braschler, Michael Hess, Michael Kluck, Carol Peters, Peter Schäuble, Páraic Sheridan: CLIR Evaluation at TREC. In: Peters, Carol (ed.): Cross-Language Information Retrieval and Evaluation, Berlin et al.: Springer 2001, S. 7-23 

Further information: Project-Homepage CLEF