Software ?>

Software

Living Labs

The Living Labs for IR Evaluation (LL4IR) is a new evaluation paradigm. I implemented an API for participants (researchers) and sites (search engines) that take part in this Living Lab (which is also run as a CLEF lab). The API allows participants (researchers) to evaluate their ranking systems on real users of real sites (search engines). On the flip site, it allows sites (search engines) to benefit from the knowledge of the research community.

The LL4IR API can be used by researchers to perform several actions such as obtaining queries, documents and feedback and to update runs. The API is RESTful, that is, everything is implemented as HTTP request, and we use the request types GET, PUT and DELETE.

The source code is available from bitbucket. It has mainly been developed by Anne Schuth and Krisztian Balog.

Several of my publications relate to Living Labs:

Anne Schuth; Krisztian Balog

Living Labs for Online Evaluation: From Theory to Practice Inproceedings

In: Tutorial at ECIR, Springer, 2016.

Links | BibTeX

Anne Schuth; Krisztian Balog; Liadh Kelly

Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015 Inproceedings

In: CLEF 2015, Springer, 2015.

Links | BibTeX

Anne Schuth; Krisztian Balog; Liadh Kelly

Extended Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015 Inproceedings

In: CLEF 2015, CEUR, 2015.

Links | BibTeX

Krisztian Balog; Liadh Kelly; Anne Schuth

Head First: Living Labs for Ad-hoc Search Evaluation Inproceedings

In: CIKM'14, 2014.

Links | BibTeX

 

Lerot: an Online Learning to Rank Framework

Lerot is a framework, designed to run experiments on online learning to rank methods for information retrieval. It has mainly been developed by Katja Hofmann and Anne Schuth.
The source code of Lerot is available from bitbucket.
A paper describing Lerot is published in the Living Labs Workshop at CIKM’13:

A. Schuth, K. Hofmann, S. Whiteson, M. de Rijke: Lerot: an Online Learning to Rank Framework. In: Living Labs for Information Retrieval Evaluation workshop at CIKM’13., 2013.

Lerot has been used to produce results in numerous publications, including these:

Harrie Oosterhuis; Anne Schuth; Maarten de Rijke

Probabilistic Multileave Gradient Descent Inproceedings

In: Proceedings of ECIR, Springer, 2016.

Links | BibTeX

Anne Schuth; Harrie Oosterhuis; Shimon Whiteson; Maarten de Rijke

Multileave Gradient Descent for Fast Online Learning to Rank Inproceedings

In: Proceedings of WSDM, ACM, 2016.

Links | BibTeX

Anne Schuth; Robert-Jan Bruintjes; Fritjof Büttner; Joost van Doorn; Carla Groenland; Harrie Oosterhuis; Cong-Nguyen Tran; Bas Veeling; Jos van der Velde; Roger Wechsler; David Woudenberg; Maarten de Rijke

Probabilistic Multileave for Online Retrieval Evaluation Inproceedings

In: Proceedings of SIGIR, 2015.

Links | BibTeX

Aleksander Chuklin; Anne Schuth; Ke Zhou; Maarten de Rijke

A comparative analysis of interleaving methods for aggregated search Journal Article

In: ACM Transactions on Information Systems, accepted , 2015.

Links | BibTeX

A. Schuth, F. Sietsma, S. Whiteson, D. Lefortier M. de Rijke

Multileaved Comparisons for Fast Online Evaluation Inproceedings

In: CIKM'14, 2014.

Links | BibTeX

A. Chuklin, K. Zhou, A. Schuth, F. Sietsma, M. de Rijke

Evaluating Intuitiveness of Vertical-Aware Click Models Inproceedings

In: Proceedings of SIGIR, 2014.

Links | BibTeX

K. Hofmann; A. Schuth; A. Bellogin; M. de Rijke

Effects of Position Bias on Click-Based Recommender Evaluation Inproceedings

In: 36th European Conference on Information Retrieval (ECIR’14), 2014.

Links | BibTeX

A. Schuth; F. Sietsma; S. Whiteson; M. de Rijke

Optimizing Base Rankers Using Clicks: A Case Study using BM25 Inproceedings

In: 36th European Conference on Information Retrieval (ECIR’14), pp. 75-87, 2014.

Links | BibTeX

A. Chuklin, A. Schuth, K. Hofmann, P. Serdyukov, M. de Rijke

Evaluating Aggregated Search Using Interleaving Inproceedings

In: Proceedings of the International Conference on Information and Knowledge Management, 2013.

Links | BibTeX

K. Hofmann, A. Schuth, S. A. Whiteson, M. de Rijke

Reusing Historical Interaction Data for Faster Online Learning to Rank for IR Inproceedings

In: Proceeding of the sixth ACM international conference on Web Search and Data Mining, 2013.

Links | BibTeX

Leave a Reply

web development