The Living Labs for IR Evaluation (LL4IR) is a new evaluation paradigm. I implemented an API for participants (researchers) and sites (search engines) that take part in this Living Lab (which is also run as a CLEF lab). The API allows participants (researchers) to evaluate their ranking systems on real users of real sites (search engines). On the flip site, it allows sites (search engines) to benefit from the knowledge of the research community.
The LL4IR API can be used by researchers to perform several actions such as obtaining queries, documents and feedback and to update runs. The API is RESTful, that is, everything is implemented as HTTP request, and we use the request types GET, PUT and DELETE.
Several of my publications relate to Living Labs:
Continuous evaluation of large-scale information access systems: a case for living labs Book Chapter
In: Tutorial at ECIR, Springer, 2016.
In: CLEF 2015, Springer, 2015.
In: CLEF 2015, CEUR, 2015.
Head First: Living Labs for Ad-hoc Search Evaluation Inproceedings
In: CIKM'14, 2014.
Lerot: an Online Learning to Rank Framework
Lerot is a framework, designed to run experiments on online learning to rank methods for information retrieval. It has mainly been developed by Katja Hofmann and Anne Schuth.
The source code of Lerot is available from bitbucket.
A paper describing Lerot is published in the Living Labs Workshop at CIKM’13:
Lerot has been used to produce results in numerous publications, including these:
Probabilistic Multileave Gradient Descent Inproceedings
In: Proceedings of ECIR, Springer, 2016.
In: Proceedings of WSDM, ACM, 2016.
Probabilistic Multileave for Online Retrieval Evaluation Inproceedings
In: Proceedings of SIGIR, 2015.
In: ACM Transactions on Information Systems, vol. accepted, 2015.
Multileaved Comparisons for Fast Online Evaluation Inproceedings
In: CIKM'14, 2014.
Evaluating Intuitiveness of Vertical-Aware Click Models Inproceedings
In: Proceedings of SIGIR, 2014.
In: 36th European Conference on Information Retrieval (ECIR’14), 2014.
In: 36th European Conference on Information Retrieval (ECIR’14), pp. 75-87, 2014.
Evaluating Aggregated Search Using Interleaving Inproceedings
In: Proceedings of the International Conference on Information and Knowledge Management, 2013.
In: Proceeding of the sixth ACM international conference on Web Search and Data Mining, 2013.