Living Labs
The Living Labs for IR Evaluation (LL4IR) is a new evaluation paradigm. I implemented an API for participants (researchers) and sites (search engines) that take part in this Living Lab (which is also run as a CLEF lab). The API allows participants (researchers) to evaluate their ranking systems on real users of real sites (search engines). On the flip site, it allows sites (search engines) to benefit from the knowledge of the research community.
The LL4IR API can be used by researchers to perform several actions such as obtaining queries, documents and feedback and to update runs. The API is RESTful, that is, everything is implemented as HTTP request, and we use the request types GET, PUT and DELETE.
The source code is available from bitbucket. It has mainly been developed by Anne Schuth and Krisztian Balog.
Several of my publications relate to Living Labs:
Frank Hopfgartner; Krisztian Balog; Andreas Lommatzsch; Liadh Kelly; Benjamin Kille; Anne Schuth; Martha Larson
Continuous evaluation of large-scale information access systems: a case for living labs Book Chapter
In: 2019.
@inbook{Hopfgartner2019,
title = {Continuous evaluation of large-scale information access systems: a case for living labs},
author = {Frank Hopfgartner and Krisztian Balog and Andreas Lommatzsch and Liadh Kelly and Benjamin Kille and Anne Schuth and Martha Larson},
year = {2019},
date = {2019-01-01},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Anne Schuth; Krisztian Balog
Living Labs for Online Evaluation: From Theory to Practice Inproceedings
In: Tutorial at ECIR, Springer, 2016.
@inproceedings{Schuth2016LiLa,
title = {Living Labs for Online Evaluation: From Theory to Practice},
author = {Anne Schuth and Krisztian Balog},
url = {https://www.anneschuth.nl/wp-content/uploads/ecir2016-ll-tutorial.pdf},
year = {2016},
date = {2016-03-20},
booktitle = {Tutorial at ECIR},
publisher = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Anne Schuth; Krisztian Balog; Liadh Kelly
Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015 Inproceedings
In: CLEF 2015, Springer, 2015.
@inproceedings{schuth_2015_overview,
title = {Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015},
author = {Anne Schuth and Krisztian Balog and Liadh Kelly},
url = {https://www.anneschuth.nl/wp-content/uploads/2015/06/clef2015-ll4ir.pdf},
year = {2015},
date = {2015-09-08},
booktitle = {CLEF 2015},
publisher = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Anne Schuth; Krisztian Balog; Liadh Kelly
Extended Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015 Inproceedings
In: CLEF 2015, CEUR, 2015.
@inproceedings{schuth_2015_extended,
title = {Extended Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015},
author = {Anne Schuth and Krisztian Balog and Liadh Kelly},
url = {https://www.anneschuth.nl/wp-content/uploads/2015/07/clef2015-ll4ir-ceur.pdf},
year = {2015},
date = {2015-09-08},
booktitle = {CLEF 2015},
publisher = {CEUR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Krisztian Balog; Liadh Kelly; Anne Schuth
Head First: Living Labs for Ad-hoc Search Evaluation Inproceedings
In: CIKM'14, 2014.
@inproceedings{Balog2014,
title = {Head First: Living Labs for Ad-hoc Search Evaluation},
author = {Krisztian Balog and Liadh Kelly and Anne Schuth},
url = {https://www.anneschuth.nl/wp-content/uploads/2014/08/cikm2014-lleval.pdf},
year = {2014},
date = {2014-11-03},
booktitle = {CIKM'14},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Lerot: an Online Learning to Rank Framework
Lerot is a framework, designed to run experiments on online learning to rank methods for information retrieval. It has mainly been developed by Katja Hofmann and Anne Schuth.
The source code of Lerot is available from bitbucket.
A paper describing Lerot is published in the Living Labs Workshop at CIKM’13:
Lerot has been used to produce results in numerous publications, including these:
Harrie Oosterhuis; Anne Schuth; Maarten de Rijke
Probabilistic Multileave Gradient Descent Inproceedings
In: Proceedings of ECIR, Springer, 2016.
@inproceedings{Oosterhuis2016,
title = {Probabilistic Multileave Gradient Descent},
author = {Harrie Oosterhuis and Anne Schuth and Maarten de Rijke },
url = {https://www.anneschuth.nl/wp-content/uploads/ecir2016-prob-multileave-gradient-descent.pdf},
year = {2016},
date = {2016-03-20},
booktitle = {Proceedings of ECIR},
publisher = {Springer},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Anne Schuth; Harrie Oosterhuis; Shimon Whiteson; Maarten de Rijke
Multileave Gradient Descent for Fast Online Learning to Rank Inproceedings
In: Proceedings of WSDM, ACM, 2016.
@inproceedings{Schuth2016Multileave,
title = {Multileave Gradient Descent for Fast Online Learning to Rank},
author = {Anne Schuth and Harrie Oosterhuis and Shimon Whiteson and Maarten de Rijke },
url = {https://www.anneschuth.nl/wp-content/uploads/wsdm2016-multileave-gradient-descent.pdf},
year = {2016},
date = {2016-02-22},
booktitle = {Proceedings of WSDM},
publisher = {ACM},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Anne Schuth; Robert-Jan Bruintjes; Fritjof Büttner; Joost van Doorn; Carla Groenland; Harrie Oosterhuis; Cong-Nguyen Tran; Bas Veeling; Jos van der Velde; Roger Wechsler; David Woudenberg; Maarten de Rijke
Probabilistic Multileave for Online Retrieval Evaluation Inproceedings
In: Proceedings of SIGIR, 2015.
@inproceedings{schuth_2015_probabilistic,
title = {Probabilistic Multileave for Online Retrieval Evaluation},
author = {Anne Schuth and Robert-Jan Bruintjes and Fritjof Büttner and Joost van Doorn and Carla Groenland and Harrie Oosterhuis and Cong-Nguyen Tran and Bas Veeling and Jos van der Velde and Roger Wechsler and David Woudenberg and Maarten de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2015/05/schuth-probabilistic-2015.pdf
https://www.anneschuth.nl/wp-content/uploads/20150809-SIGIR-multileave.pdf},
year = {2015},
date = {2015-08-09},
booktitle = {Proceedings of SIGIR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Aleksander Chuklin; Anne Schuth; Ke Zhou; Maarten de Rijke
A comparative analysis of interleaving methods for aggregated search Journal Article
In: ACM Transactions on Information Systems, vol. accepted, 2015.
@article{Chuklin2015,
title = {A comparative analysis of interleaving methods for aggregated search},
author = {Aleksander Chuklin and Anne Schuth and Ke Zhou and Maarten de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2015/02/tois2015-interleaving-for-aggregated-search.pdf},
year = {2015},
date = {2015-01-01},
journal = {ACM Transactions on Information Systems},
volume = {accepted},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
A. Schuth, F. Sietsma, S. Whiteson, D. Lefortier M. de Rijke
Multileaved Comparisons for Fast Online Evaluation Inproceedings
In: CIKM'14, 2014.
@inproceedings{schuth2014multileaved,
title = {Multileaved Comparisons for Fast Online Evaluation},
author = {A. Schuth, F. Sietsma, S. Whiteson, D. Lefortier M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2014/08/ir1190-schuth-cikm2014-multileave.pdf},
year = {2014},
date = {2014-11-03},
booktitle = {CIKM'14},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
A. Chuklin, K. Zhou, A. Schuth, F. Sietsma, M. de Rijke
Evaluating Intuitiveness of Vertical-Aware Click Models Inproceedings
In: Proceedings of SIGIR, 2014.
@inproceedings{Chuklin2014,
title = {Evaluating Intuitiveness of Vertical-Aware Click Models},
author = {A. Chuklin, K. Zhou, A. Schuth, F. Sietsma, M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2014/05/Chuklin-et-al.-2014-Evaluating-Intuitiveness-of-Vertical-Aware-Click-Models.pdf},
year = {2014},
date = {2014-07-07},
booktitle = {Proceedings of SIGIR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
K. Hofmann; A. Schuth; A. Bellogin; M. de Rijke
Effects of Position Bias on Click-Based Recommender Evaluation Inproceedings
In: 36th European Conference on Information Retrieval (ECIR’14), 2014.
@inproceedings{Hofmann2014,
title = {Effects of Position Bias on Click-Based Recommender Evaluation},
author = { K. Hofmann and A. Schuth and A. Bellogin and M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2014/01/ecir2014-sp-interleaving.pdf},
year = {2014},
date = {2014-01-01},
booktitle = {36th European Conference on Information Retrieval (ECIR’14)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
A. Schuth; F. Sietsma; S. Whiteson; M. de Rijke
Optimizing Base Rankers Using Clicks: A Case Study using BM25 Inproceedings
In: 36th European Conference on Information Retrieval (ECIR’14), pp. 75-87, 2014.
@inproceedings{Schuth2014,
title = {Optimizing Base Rankers Using Clicks: A Case Study using BM25},
author = { A. Schuth and F. Sietsma and S. Whiteson and M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2014/01/ecir2014-schuth-bm25.pdf},
year = {2014},
date = {2014-01-01},
booktitle = {36th European Conference on Information Retrieval (ECIR’14)},
pages = {75-87},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
A. Chuklin, A. Schuth, K. Hofmann, P. Serdyukov, M. de Rijke
Evaluating Aggregated Search Using Interleaving Inproceedings
In: Proceedings of the International Conference on Information and Knowledge Management, 2013.
@inproceedings{chuklin_evaluating_2013,
title = {Evaluating Aggregated Search Using Interleaving},
author = {A. Chuklin, A. Schuth, K. Hofmann, P. Serdyukov, M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2013/09/ir0656-chuklin1.pdf},
year = {2013},
date = {2013-10-01},
booktitle = {Proceedings of the International Conference on Information and Knowledge Management},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
K. Hofmann, A. Schuth, S. A. Whiteson, M. de Rijke
Reusing Historical Interaction Data for Faster Online Learning to Rank for IR Inproceedings
In: Proceeding of the sixth ACM international conference on Web Search and Data Mining, 2013.
@inproceedings{hofmann_2013_reusing,
title = {Reusing Historical Interaction Data for Faster Online Learning to Rank for IR},
author = {K. Hofmann, A. Schuth, S. A. Whiteson, M. de Rijke},
url = {https://www.anneschuth.nl/wp-content/uploads/2012/12/wsdm-2013-learning.pdf},
year = {2013},
date = {2013-02-04},
booktitle = {Proceeding of the sixth ACM international conference on Web Search and Data Mining},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}