12-06, 13:00–13:30 (UTC), General Track
How can you make sure that your recommender systems work as expected? Once you put them into production and users start interacting with the model predictions, evaluating the model output quality might become tricky. In this talk, we will explore how to monitor the quality of recommender systems in production, detect data drift, and prevent known model failure modes.
Once recommender systems are up and running, they must keep doing a good job and improve conversions and user experience. At the same time, things can get complex in production.
Users' preferences change frequently, which requires a dynamic response. Additionally, user interactions create feedback loops that complicate model evaluation. Models also suffer from specific failure modes, such as recommending the same thing too often or not providing enough novelty and diversity. On top of this, you often deal with varied data inputs, combining structured tabular data, embeddings, raw text, or images. This makes it harder to detect data quality issues and understand changes in the model environment.
In this talk, I will explore possible approaches to monitoring recommender systems in production:
- How to design an ML model monitoring framework, starting from high-level quality metrics to a granular understanding of the system behavior;
- How to combine simple heuristics and sophisticated statistical tests to evaluate different aspects of the model quality ;
- How to detect distribution drift in mixed input data and decide on the optimal model retraining approach;
- How to implement ML model monitoring architecture to spot changes in near real-time.
Throughout the talk, I'll also introduce several open-source tools and testing methods that might help evaluate the performance of recommender systems in production.
No previous knowledge expected
Emeli Dral is a Co-founder and CTO at Evidently AI, a startup developing open-source tools to evaluate, test, and monitor the performance of machine learning models.
Earlier, she co-founded an industrial AI startup and served as the Chief Data Scientist at Yandex Data Factory. She led over 50 applied ML projects for various industries - from banking to manufacturing. Emeli is a data science lecturer at GSOM SpBU and Harbour.Space University. She is a co-author of the Machine Learning and Data Analysis curriculum at Coursera with over 100,000 students.