Learning to rank (LTR) has been considered the next generation tool to improve relevance of product search solutions. Inspired by its popularity (Buzzword 2017) and by the challenges we have at GetYourGuide, a global marketplace for tours and activities, we started a project to introduce LTR in our Search Engine. In this talk, we would like to share our Logbook from day 1 to the current project status. It covers challenges, lessons learned, good practices and pain points you need to know when navigating on  LTR tides, such as:

• Pick the right tools to collect data, train and run ML models on your Search Engine
• Distribute and scale training and test into Spark to move faster
• Know your domain (what is relevance for the business) and the diversity of real user queries
• Challenge the quality of your training set and how you collect judgments after each iteration
• Design A/B experiments and define metrics to evaluate the quality of your models

Improving ranking is always an ongoing project. Each of previous topics brought, and they are still bringing, us many good learnings. I would like to share them on this talk and bring good practical contributions to new adopters of LTR for ranking on product (enterprise) search.