[RMP] Provide to our customers best practices guidance for training Retrieval, Ranking and Multi-Stage RecSys models

[gabrielspmoreira]

Problem:

Merlin platform provides tools for building multi-stage recommender systems (which we are going to present at RecSys’22 conference on our tutorial and demo).
In particular, the retrieval and ranking stages are implemented in Merlin Models library and tied together during inference with Merlin Systems library.
Although we provide to our customers an implementation of retrieval and ranking models, and a few example notebooks with toy datasets to get them started, it might be hard for our customers to obtain a reasonable accuracy and performance without too much experimentation on their side with their datasets. In addition, it is expected that customers find some issues in their experiments related to our API flexibility and to models accuracy/performance with real datasets.
This scenario might reduce customers’ interest and engagement if they think Merlin has not been well experimented and refined with real datasets and is not mature enough for their purposes.

Goal:

The goal of this work is to improve customers experience when starting to experiment Merlin with their own datasets.
We want to leverage NVIDIA internal research and computational resources to perform comprehensive experimentation of our retrieval and ranking models with a diversity of public datasets, so that we can:

• Learn and share with our customers best practices on how to train Two-Stage RecSys.
• Test our implementation, scalability, performance, and models accuracy with real datasets, finding and fixing potential issues before our customers
• Refine our API to provide the flexibility a Data Scientist / ML Engineer would reasonably expect from an ML framework

Constraints:

• The deliverables of our best practices for our customers (and for our internal solution architects) will include in our documentation and examples:
  • Merlin Model Zoo Leaderboard - Comparison of the accuracy and performance of different models with different public datasets for the retrieval and ranking tasks
  • Hypertuning guidance - Identifying the most important hyperparameters to tune in such models and providing guidance in terms of the search space as a function of the dataset size (# of samples and features)
  • Advanced examples - Experimentation scripts showcasing advanced usage of our API to set the important hyperparameters.
• CI integration tests on perf and acccuracy regressions - Usage of the experimentation scripts in our CI, so that we can detect training performance or accuracy regressions.
• API improvement – It is common in the experimentation to process to find bugs or flexibility limitation in our API for some use cases or datasets. This work involves identifying the issues and fixing the critical issues.

Starting Point:

Our research team has already done or is doing experimentation work for some RecSys use cases:

• Session-based recommendation – For the research where Transformers4Rec library was designed and experimented, we provided in our RecSys paper in its online appendix:
  • The model zoo leaderboard, comparing models accuracy and performance
  • Some hypertuning guidance (search space but not the ranking of most important of the hparams)
  • Advanced example of the API
• Retrieval models – This experimentation process from our research on retrieval models allowed to identify a bunch of API, performance and accuracy issues, all of them either fixed or in progress. The preliminary experiment results are here (internal to NVIDIA), comparing retrieval models like MF (both Implicit and Merlin Models), TwoTower and YouTubeDNN for different datasets. We are going to refine the experiments after the on-going refactory being performed in some related Merlin Models building blocks. Based on the retrieval experiments script, now we have integration tests of retrieval models to monitor accuracy.
  The effort in the retrieval research is tightly related to the scope of this RMP.
• Ranking models – We have currently a research project focused in Single-Task Learning (STL), Multi-Task Learning (MTL) in ranking models (e.g. predicting the likelihood of a user clicking, liking, sharing and/or purchasing an item), a typical use case of large companies using Two-Stage recommendation with rich user feedback (like in Tenrec dataset provided by Tencent and Twitter dataset provided for RecSys challenge).
  We have been refining/fixing and experimenting with a number of ranking models (like DCN-v2, DLRM, DeepFM, Wide&Deep) and MTL ranking models (e.g. MMoE, PLE) for AliCCP and Tenrec public dataset. That research work is directly related to this RMP.
• Two-stage RecSys models training – We have started for the Hack week a research project to investigate the best practices to train two-stage recsys pipelines (proposal here - internal only), composed of Retrieval and Ranking models. In particular, we are interested to understand how to train the ranking model, so that it is able to improve the ranking of items provided by a retrieval models (e.g. by usage of additional features, real x sampled negatives, transfer learning).

Tasks

Ranking Models

• Update the research STL/MTL ranking training script to use the latest API (PredictionBlock instead of PredictionTask) #917

AliCCP dataset experiments

• Create a preprocessing notebook for AliCCP dataset based on the research preprocessing scripts #674
• Create a STL/MTL ranking training/eval script for AliCCP preprocessed dataset based on the research scripts #675
• Run hypertuning experiments for the STL ranking models: MLP, Wide&Deep, DCN-v2, DeepFM, DLRM for AliCCP dataset #676
• Run hypertuning experiments for the MTL ranking models: MMoE, PLE for AliCCP dataset #677

Tenrec dataset experiments

• Create a preprocessing notebook for Tenrec dataset based on the research preprocessing scripts #678
• Create a generic STL/MTL ranking training/eval script based on the research scripts #918
• Run hypertuning experiments for the STL ranking models: MLP, Wide&Deep, DCN-v2, DeepFM, DLRM for Tenrec dataset #680
• Run hypertuning experiments for the MTL ranking models: MMoE, PLE for Tenrec dataset #681

Testing

• Create integration tests for ranking models on CI to track accuracy or performance regressions over time #667

Documentation

• Create documentation for Quick-Start for Ranking (CLI args, best practices and tutorials) #666
• Example notebook of how to train Multi-Task Learning ranking models with Merlin Models #682
• Blog post about Multi-Task Learning ranking models with Merlin Models #683

Retrieval Models

Finish the refactory of the YouTubeDNN retrieval model
- [x] NVIDIA-Merlin/models#540 - Provide flexibility to define the targets of YouTube DNN sequential input (e.g. next-item prediction x last item prediction)
- [x] NVIDIA-Merlin/models#622 - Make YouTubeDNN and retrieval model, that exports a Top-K recommender model for evaluation and inference

• Update the existing examples to use the refactored retrieval models (V2) #684

Datasets preprocessing

• Create a preprocessing notebook for H&M dataset based on the research preprocessing scripts #685
• Create a preprocessing notebook for LastFM dataset based on the research preprocessing scripts #686
• Create a preprocessing notebook for REES46 dataset based on the research preprocessing scripts #687

Hypertuning

• Create a generic retrieval train/eval script based in the original retrieval research script with the refactored retrieval models (V2) #688
• Investigate why our implementation of MF models provides better accuracy than TwoTower models in previous hypertuning experimentation models#802
• Run the hypertuning of retrieval models (MF, Two-Tower, YouTubeDNN) with the refactored implementation (V2) for TenRec dataset models#803

Documentation

• Create documentation with best practices for training retrieval models (model zoo leaderboard and hypertuning guidance) learned during experimentation models#804

Testing

• Update the retrieval integration tests on CI to use the refactored MF and TwoTower models#805

Two-stage recommendation

• Create an advanced example showcasing the best practices to train and evaluate Two-stage recommenders #671

Investigate/research how to better train two-stage recsys pipelines

• When training ranking models, should we use explicit negative samples or negative samples sampled from the batch, or provided by the retrieval model? #669
• Can we improve the accuracy of both retrieval and ranking models by using transfer learning of embedings/shared layers? #670

[EvenOldridge]

@gabrielspmoreira @viswa-nvidia to update checkboxes and convert to issues across the releases.

[gabrielspmoreira]

fyi, I have updated the check boxes and created the tasks a few days ago

[gabrielspmoreira]

Created some slides with an overview of the proposal to discuss in the next grooming meeting

[gabrielspmoreira]

I have written a new RMP #732 as a rewrite of this one in a more customer-centric and pragmatic approach: a quick start pipeline for customers. I keep this RMP unchanged as it was prioritized, but propose closing it and taking the #732 as the replacement. The new one also reduces the scope in terms of dataset experiments, as we are going to stick initially to a single dataset (TenRec) as an example on how to use the quick-start template pipeline.

[gabrielspmoreira]

@viswa-nvidia @EvenOldridge I am closing this ticket, as I rewrote it in a customer-centric approach in #732 , which is now assigned to 23.01