[RMP] Provide PyTorch serving support for T4R models in Torchscript

[karlhigley]

Problem:

Users should be able to serve pytorch models that were produced with Transformers4Rec or any other process using a Systems ensemble. This will work towards supporting Session-based models as well as expanding System's support for a new modeling framework.

Goal:

Systems should be able to serve all pytorch models that are currently supported by Triton.

Definition of Done

Have an example that serves PyT session based model in conjunction with a NVT workflow where the session based models scores the whole catalog

Open questions

Constraints:

Not all pytorch models can be served via Triton's pytorch backend, so we will need to be able to use multiple backends in order to serve all Triton-compatible pytorch models.

Starting Point:

Transformers4Rec

• Update T4R to use Merlin std schema
  • Add save/load & input/output schema methods to T4Rec Model class  Transformers4Rec#507
• [Task] Transformers4Rec Serving using Merlin Schema Transformers4Rec#499
• Have a padding operator and add Support for ragged inputs that we can use consistently with Transformers for rec systems#322
• [Task] Add topK layer in model to return topK items at inference step Transformers4Rec#668

Systems

• Use Triton's pytorch backend for "torchscriptable" models for optimized performance
  • Create a PredictPytorch operator systems#153
  • Write an integration test for T4R Torchscript serving systems#176
• Load the schema files written out by T4R

Integration Issues

• Address the issues @rnyak surfaced while trying to put together an example
  • Fix missing value_count property and/or error from value_count op if sequence min and max length is same: This PR Remove specifying is_ragged in LocalExecutor _transform_data core#173 should fix that issue.
  • Fix wrong response dimension from TIS after sending request. This PR Refactoring part1- flags modification Transformers4Rec#543 should be fixing that issue.
  • Pad the rows when converting dataframes to dictionaries for Triton requests systems#234
  • [FEA] Create an easy functionality to generate dict of tensors- a standard way to move array data across frameworks core#175
  • [BUG] getting error when serving Session-based NVT workflow model on TIS with TransformWorkflow op  systems#240

Nice to have: (P1)

• [FEA] Add list column support in Merlin Systems systems#135

Documentation

• [Task] Add Example for Transformer4Rec Serving Example Transformers4Rec#491
• [BUG] Bugs to deploy NVTabular + Transformer4Rec with TorchScript backend systems#257
• Review API documentation for PyT serving support for T4R models #904
• Add/Fix the doc strings for PyT serving support for T4R models in TorchScript Transformers4Rec#692
• [DOC] Improve Code Coverage in Transformers4Rec to X% Transformers4Rec#664

Examples

• Deprecate the old example and update example for PyT serving support for T4R models with systems support and Torch script backend Transformers4Rec#660
• [Task] Update Legacy Inference Example to use Merlin Systems + TorchScript Transformers4Rec#676

Blockers:

• [INF] Unresolved architectural decisions
• Support for ragged tensors in T4R
• Start with fix padding and have it all padded to the same length and then investigate whether it's worthwhile to add the like padding to the Max Sequence length
• Padding support in dataloader that supports systems along with cross framework support

[viswa-nvidia]

@karlhigley , I saw the comment after 22.05 and wondered whether we should have this ticket defined and ready for discussion at the next grooming meeting. Can you help in definition ?

[viswa-nvidia]

@karlhigley , do you see this getting over by 22.08?

[viswa-nvidia]

@karlhigley , should this ticket also have these additional requirements that we discussed in the backlog grooming meeting?

• enable PyT T4R
  • Multi-gpu training
    - DP (working)
  • Serving
    - Python-backend (working)

[karlhigley]

No, because they're not part of the scope as described at the top of this issue and they've already been completed

[viswa-nvidia]

@karlhigley , the original ask from product team is Session based recommender - Add PyT API to Merlin models / Systems. This ticket only covers the systems side and not models. I think this should be a [Task] that must be part of a RMP ticket that will cover the models side also. Let me know your thoughts on this.

[karlhigley]

@viswa-nvidia That was my original thought too, until I got talked out of it. 🤷🏻

[EvenOldridge]

@edknv does the work you've done for Transformers4Rec support this functionality from the model training side?
@karlhigley excluding onnx support can we currently serve Torchscript based models in systems?
Looking to close this

[karlhigley]

Each side of this equation works independently, but no one has tested them together AFAIK, so we're not ready to close this yet

[EvenOldridge]

@rnyak @bschifferer can one of you please test this functionality and ensure that we're able to save models via torchscript and serve via systems. I think this is relevant to a number of your customers.

[karlhigley]

I'm working on an integration test for that here, but it isn't working yet: NVIDIA-Merlin/systems#176

[viswa-nvidia]

@marcromeyn @oliverholworthy please provide more details for the blockers

[viswa-nvidia]

@marcromeyn , Add tickets to the backlog for the optimizations