Implement Robotoff nutrition prediction route

[teolemon]

What

• Implement Robotoff nutrition prediction route
• https://openfoodfacts.github.io/robotoff/references/api/#tag/Predict/paths/~1predict~1nutrition/get
• Related smoothie issue: Offer in-line validation of Robotoff predictions for nutrition facts smooth-app#3278

[monsieurtanuki]

@teolemon @raphael0202 Not really sure what we try to achieve here, but maybe I'm mislead by my unsuccessful tests.

I get the following result with https://robotoff.openfoodfacts.org/api/v1/predict/nutrition?barcode=7300400481588&image_ids=55:

{"predictions":[],"image_ids":["55"]}

While the image 55 is something like that:

Other points:

• we don't specify language or country, which is also disconcerting
• "imageIds" images can be big and include much more that nutrition facts, which may be problematic (for the AI) in order to focus on nutrition only

[raphael0202]

The current endpoint only works well on "text" nutrition tables. I would be in favor of waiting for a proper generic model for nutrition information extraction to be developed before its integration in Smoothie.

[teolemon]

• @monsieurtanuki @raphael0202 We now have the real thing, and a better route for nutrition prediction: Allow to auto-magically extract nutrition facts from a photo smooth-app#4896

[teolemon]

@raphael0202 @monsieurtanuki We should integrate this route, which is now mature, and tested on Hunger Games.

https://robotoff.openfoodfacts.org/api/v1/insights?insight_types=nutrient_extraction&barcode={BARCODE}

[monsieurtanuki]

@teolemon @raphael0202 That feature is hard to test. The returned values are only the new/different values, and most of the time we get nothing.
One suggestion: make it possible to return all the values, and let the developer compare with what is already there in the product.

UX-wise it's not easy either, as we need to call the API (when?) and to overwrite whatever the user is typing. (how?)

[raphael0202]

One suggestion: make it possible to return all the values, and let the developer compare with what is already there in the product.

This is a valid point, what I fear if we have the current nutrient data saved in the insight is that it gets out of sync. Due to how Robotoff work, in case of massive updates, we can wait a long time before insights get refreshed.

We can add the data on the fly when querying the insight, but it would considerably slow down the request, I'm not sure it's a good strategy here.

[monsieurtanuki]

@raphael0202 Misunderstanding alert!

My assumptions were:

• robotoff works on the nutrition facts image
• robotoff manages to OCR nutrition facts from that image
• robotoff would be able to return the nutrition facts OCR'ed from that image, et voilà!

I don't assume that robotoff should be aware of the nutrition facts as current values at the product level.
Just a bland OCR like ingredients: from what it reads on the image, and considering that it's a text about ingredients, robotoff returns "ingredients" as a text, regardless of the product.

Therefore I don't understand your "out of sync" concerns. Out of sync with the nutrition facts image?
No "on the fly" refresh either.

Please tell me what's wrong with my assumptions and my analysis.

[raphael0202]

Your assumptions are right, I didn't understand correctly this message of yours:

One suggestion: make it possible to return all the values, and let the developer compare with what is already there in the product.

To make sure I understand your suggestion: by returning "all the values", you mean returning in nutrients dictionary all existing nutrient keys, even for the nutrients the model didn't detect?

The documentation about this model is here for the record: https://openfoodfacts.github.io/robotoff/references/predictions/nutrient-extraction/

[monsieurtanuki]

@raphael0202 I just want all populated values.

It's just that when I see this:

...I would expect from https://robotoff.openfoodfacts.org/api/v1/insights?insight_types=nutrient_extraction&barcode=3329770077003 something more verbose than:

{
  "count": 0,
  "insights": [],
  "status": "no_insights"
}

What triggers an OCR on your side? A new nutrition facts image?