Add support for Reader API to convert HTMLs into Documents

[bilgeyucel]

Is your feature request related to a problem? Please describe.
There's no component to use Jina's Reader API with Haystack.

Describe the solution you'd like
A new JinaHTMLtoDocument (name TBD) component to use Jina's Reader API to convert URLs into Haystack Documents. This component should accept a URL and output a Haystack Document.

Describe alternatives you've considered

• This component can output a markdown file and users might use MarkdownConverter to use that component in a pipeline (not Haystack intuitive but might have advantages)
• Depending on how the Reader API works, it can accept a list of URLs and return a list of Haystack Documents

Additional context
Add any other context or screenshots about the feature request here.

Tasks

Preview Give feedback

1. The code is documented with docstrings and was merged in the main branch
2. docs: docs for JinaReaderConnector #1212
   P1 type:documentation +2
   
3. There is a Github workflow running the tests for the integration nightly and at every PR
4. The package has been released on PyPI
5. Integration tile in https://github.com/deepset-ai/haystack-integrations has been updated
6. The integration has been listed in the Inventory section of this repo README
7. There is an example available to demonstrate the feature
8. The feature was announced through social media

Tasks

Preview Give feedback

No tasks being tracked yet.

[anakin87]

I was the one who proposed this component.
Unfortunately, I tried the service and it is quite unstable at the moment.

[jlonge4]

Hey there @bilgeyucel @anakin87!

Is this still a thing? I just toyed around with the API and got good results. Would be happy to knock this out if you guys think its valuable.

[anakin87]

@jlonge4 I think the API improved over time.

I see they now have different endpoints for converting a page into markdown, searching the web and grounding (experimental).
What's your idea?

[jlonge4]

@anakin87 I think it's pretty cool. Do you think the existing LinkContentFetcher/Web Search components have too much overlap in functionality with it?

[anakin87]

I would say it is just another nice option.

Are you thinking of a single component or more than one?

[jlonge4]

@anakin87 I agree! Would passing modes at init time to a single component make sense?
Like reader = JinaReader(mode="read") or something to designate which endpoint to use.

[anakin87]

I'm thinking of something like:

@component
class JinaReader():

    def __init__(
        self,
        api_key: Secret = Secret.from_env_var("JINA_API_KEY"),
        mode: Union[Mode, str],
        ...
    ):
    ...

    @component.output_types(document=Document)
    def run(self, input:str):

    # check input depending on mode
    ...

Mode can be an Enum like this (with a convenient from_str method):

haystack-core-integrations/integrations/nvidia/src/haystack_integrations/components/embedders/nvidia/truncate.py

Line 4 in ac0e4c2

class EmbeddingTruncateMode(Enum):

[bilgeyucel]

@anakin87 @jlonge4, what are the exact features of Jina Reader API now? I'm asking because we use reader components for extractive QA tasks, and I don't think the JinaReader component will fit well into that category. Does it make sense to name it JinaReaderConverter, maybe?

[anakin87]

• Convert URL into Markdown
• Search the web and convert results to Markdown
• Ground a statement with web knowledge (only paid, haven't tried)

https://jina.ai/reader/

[jlonge4]

@anakin87 looks great, I'll get it cooking asap!
@bilgeyucel you have a great point, it definitely is more of a converter or fetcher vs a reader.

[anakin87]

@jlonge4 I have added a tasklist to #663 (comment).

Could you maybe help with opening a PR to mention the JinaReaderConnector in https://github.com/deepset-ai/haystack-integrations/blob/main/integrations/jina.md?
(I see the focus is on embedding models, so maybe a brief mention + link to examples is OK)

[jlonge4]

@anakin87 you've got it, no problem 😎

[jlonge4]

@anakin87 deepset-ai/haystack-integrations#288

[anakin87]

Closing this issue.

(Only social media announcement is missing.)
Added an item for this component to Weekly Announcements - https://github.com/deepset-ai/devrel-board/issues/533