Add autogenerated messages

[sorgfresser]

Adds OpenAI integration to automatically generate messages.

Right now, this only works for Immobilienscout as it requires get_expose_details to also return some kind of description and the lessor.

I simply added the functionality for OpenAI for personal use but thought others might be interested.
Certainly, there are a lot of improvements that should be done before merging.
Please point them out to me so I can edit whatever should be rewritten!

Features:

• Adds OpenAIProcessor
• Adds OpenAI yaml section and Env Vars
• Adds a new hunter (debatable whether this is necessary, I did it because I needed crawl_expose_details)
• Modifies the message format to also include application

[sorgfresser]

Aaaand the Lockfile looks broken - I'll dive into it
Edit: should be fixed.

[codders]

Hi @sorgfresser ,

Thanks for the PR. I've got to say, I have pretty mixed feelings about this. On the one hand I'm a bit worried about hastening the time when all actual content on the internet is drowned out by LLM garbage - https://marketoonist.com/2023/03/ai-written-ai-read.html . On the other hand, the market for apartments is already a trash market full of trash and flathunter exists to make that problem worse.

I'll review the code anyway, and if we get it in a good shape I'm happy to merge it.

[codders]

The code looks good - thanks for this. Some changes I would like to see from my side, and please take a look at CI to see what pyright and pylint want.

[codders]

Why do you want to create an OpenAIHunter here, instead of just adding generate_text to the filters for the default Hunter?

[sorgfresser]

Hi! Sorry it's been a while...
It's not just about the generate_text, it's mostly about crawl_expose_details which is needed for the description of an exposé and the description in turn is needed to extract the features worth mentioning in an application (at least that was my idea).
I do agree that changing the Hunter is silly and maybe the whole Hunter itself, but I did not like the idea of crawling details everytime.

[sorgfresser]

As another idea: I could introduce some additional logic in generate_text() for this. I was thinking something like

    def generate_text(self):
        """Add processor to generate text, if enabled"""
        if self.config.openai_enabled():
            if not any([isinstance(processor, CrawlExposeDetails) for processor in self.processors]):
                self.crawl_expose_details()
            self.processors.append(OpenAIProcessor(self.config))
        return self

How about this to prevent the need for a new hunter?

[codders]

So do I understand from this that you need to open a second tab to load the expose to get the details? Is there a reason to do this here instead of in the OpenAIProcessor? If we have to do it here, can we have a switch to skip this if the OpenAI feature is disabled? Otherwise everybody not using OpenAI will make a bunch of unnecessary fetches.

[sorgfresser]

Actually, the whole get_expose_details is broken for me with immobilienscout if one does not add a driver. This is due to a request initially returning "Checking that you're not a robot" and only once this check is finished the final content of the page will be returned. But I can simply cherry-pick this into a different pull request.

[codders]

This looks fine but per my other comment I think I would prefer just to change the default Hunter.

[codders]

Just wondering if it makes sense also to have a German option here - presumably a lot of people what to generate their texts in German.

[sorgfresser]

Interesting idea! I'm currently running this with a german template and it works just fine - nonetheless, I can add an option for German if you'd like me to.