Trunk file sent to LLM pipeline if it surpass a given number of tokens

[wilderlopes]

The Problem

LLMs have limited context window. The file generated by miniogre to prompt LLMs needs to comply with that.
So far, there is no automation for that, and as a result, the README generation breaks if the context is larger than the LLM's context window.

Proposed solution

Currently, we have a system that is able to count the tokens (approximately) of the file sent to the LLM pipeline. We should use it to trunk the file and only send the tokens a given LLM can support.