Wishlist: see if we can use a cheaper model to generate adequate summaries

[scottleibrand]

Below is a screenshot from https://beta.openai.com/account/usage for the two script runs used to generate the https://github.com/scottleibrand/gpt-summarizer/tree/main/examples. At $0.02 per 10k tokens, that totals to $0.76 worth of usage to summarize those two documents. That's worthwhile, if they're documents I'd have had to read otherwise: if my free time is worth $20/hr, that means the summaries would only need to save me at least 1-2 minutes of reading time per document to be worth doing.

That said, it would be better if we could figure out how to generate adequate summaries with a smaller model. I tried directly using text-curie-001, but it generated empty summaries with the current prompts: when I ran the same prompts in the playground, it said that is because the generation started with a stop token. Removing the reiterated instructions from the prompt allowed it to generate a brief summary, but not a very good one: it seemed to only briefly be summarizing the last paragraph.

To make it work better with curie or other models, we might need to change the chunking to summarize much smaller chunks, and possibly add another layer of summarization to deal with larger documents. This might warrant a refactor of the main code block.

If anyone is interested in taking this on, please comment below.