Epic: docsite RAG

[josephjclark]

This is an epic issue to track requirements for a good, trustworthy RAG system which we can plug into the AI assistant

We have a great prototype - it just needs bringing into line with our latest ideas about the server, testing, and integrating with job chat

• Port the existing generate_docs_embeddings.py to use the new embeddings framework
  • Use pinecone
  • keep openai for embeddings I guess
• Port the existing search.py function into a docs_search function, which uses a store in embeddings/
• analyse and optmise the chunking of the docsite and adaptor APIs, so that the documents we embed are reasonably useful
• setup github automation to automatically rebuild the embeddings database when the docsite is updated
• Integrate with job chat and re-work the old job chat prompt
• Include metadata for each document returned, in such a way that we can present them to the user
• work out what tests we can add around this to prevent regressions and validate behaviour