Test Strategy : Automated testing

[planetf1]

We need to consider how we test beehive

• locally
• automatically as part of PR/CI process

• What framework/approach do we want to take with fairly standalone python code? Pytest? Coordinated by poetry?
• How much do we focus on isolated testing (stubbing out LLM for example) vs integrating with LLM?
• If we need LLM, what/where/configuration?
• How do we manage PR/CI verification - generally easy with github actions. Could run a small LLM within a github runner. How do bee work? Can we use same backend for llm? Needs secrets
• How do we check results from tests. With regular code this is easy - assertion of output. However with LLMs involved, so we need another LLM to interpret the results?

This was brought up in issue #125

Originally posted by @planetf1 in #125 (comment)

[psschwei]

For me, the simple first step would be to be make sure the weather example still works (i.e. run that locally after you make your code changes).

As far as testing framework, I think we should sync with @vabarbosa to use the same thing that the bee-py framework uses.

[planetf1]

See #148 for the first version of this which adds a test for crewai using poetry & pytest

I can add the base test once merged.

[planetf1]

See #153 for similar test, but for an agent defined in bee.

There is also now a github action. This runs pytest (under poetry) for the bee-hive subdirectory, since this is the top of our code structure. Unit tests could be included within this tree.

Leaving open to consolidate existing tests