Automated Unit Testing

[alex-straw]

Hi all,

As this repository grows, I expect that automated testing would be useful to ensure changes are fit for purpose and don't break existing work. This should also make the PR process easier.

I was thinking of implementing a mock API agent (which could be selected with some debug OpenAI API key) so that we can unit test large parts of the repository without incurring the associated API costs 💵.

Is there an appetite for this yet? If so, do people have any preferences regarding testing frameworks: pytest, unittest?

Related discussions (but pertaining more so to integration testing): The cost of tokens

[daveshap]

We will need this eventually so the swarm can test itself and integrate. Maybe most useful over in the tools discussion

[alex-straw]

Swapped the category to Tool Making

[OWigginsHay]

I agree, was thinking about this yesterday. I approved a PR which broke code due to hard coded string paths "my code oops" and someone had done the repo restructure. Having tests to make sure functionality like that doesn't break will be key moving on from here

[RomanGoEmpire]

My proposal for the first test is to verify that each agent in agents follows a certain folder structure that is defined.

An example would be:

• Agent
  • documentation
  • files
  • code
  • tests
• Readme.md
• requirements.txt
• demo.py (can be one or multiple files that can be executed to run the agent as one.

If a pull request fails this test it should be not able to be merged while the test failes.
How is it done in Github. Via pipeline?

[FireMMDC]

https://github.com/daveshap/OpenAI_Agent_Swarm/actions I believe are the built in pipelines for github.
https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration looks like they are free for public repos.

[alex-straw]

@RomanGoEmpire, @OWigginsHay I'm going to take a look this weekend