DesignGoals

Autotest Design Goals

• Open source - share tests and problem reproductions
• Make it simple to write control files, and yet a powerful language
• Make it simple to write tests and profilers - encourage people to contribute
• Client is standalone, or will plug into any server/scheduler harness
  • Some people just have single machine, want simple set up.
  • Corporations each have their own scheduling harness, difficult to change
  • Very little interaction is needed, simple API
  • Simple handoff from full automated mesh to individual developer
• Maintainable
  • Written in one language, that is powerful, and yet easy to maintain
  • Infrastructure that is easily understandable, and allows wide variety of people to contribute
• Modular - the basic infrastructure is cleanly separated with well defined APIs.
  • Easy writing of new tests, profilers, bootloaders, etc
  • New non-core changes (eg new test) doesn't break other things.
  • Lower barrier to entry for new developers.
  • Distributed/scalable maintainership - code controlled by different people.
• Core is small, with few developers
  • This isn't a super-hard problem.
  • Most of the intelligence is in sub-modules (eg the tests).
• Error handling.
  • Tests that don't produce reliable results are useless in an automated world.
  • Developers don't write error checking easily - need 'encouragement'.

Modules

• Core - ties everything together

• Tests - execute each tests. many, many separate tests modules.

  eg kernbench, aim7, dbench, bonnie, etc.

• Profilers - gather information on a test whilst it's running, or before/after.

  eg readprofile, oprofile, schedstats, gcov, /proc info

• Results Analysis - Compare equivalent sets of test results. Per test / profiler.

• Kernel build - build kernels, with different patches, configs, etc.

  Will need different variations to cope with mainline, distro kernels, etc.

• Bootloader handling - Install new kernels, and reboot to them, pass parameters, etc

  eg. Grub, Lilo, yaboot, zlilo, etc

Key differences

Here are some of the key changes from previous systems I have seen / used / written:

• The job control file is a script. This allows flexibility and power.
• The client is standalone, so we can easily reproduce problems without the server.
• Code and tests are modular - you can allow loser control over tests than the core.
• Code is GPL.