- Authors: Max Howell, Daniel Dunbar, Mattt Thompson
- Status: Accepted (Bug)
- Review Manager: Rick Ballard
Testing is an essential part of modern software development. Tight integration of testing into the Swift Package Manager will help ensure a stable and reliable packaging ecosystem.
We propose to extend our conventional package directory layout to accomodate test modules. Any subdirectory of the package root directory named "Tests" or any subdirectory of an existing module directory named "Tests" will comprise a test module. For example:
Package
├── Sources
│ └── Foo
│ └──Foo.swift
└── Tests
└── Foo
└── Test.swift
Or:
Package
└── Sources
├── Foo.swift
└── Tests
└── Test.swift
Or, for projects with a single module:
Package
├── Sources
│ └── Foo.swift
└── Tests
└── TestFoo.swift
The filename: TestFoo.swift is arbitrary.
In the examples above
a test case is created for the module Foo
based on the sources in the relevant subdirectories.
A test-module is created per subdirectory of Tests, so:
Package
├── Sources
│ └── Foo
│ └──Foo.swift
└── Tests
└── Foo
└── Test.swift
└── Bar
└── Test.swift
Would create two test-modules. The modules in this example may test different aspects of the module Foo, it is entirely up to the package author.
Additionally, we propose that building a module also builds that module's corresponding tests. Although this would result in slightly increased build times, we believe that tests are important enough to justify this (one might even consider slow building tests to be a code smell). We would prefer to go even further by executing the tests each time a module is built as well, but we understand that this would impede debug cycles.
As an exception, when packages are built in release mode we will not build tests because for release builds we should not enable testability. However, considering the need for release-mode testing this will be a future direction.
This feature is controversial; many are worried always building tests will slow debug cycles. We understand these concerns but think knowing quickly tests have been broken by code changes is important. However we will provide an command line option to not build tests and will be open to reversing this decision in future should it be proven unwise.
We propose the following syntax to execute all tests (though not the tests of any dependent packages):
$ swift test
The command line should accept the names of specific test cases to run:
swift test TestModule.FooTestCase
Or specific tests:
swift test TestModule.FooTestCase.test1
In the future we would like to support running specific kinds of tests:
swift test --kind=performance
swift test would forward any other arguments to the underlying testing framework and it
would decide how to interpret them.
Sometimes test sources cannot compile and fixing them is not the most pressing priority. Thus it will be possible to skip building tests with an additional flag:
swift build --without-tests
It is desirable to sometimes specify to only build specific tests, the
command line for this will fall out of future work that allows specification
of targets that swift build should specifically build in isolation.
Users should be able to run the tests of all their dependencies. This is not the default behavior, but a flag will be provided.
Executing a test from the terminal will produce user-readable output. This should incorporate colorization and other formatting similar to other testing tools to indicate the success and failure of different tests. For example:
$ swift test --output module
Running tests for PackageX (x/100)
.........x.....x...................
Completed
Elapsed time: 0.2s
98 Success
2 Failure
1 Warning
FAILURE: Tests/TestsA.swift:24 testFoo()
XCTAssertTrue expected true, got false
FAILURE: Tests/TestsB.swift:10 testBar()
XCTAssertEqual
WARNING: Tests/TestsC.swift:1
"Some Warning"
An additional option may be passed to the testing command to output JUnit-style XML or other formats that can be integrated with continuous integration (CI) and other systems.
Running swift test will firstly trigger a build. We feel this
is the most expected result considering tests must be built before
they can be run and almost all other tools build before running
tests. However we will provide a flag to not build first.
There is already a mechanism to specify test-only dependencies.
It is very basic, but a new proposal should be made for more
advanced Package.swift functionality.
This proposal also does not cover the need for utility code, ie. a module that is built for tests to consume that is provided as part of a package and is not desired to be an external package. This is something we would like to add as part of a future proposal.
This proposal does not allow a test-module to have module dependencies
from its own package, and thus there is no provided mechanism to
specify or configure tests in the Package.swift file.
This will be added, but as part of a broader proposal for the future
of the Package.swift file.
Testing is important and it is important to make the barrier to testing
as minimal as possible. Thus, by analyzing the names of test targets,
we will automatically determine the most likely dependency of that test
and accommodate accordingly.
For example,
a test for "Foo" will depend on compilation of the library target Foo.
Any additional dependencies or dependencies that could not be automatically determined
would need to be specified in a package manifest separately.
Although tests built in debug configuration are generally run against modules also build in debug configuration, it is sometimes necessary to specify the build configuration for tests separately. It is also sometimes necessary to explicitly specify this information for every build, such as when building in a release configuration to execute performance tests. We would like to eventually support these use cases, however this will not be present in the initial implementation of this feature.
Swift can build modules with "testability",
which allows tests to access entities with internal access control.
Because it would be tedious for users to specify this requirement for tests,
we intend to build debug builds with testability by default.
It is desirable that modules that are built for testing can identify this fact in their sources. Thus at a future time we will provide a define.
Initially,
the Swift Package Manager will use XCTest as its underlying test framework.
However, testing is an evolving art form, so we'd like to support other approaches that might allow frameworks other than XCTest to be supported by the package manager. We expect that such an implementation would take the form of a Swift protocol that the package manager defines, which other testing frameworks can adopt.
Current releases of the package manager already exclude directories named
"Tests" from target-determination. Directories named FooTests are not
excluded, but as it stands this is a cause of compile failure, so in fact
these changes will positively impact existing code.
We considered supporting the following layout:
Package
└── Sources
│ └── Foo.swift
└── FooTests
└── Test.swift
This was considered because of the vast number of existing
Xcode projects out there that are laid out in the fashion.
However it was decided that the rules for these layouts are
inconsistent with our existing simple set. When users experience
unexpected consequences of layouts with our convention approach
it can be confusing and tricky to diagnose, so we should instead
submit another proposal in the future that allows easy
configuration of targets in Package.swift.
We considered decoupling testing from SwiftPM altogether.
However, since tests must be built and dependencies of tests must be managed complete decoupling is not possible. The coupling will be minimal, with a separate library and executable for tests, this is about as far as we think it is prudent to go.
We considered not baking in support for XCTest and only having a protocol for testing.
We would like to get testing up to speed as soon as possible. Using XCTest allows this to occur. We also think there is value in packages being able to depend on a testing framework being provided by the Swift system.
However nothing stops us eventually making the support for XCTest work via our protocol system.