Proposal: Better Integration testing Construction that supports Seeding and Environment Setting

[jhancock-taxa]

Background and Motivation

Proper integration testing that scales and actually gets run is really hard. Aspire is close but needs a few things to make sure that the Aspire Test Host becomes a good Arrange phase:

1. The ability to spin up all of the resources with random, unused ports and database names, while maintaining service discovery.
2. The ability to seed data into databases and other services for a given set of tests before anything but the external resources are spun up.
3. The ability to run a group of tests against that share a host so that you can save that overhead of spin up and seeding whenever possible to run tests in parallel.
4. The ability to orchestrate all of that to front-end interfaces (web apps etc) that now still have the right Urls for calling REST endpoints etc.

Right now, none of these things are possible, but are required for a good integration and E2E test system that can scale and be viable for execution before release to production etc.

Proposed API

Right now we have something like this:

        var appHostBuilder = await DistributedApplicationTestingBuilder.CreateAsync<Projects.AppHost>();
        var app = await appHostBuilder.BuildAsync();
        await app.StartAsync();

Instead, we're proposing something like this:

        var appHostBuilder = await DistributedApplicationTestingBuilder.CreateAsync<Projects.AppHost>(**Func<Task<IDictionary<string, string>> environmentVariables, Func<IConfiguration, Task> seed**);
        var app = await appHostBuilder.BuildAsync();
        await app.StartAsync();
        
        **var testHostBuilder = app.CreateApplicationHost();**

There are 2 main changes:

1. The ability to seed environment variables into the AppHost. This would be to do things like set the netcore_environment to "Tests" instead of development, and create others that could signal not to start PgAdmin which blows up in tests and github ci/cd right now.
2. The seed function. What this would do, is collect all of the resources and create a new IConfiguration that could be used to do any seeding of data, be that database, or otherwise because you'd now have all of the configuration information that anything had at spin up.

These 2 constitute "arrange".

3. This is optional but useful in some cases where it will generate a new IHostApplicationBuilder automatically with the identical environment and all of the Configuration injected automatically. You could then use this for other DI processes as desired like a standard Hosted Application. The use case for this is: Test, Mutate, and Test again with the mutation.

Usage Examples

1. I have to have a specific organization with a specific user with specific roles on that specific organization, request data from my micro service after authentication through our OpenIddict (OpenIDConnect) authentication service. To do this, I have to be able to seed a database with all of that scenario before using password flow to login with that seeded user, and then call REST endpoints using a Kiota Client through a Yarp Proxy as if it was production. I have 30 of these possible read scenarios for that specific set of roles for a user that I need to validate against know outcomes from that know data. These could all be run in parallel because none write thus I only need to spin up the app host once, and seed it once thus having a major time savings.

2. I have a React website that I need to run a full User Acceptance test of a flow that we support in the application to make sure that it works all of the way through the process from beginning to end from user login that loops through a Blazor accounts portal using code flow with pkce, to stepping through the UI with puppeteer in the proscribed way, which induces REST calls with the authentication token with specific outputs and I want to run 5 separate app hosts all on unique ports and script the whole thing with puppeteer or similar for each of those tests in parallel but each has a separate app host because there is data modification. The React website has to know the right URLs for the proxy that was spun up to be able to make the calls all automatically and transparently, and it needs to be able to scale to X number of tests running simultaneously on the same hardware.

Alternative Designs

The alternative is to put the arrange of data after the app host is spun up instead of as it's being spun up by just using No. 3 above after the fact to do the data injection, but still doing No. 1. The issue with this, is that the data may need to be there for the other services to spin up correctly and get in the right state.

I also considered creating a custom type of AppHost that is a class that implements an interface or similar that has more control over the entire process and doesn't share most of the code with the normal app host. While this is more flexible and could get to the same thing, I think it runs the risk of not maintaining the "perfect replica" scenario of deploy and test and thus should be avoided if at all possible.

It should be noted that Aspire itself follows Collect, arrange, and act already because you:

1. Setup your external dependencies like Redis, Postgres, etc.
2. Setup your Services with their dependencies.
3. Execute that and spin it up.

So it would be entirely doable to split the AppHost into 3 parts for testing since after No. 1 here, you would then do your seeding, and then No. 2 and then No. 3 would execute in that order.

The key here is that with the simplified interface extension proposed, it still needs to run the environment settings first, and then the seed before the projects are run. I.e. all of the resources that aren't projects need to come up, generate configuration settings and then the seed needs to run, and then the projects need to be brought up.

Risks

The risk is over ticking the plumbing. But the bigger risk is Aspire not being usable in real world scenarios for Integration testing which is a vital component of what Aspire has to offer. The alternatives right now with Web Test hosts like @davidfowl as done a WIP on, don't solve the larger problem of other clients testing E2E, and become phenomenally ugly code that is hard for developers to use.

Notes

.NET Aspire uses the web concept of Launch Profiles. But dotnet test doesn't and doesn't take a parameter for launch profiles. As a result, a lot of the environment injection you'd want to do can't be done and is ignored right now as a result. This could be improved as well so that you can specify a launch profile in the host to be used when it is spun up so that you can have an Environment named "Test" as an example.

[SteveSandersonMS]

This is a great area for us to improve. I think Aspire is really close to enabling an exceptionally good flow for E2E testing in which all the services start up seeded in a known state.

I produced a sample approach for this in eShopSupport (e.g., see https://github.com/dotnet/eShopSupport/blob/main/src/AppHost/Program.cs#L54 - apologies to people who can't yet see this - we're working on making it public). Generally that works great, though depends on the app having a consistent and formal approach to seeding all the services from datasets tracked in source control.

Perhaps in the short term we could extend the docs at https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/testing to give an example of seeding a DB to a known state as part of starting an E2E test (and ensuring the Docker containers don't use any other persistent storage while running under E2E test). Developers can extrapolate from that to similarly seeding any other storage they use in their app.

Longer term, the suggestions made by @jhancock-taxa above are great and will help us dig into further requirements.

[ReubenBond]

There are 2 main changes:

1. The ability to seed environment variables into the AppHost. This would be to do things like set the > > netcore_environment to "Tests" instead of development, and create others that could signal not to start > PgAdmin which blows up in tests and github ci/cd right now.
2. The seed function. What this would do, is collect all of the resources and create a new IConfiguration that could be used to do any seeding of data, be that database, or otherwise because you'd > now have all of the configuration information that anything had at spin up.

Change 1 is partially addressed now: you can set config for the app host via the Create & CreateAsync methods which accept a Action<DistributedApplicationOptions,HostApplicationBuilderSettings> parameter. See: https://learn.microsoft.com/en-us/dotnet/api/aspire.hosting.testing.distributedapplicationtestingbuilder.createasync?view=dotnet-aspire-8.0&devlangs=csharp&f1url=%3FappId%3DDev17IDEF1%26l%3DEN-US%26k%3Dk(Aspire.Hosting.Testing.DistributedApplicationTestingBuilder.CreateAsync)%3Bk(SolutionItemsProject)%3Bk(DevLang-csharp)%26rd%3Dtrue

There is a second feature required to complete (AFAICT) point 1, which is to allow overriding DOTNET_LAUNCH_PROFILE and to propagate env vars from there. That is covered in part by issue #5093 and is addressed by PR #7363

Change 2 I don't quite understand. Seeding is not a test-only requirement: you likely want/need it at development time, too, so it should be addressed on the core app host infrastructure. Since this issue was created, we have improved the functionality in Aspire for seeding databases (eg, via WaitFor and eventing). @davidfowl posted an example

There are some changes to make this scenario easier to implement at test time, too. Eg:

• The ability to get an already-added resource builder from the app builder Add IDistributedApplicaitonBuilder.GetResourceBuilder API #7354
• Synchronous Create APIs on DistributedApplicationTestingBuilder which let you more easily create test-scenario-specific apps (no need for a separate app host) Add APIs to DistributedApplicationTestingBuilder for testing without a separate AppHost project #7335

The above improvements will be available in v9.1.

@jhancock-taxa, do those changes address your needs?