TT-14070: [Testing/Concurrency] Test gateway lifecycle, enable start/stop parallelism

[titpetric]

User description

JIRA: https://tyktech.atlassian.net/browse/TT-14070

This adds an aggresive test against the gateway test lifecycle. It concurrently starts and stops an unbounded amount of gateway test servers and attempts to exit the test after 5 seconds with a timeout. As long as garbage collection occurs often enough, the test seems stable but still fails sporadically (races, leaks).

The test is also significantly bottlenecked by runtime.GC. If to run it once every 256 gateway cycles, the throughput is 6657 start/stops in 5 seconds, if to run it on every start stop then only 500 start/stops get done in 5 seconds.

To opt into the test:

cd tests/lifecycle
task test

The test is not run in CI, would need TEST_LIFECYCLE=1 enabled, which the taskfile does.

PR fixes a set of minor issues:

• config.WriteDefault would not work correctly for built gateway, now uses os.Getwd (working dir),
• tests require a test-environment driven TemplatePath setting (gateway/testutil)
• cleared ResetDefaultConfig in favor of StartTest practices established
• moves gateway globalMu usage into initSystem, preventing most concurrency issues over globals
• cli.Init is invoked in each test, as it holds global app state, overwrites needlessly
• apidef/oas taskfile didn't include schema updates under the lint target, updated tasks

The ideal outcome is us being able to run all tests that rely on StartTest/Gateway lifecycle in parallel.

---

PR Type

Tests, Enhancement, Bug fix

---

Description

• Introduced a new lifecycle test for gateway concurrency.

• Enhanced configuration handling and reduced global state races.

• Improved test utilities and removed deprecated methods.

• Added a Taskfile for lifecycle test execution and dependencies.

---

Changes walkthrough 📝

	Relevant files
Enhancement	5 files cli.go Added synchronization to `Init` to prevent race conditions. +11/-0    api_definition.go Refactored template loading for better modularity.              +8/-6      server.go Moved globalMu locking to initSystem for better concurrency handling. +3/-3      testutil.go Deprecated ResetTestConfig and improved test gateway initialization. +6/-5      doc.go Embedded assets into the `templates` package.                        +6/-1
Bug fix	1 files config.go Improved `WriteDefault` to use working directory for templates. +8/-8
Tests	3 files api_loader_test.go Added cleanup for test lifecycle in TestDifferentDomainsIdenticalListenPaths. +1/-0      gateway_test.go Updated `TestCustomDomain` to use `StartTest` for better isolation. +6/-7      lifecycle_test.go Added a new lifecycle test to stress gateway start/stop.  +84/-0
Dependencies	2 files go.mod Added `gopsutil` dependency for memory stats in lifecycle tests. +2/-0      go.sum Updated dependencies to include `gopsutil` and related modules. +4/-0
Configuration changes	1 files Taskfile.yml Added Taskfile for lifecycle test execution and formatting. +56/-0

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Race Condition The use of sync.Once in the Init function ensures that setup is only called once, but it is important to validate that all global state changes within setup are thread-safe and do not introduce subtle concurrency issues. var initOnce sync.Once // Init sets all flags and subcommands. // It's only run once to avoid races over the globals. // The arguments are ignored for subsequent runs. func Init(confPaths []string) { initOnce.Do(func() { setup(confPaths) }) Resource Management The TestGateway_TestLifeCycle function aggressively starts and stops gateways, which could lead to resource exhaustion. Ensure proper cleanup and resource management to avoid system instability during test execution. func TestGateway_TestLifeCycle(t *testing.T) { if ok, _ := strconv.ParseBool(os.Getenv("TEST_LIFECYCLE")); !ok { t.Skipf("Enable test with TEST_LIFECYCLE=1 (or 'true')") } ctx, cancel := context.WithTimeout(context.Background(), time.Second*5) defer cancel() var i int64 func() { for { if err := printMemStats(t); err != nil { t.Logf("Breaking out on error: %v", err) break } go func() { t1 := gateway.StartTest(nil) t.Log(t1.URL) t1.Close() }() atomic.AddInt64(&i, 1) select { case <-ctx.Done(): return default: } runtime.GC() } }() t.Logf("Got %d gateway Start/Stop's", atomic.LoadInt64(&i)) } Default Configuration Path Handling The WriteDefault function now uses os.Getwd to determine the working directory. Validate that this change does not introduce unintended behavior in environments with varying working directories. func WriteDefault(in string, conf *Config) error { wd, err := os.Getwd() if err != nil { return fmt.Errorf("Can't get working directory: %w", err) } *conf = Default conf.TemplatePath = filepath.Join(wd, "templates") if err := envconfig.Process(envPrefix, conf); err != nil { return err } if in == "" { return nil } return WriteConf(in, conf)

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Limit goroutines to prevent overload Add a mechanism to limit the number of goroutines spawned in the TestGateway_TestLifeCycle function to prevent excessive resource usage and potential system crashes. tests/lifecycle/lifecycle_test.go [45-49] +const maxGoroutines = 100 +sem := make(chan struct{}, maxGoroutines) +sem <- struct{}{} go func() { + defer func() { <-sem }() t1 := gateway.StartTest(nil) t.Log(t1.URL) t1.Close() }() Suggestion importance[1-10]: 10 __ Why: The suggestion introduces a mechanism to limit the number of goroutines spawned in the TestGateway_TestLifeCycle function. This is a critical improvement as it prevents excessive resource usage and potential system crashes, addressing a significant issue in the test's design.	High
	Handle errors during template parsing Add error handling for the ParseFiles call in loadFileTemplate to ensure that any parsing errors are logged or returned appropriately. gateway/api_definition.go [850] -return apidef.Template.New(tmpName).Funcs(a.filterSprigFuncs()).ParseFiles(in) +tmpl, err := apidef.Template.New(tmpName).Funcs(a.filterSprigFuncs()).ParseFiles(in) +if err != nil { + return nil, fmt.Errorf("failed to parse template file: %w", err) +} +return tmpl, nil Suggestion importance[1-10]: 9 __ Why: The suggestion introduces error handling for the ParseFiles call in loadFileTemplate. This is a significant improvement as it ensures that any parsing errors are appropriately handled and returned, improving the robustness of the function.	High
	Validate working directory before use Add a check to ensure that the wd variable in WriteDefault is not empty before using it to construct the TemplatePath, to avoid potential runtime errors. config/config.go [1350] +if wd == "" { + return fmt.Errorf("Working directory is empty") +} conf.TemplatePath = filepath.Join(wd, "templates") Suggestion importance[1-10]: 8 __ Why: The suggestion adds a validation check for the working directory (wd) before using it to construct the TemplatePath. This is a meaningful improvement as it prevents potential runtime errors if wd is unexpectedly empty.	Medium

[github-actions]

API Changes

--- prev.txt	2025-02-13 21:09:40.338079939 +0000
+++ current.txt	2025-02-13 21:09:36.241055024 +0000
@@ -4938,7 +4938,8 @@
 FUNCTIONS
 
 func Init(confPaths []string)
-    Init sets all flags and subcommands.
+    Init sets all flags and subcommands. It's only run once to avoid races over
+    the globals. The arguments are ignored for subsequent runs.
 
 func Parse()
     Parse parses the command-line arguments.
@@ -5099,7 +5100,7 @@
     valid config file.
 
 func WriteConf(path string, conf *Config) error
-func WriteDefault(path string, conf *Config) error
+func WriteDefault(in string, conf *Config) error
     writeDefault will set conf to the default config and write it to disk in
     path, if the path is non-empty.
 
@@ -10818,7 +10819,11 @@
     RemoveApis clean all the apis from a living gw
 
 func (s *Test) ResetTestConfig()
-    ResetTestConfig resets the config for the global gateway
+    Deprecated: ResetTestConfig resets the config for the global gateway.
+
+    The function does nothing, the correct way is to reuse StartTest(), filling
+    the config from the provided callback. Usage impacts a few tests (small).
+    See TestCustomDomain for an updated test.
 
 func (s *Test) Run(t testing.TB, testCases ...test.TestCase) (*http.Response, error)
 
@@ -12617,6 +12622,10 @@
 
 package templates // import "github.com/TykTechnologies/tyk/templates"
 
+
+VARIABLES
+
+var Assets embed.FS
 # Package: ./test
 
 package test // import "github.com/TykTechnologies/tyk/test"
@@ -12859,6 +12868,8 @@
 
 package coprocess // import "github.com/TykTechnologies/tyk/tests/coprocess"
 
+# Package: ./tests/lifecycle
+
 # Package: ./tests/policy
 
 package policy // import "github.com/TykTechnologies/tyk/tests/policy"

[sonarqubecloud]

Quality Gate failed

Failed conditions
11 Security Hotspots
E Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE