[Spaces] Moved tests to agnostic setup #200606

elena-shostak · 2024-11-18T16:17:11Z

Summary

Moved space tests to deployment agnostic setup.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Closes: #194584

elena-shostak · 2024-11-18T16:17:19Z

/ci

elena-shostak · 2024-11-18T16:43:45Z

/ci

elena-shostak · 2024-11-18T17:46:29Z

/ci

elena-shostak · 2024-11-18T19:18:48Z

/ci

elena-shostak · 2024-11-18T20:07:24Z

/ci

elena-shostak · 2024-11-18T22:54:58Z

/ci

elena-shostak · 2024-11-19T07:59:42Z

/ci

elena-shostak · 2024-11-19T08:23:04Z

/ci

elena-shostak · 2024-11-19T12:06:54Z

/ci

elena-shostak · 2024-11-19T12:37:46Z

/ci

elena-shostak · 2024-11-19T15:01:45Z

/ci

kibanamachine · 2024-12-27T13:29:10Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7634

[❌] x-pack/test/spaces_api_integration/deployment_agnostic/stateful.config_trial.ts: 0/10 tests passed.

see run history

kibanamachine · 2024-12-27T17:36:41Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7635

[❌] x-pack/test/spaces_api_integration/deployment_agnostic/stateful.config_trial.ts: 0/1 tests passed.
[❌] x-pack/test/spaces_api_integration/deployment_agnostic/stateful.copy_to_space.config_basic.ts: 0/1 tests passed.
[❌] x-pack/test/spaces_api_integration/deployment_agnostic/stateful.copy_to_space.config_trial.ts: 0/1 tests passed.
[❌] x-pack/test/spaces_api_integration/deployment_agnostic/spaces_only/config.ts: 0/1 tests passed.
[❌] x-pack/test/spaces_api_integration/security_and_spaces/config_basic.ts: 0/1 tests passed.
[❌] x-pack/test/spaces_api_integration/security_and_spaces/config_trial.ts: 0/1 tests passed.

see run history

elena-shostak · 2024-12-27T17:56:29Z

/ci

elena-shostak · 2024-12-27T20:41:35Z

/сi

elena-shostak · 2025-01-03T14:01:15Z

x-pack/test/spaces_api_integration/deployment_agnostic/services/role_scoped_supertest.ts

+    ) {
+      const license = config.get('esTestCluster.license');
+
+      if (!user || license !== 'trial') {


Important
We have a shared test suite for basic/trial licenses, which includes tests for both authorized and unauthorized users. Decoupling these tests is a complex and time-consuming task that requires careful consideration to avoid ending up with a significant amount of duplicated test code for different licenses.

Our custom service for RoleScopedSupertest takes it into account license and user passed/not passed to adjust needed supertest accordingly.

elena-shostak · 2025-01-03T14:04:18Z

x-pack/test/spaces_api_integration/common/fixtures/kbn_archiver/default_space.json

@@ -139,7 +139,7 @@
  "id": "conflict_2_default",
  "originId": "conflict_2",
  "references": [],
-  "type": "sharedtype",


Note
sharedtype changed to event-annotation-group SO
isolatedtype changed to url SO

I failed to change sharedtype to index-pattern SO type since I was getting a lot of missing references errors, probably it was due to the fact that index-pattern was migrated to the multiple namespace from 8.0.0

elena-shostak · 2025-01-03T15:05:19Z

src/dev/eslint/types.eslint.config.template.cjs

@@ -34,5 +34,11 @@ module.exports = {
        '@typescript-eslint/no-floating-promises': 'error',
      },
    },
+    {
+      files: ['*spaces_api_integration/common/services/basic_auth_supertest.ts'],


agent.auth(this.user.username!, this.user.password!);

Needed to suppress the eslint error for this line in our custom test service, the promise is awaited in the tests itself and adding a catch block breaks some of the test cases

eslint-disable line gets removed because this rule runs dynamically

x-pack/test/spaces_api_integration/common/services/test_data_loader.ts

elena-shostak · 2025-01-07T13:55:44Z

x-pack/test/spaces_api_integration/common/services/test_data_loader.ts

+            .join(', ')}`
+        );
+
+        // _update_objects_spaces route is internal in serverless and public in stateful


Note
Decided to not add complexity to the custom roleScopedSupertest to accommodate this use case, so made it in place.

elena-shostak · 2025-01-07T13:58:29Z

x-pack/test/spaces_api_integration/deployment_agnostic/security_and_spaces/apis/get_all.ts

-          includeAuthorizedPurposes: {
-            statusCode: 403,
-            response: expectRbacForbidden,
+      if (!isServerless) {


Note
There are no machine_learning_admin, machine_learning_user and monitoring_user roles in serverless.
We could rather move it to it's own test only for stateful or define custom roles, would like to have suggestions on this one 🙂

elasticmachine · 2025-01-07T14:00:36Z

Pinging @elastic/kibana-security (Team:Security)

elena-shostak · 2025-01-07T14:03:16Z

@elasticmachine merge upstream

jbudz

.buildkite and src/dev/eslint/types.eslint.config.template.cjs LGTM

dmlemeshko · 2025-01-10T08:47:32Z

packages/kbn-es/src/stateful_resources/roles.yml

@@ -128,3 +128,65 @@ system_indices_superuser:
      privileges: ['*']
      resources: ['*']
  run_as: ['*']
+
+machine_learning_admin:


roles.yml is expected to list only default Cloud roles, we keep consistency with Cloud (both MKI and ESS) because we can't provision roles from this file to Cloud deployment. In order to make things work, we introduced custom roles support in FTR:

// First set privileges for custom role await samlAuth.setCustomRole({ elasticsearch: { indices: [{ names: ['logstash-*'], privileges: ['read', 'view_index_metadata'] }], }, kibana: [ { feature: { discover: ['read'], }, spaces: ['*'], }, ], }); }); // Then you can login in browser as a user with newly defined privileges await pageObjects.svlCommonPage.loginWithCustomRole()

hmm, but in stateful they are built in already, isn't it? (screenshots are from cloud deployment)
Am I missing smth?

These roles are built-in to stateful, but not serverless. The predefined serverless roles can be found in the roles.yml files in packages/kbn-es/src/serverless_resources/project_roles, nested under solution-specific folders.

Any tests that need a role other than these built-in serveless roles, will require a custom role, and will not be able to run on MKI from the FTR (at least for the time being, as it is not yet supported).

The stateful roles file is meant to mirror the serverless roles to keep the testing methodology the same.

ok, I will move it to custom role

UPD: we cannot use our api for custom roles or at least I don't see how. All those 3 roles rely on reserved built in privileges: ml_admin, ml_user, reserved_monitoring . I cannot pass those in, since we accept only read, all, none. Going one layer down to use es api to create those roles and adding other if/else statements seems not feasible.

I've moved those 3 tests to only stateful for now.

dmlemeshko · 2025-01-10T08:50:02Z

.buildkite/ftr_security_stateful_configs.yml

@@ -104,3 +104,8 @@ enabled:
  - x-pack/test/cloud_security_posture_functional/data_views/config.ts
  - x-pack/test/automatic_import_api_integration/apis/config_basic.ts
  - x-pack/test/automatic_import_api_integration/apis/config_graphs.ts
+  - x-pack/test/spaces_api_integration/deployment_agnostic/spaces_only/config.ts
+  - x-pack/test/spaces_api_integration/deployment_agnostic/security_and_spaces/stateful.config_basic.ts


I think we discussed it with @azasypkin and @jeramysoucy , that while basic license is stateful-only, these tests cannot be considered deployment-agnostic. wdyt?

I can move them out of the deployment_agnostic folder to eliminate confusion, but all those tests still use the same test suite the serverless/trial config does

dmlemeshko

Great progress! I left few questions

elena-shostak · 2025-01-17T17:36:21Z

@elasticmachine merge upstream

elasticmachine · 2025-01-17T19:07:02Z

💚 Build Succeeded

Buildkite Build
Commit: 5e7de03

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`@kbn/test-suites-xpack`	737	721	-16

Total ESLint disabled count

id	before	after	diff
`@kbn/test-suites-xpack`	762	746	-16

History

💛 Build #264772 was flaky e8c8bfc
💔 Build #264710 failed 4f21b9e
💛 Build #264620 was flaky 89482c1
💔 Build #264515 failed 89482c1
💔 Build #264464 failed 6f5f03c
💔 Build #264447 failed f4b09c7

elena-shostak changed the title ~~Moved tests to agnostic setup~~ [Spaces] Moved tests to agnostic setup Nov 19, 2024

elena-shostak added Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! Feature:Security/Spaces Platform Security - Spaces feature FTR release_note:skip Skip the PR/issue when compiling release notes labels Nov 19, 2024

elena-shostak force-pushed the space-deployment-agnostic branch from 8fedf64 to f6def7f Compare November 19, 2024 14:11

azasypkin self-requested a review November 19, 2024 16:02

elena-shostak force-pushed the space-deployment-agnostic branch 2 times, most recently from 2df08a0 to 1464f66 Compare December 27, 2024 12:55

elena-shostak force-pushed the space-deployment-agnostic branch from efed4f7 to 585fcec Compare December 30, 2024 13:25

elena-shostak added 5 commits January 3, 2025 11:07

Moved tests to agnostic setup

5f94ea3

wip

6b93044

Fixes

3ec74cd

Fixes

2cec1ea

fixes

1231e09

elena-shostak commented Jan 3, 2025

View reviewed changes

elena-shostak added 2 commits January 3, 2025 18:37

Fixup

c91bfc0

Serverless config

1d1e6b8

azasypkin reviewed Jan 6, 2025

View reviewed changes

x-pack/test/spaces_api_integration/common/services/test_data_loader.ts Show resolved Hide resolved

elena-shostak added 5 commits January 6, 2025 14:32

Fixes

7d68cd6

Fixes

f4b09c7

Fixes

6f5f03c

Service fix

89482c1

Fixes

4f21b9e

elena-shostak commented Jan 7, 2025

View reviewed changes

elena-shostak added the backport:skip This commit does not require backporting label Jan 7, 2025

elena-shostak marked this pull request as ready for review January 7, 2025 14:00

elena-shostak requested review from a team as code owners January 7, 2025 14:00

Merge branch 'main' into space-deployment-agnostic

e8c8bfc

jbudz approved these changes Jan 7, 2025

View reviewed changes

wayneseymour approved these changes Jan 7, 2025

View reviewed changes

jeramysoucy requested a review from dmlemeshko January 9, 2025 17:04

dmlemeshko reviewed Jan 10, 2025

View reviewed changes

jeramysoucy self-requested a review January 10, 2025 17:17

Fixes

ea7beae

Merge branch 'main' into space-deployment-agnostic

5e7de03

[Spaces] Moved tests to agnostic setup #200606

Are you sure you want to change the base?

[Spaces] Moved tests to agnostic setup #200606

Conversation

elena-shostak commented Nov 18, 2024 • edited Loading

Summary

Checklist

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 18, 2024

elena-shostak commented Nov 19, 2024

elena-shostak commented Nov 19, 2024

elena-shostak commented Nov 19, 2024

elena-shostak commented Nov 19, 2024

elena-shostak commented Nov 19, 2024

kibanamachine commented Dec 27, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7634

kibanamachine commented Dec 27, 2024

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7635

elena-shostak commented Dec 27, 2024

elena-shostak commented Dec 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elena-shostak Jan 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elena-shostak Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Jan 7, 2025

elena-shostak commented Jan 7, 2025

jbudz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeramysoucy Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elena-shostak Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

dmlemeshko left a comment

Choose a reason for hiding this comment

elena-shostak commented Jan 17, 2025

elasticmachine commented Jan 17, 2025

💚 Build Succeeded

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

elena-shostak commented Nov 18, 2024 •

edited

Loading

elena-shostak Jan 3, 2025 •

edited

Loading

elena-shostak Jan 7, 2025 •

edited

Loading

jbudz left a comment •

edited

Loading

jeramysoucy Jan 14, 2025 •

edited

Loading

elena-shostak Jan 10, 2025 •

edited

Loading