[Fleet] Fix agent policy mappings for space awareness #201689

nchaulet · 2024-11-25T20:39:20Z

Summary

In #193361 we introduced new mappings for the agent policy.

We now have two saved object for agent policies, one space aware, one not, and mappings should be added to both, that PR fix that.

I also added a unit test so we make sure that both mappings stay the same

How to reproduce

Enable space awareness (feature flag + API call) than try to enable http monitoring

elasticmachine · 2024-11-25T20:39:23Z

Pinging @elastic/fleet (Team:Fleet)

…te --fix'

jen-huang

Thanks for fixing this

jen-huang · 2024-11-25T21:10:54Z

x-pack/plugins/fleet/server/saved_objects/index.test.ts

+      Object.keys(soTypes['fleet-agent-policies'].mappings).sort()
+    );
+  });
+  it('should have the same mappings for space and non-space aware package policies', () => {


… src/core/server/integration_tests/ci_checks'

…nchaulet/kibana into fix-agent-policies-space-aware-mappings

rudolf

Please avoid using index: false or enabled: false.

There's more context in this internal RFC https://docs.google.com/document/d/1wCAaIQ68J1EAPH4eOdWe8LPQvwr3AXPQZx0ryHC7bK0/edit?tab=t.0#bookmark=id.trb1dli8hz4n

But we recently learned that not all teams were aware of this. So we've added some docs https://github.com/elastic/kibana/pull/201969/files and will also remove it from the mappings type so that it causes a type failure (fleet uses this quite extensively so we'll need to add ts-expect-error to ignore these).

nchaulet · 2024-11-27T14:59:42Z

@rudolf can we update dynamic: false on existing mappings? I tried to make the change but I got the following error when trying to write my saved object?

Unexpected bulk response [400] strict_dynamic_mapping_exception: [1:196] mapping set to strict, dynamic introduction of [monitoring_http] within [fleet-agent-policies] is not allowed

rudolf · 2024-11-27T15:03:36Z

Yeah it should be possible. That error indicates that somehow you don't have dynamic: false inside fleet-agent-policies. Did you introduce the new mappings by adding a new model version?

nchaulet · 2024-11-27T15:18:27Z

Yeah it should be possible. That error indicates that somehow you don't have dynamic: false inside fleet-agent-policies. Did you introduce the new mappings by adding a new model version?

@rudolf thank it seems to work when I add an empty modelVersion I will update my PR

  modelVersions: {
        '1': {
          changes: [
            {
              type: 'mappings_addition',
              addedMappings: {},
            },
          ],
        },
      },

elasticmachine · 2024-11-27T15:35:07Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: fbacaea

Failed CI Steps

Checks

History

💛 Build #254460 was flaky d2d77a5
💚 Build #254240 succeeded 5a71934

cc @nchaulet

…te --fix'

… src/core/server/integration_tests/ci_checks'

nchaulet · 2024-11-27T17:07:21Z

@elasticmachine merge upstream

rudolf · 2024-11-28T13:04:30Z

x-pack/plugins/fleet/server/saved_objects/index.ts

+          changes: [
+            {
+              type: 'mappings_addition',
+              addedMappings: {},


Suggested change

addedMappings: {},

addedMappings: {

dynamic: false,

},

You should also specify the change in mappings here. It's unfortunate that we have this duplication of the mappings definitions, but we didn't see a good way around it.

https://github.com/elastic/kibana/blob/skip-transform-on-mappings-change/docs/developer/architecture/core/saved-objects-service.asciidoc?plain=1#L324-L325

@rudolf I tried this but I got a type error Type 'boolean' is not assignable to type 'SavedObjectsFieldMapping' Should we ignore that error?

ah I see, you're right, looks like a limitation in the modelVersions API in that you can't actually specify a change that sets the top-level dynamic option.

rudolf · 2024-11-28T13:04:35Z

x-pack/plugins/fleet/server/saved_objects/index.ts

+          changes: [
+            {
+              type: 'mappings_addition',
+              addedMappings: {},


Suggested change

addedMappings: {},

addedMappings: {

dynamic: false,

},

rudolf · 2024-11-28T13:09:09Z

We have sent previous guidance around strict API validation. But with making the mappings dynamic: false it's worth double checking that HTTP APIs that write AGENT_POLICY_SAVED_OBJECT_TYPE saved objects have validation that would prevent adding an introduce a new invalid field to these saved objects.

nchaulet · 2024-11-28T14:05:00Z

We have sent previous guidance around strict API validation. But with making the mappings dynamic: false it's worth double checking that HTTP APIs that write AGENT_POLICY_SAVED_OBJECT_TYPE saved objects have validation that would prevent adding an introduce a new invalid field to these saved objects.

Yes the policy API seems safe on that side, we valid every field added to that saved object.

rudolf · 2024-11-28T22:10:56Z

x-pack/plugins/fleet/server/saved_objects/index.ts

+          changes: [
+            {
+              type: 'mappings_addition',
+              addedMappings: {},


ah I see, you're right, looks like a limitation in the modelVersions API in that you can't actually specify a change that sets the top-level dynamic option.

kibanamachine · 2024-11-28T22:11:29Z

Starting backport for target branches: 8.16, 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/12075965885

(cherry picked from commit 721b4be)

kibanamachine · 2024-11-28T22:16:36Z

💚 All backports created successfully

Status	Branch	Result
✅	8.16
✅	8.17
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

#202240) # Backport This will backport the following commits from `main` to `8.17`: - [[Fleet] Fix agent policy mappings for space awareness (#201689)](#201689)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Nicolas Chaulet <[email protected]>

#202239) # Backport This will backport the following commits from `main` to `8.16`: - [[Fleet] Fix agent policy mappings for space awareness (#201689)](#201689)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Nicolas Chaulet <[email protected]>

…#202241) # Backport This will backport the following commits from `main` to `8.x`: - [[Fleet] Fix agent policy mappings for space awareness (#201689)](#201689)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Nicolas Chaulet <[email protected]>

[Fleet] Fix agent policy mappings for space awareness

d129bc1

nchaulet added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.17.0 labels Nov 25, 2024

nchaulet self-assigned this Nov 25, 2024

nchaulet requested review from a team as code owners November 25, 2024 20:39

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

f9b3035

…te --fix'

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner November 25, 2024 20:54

nchaulet added the v8.16.2 label Nov 25, 2024

fix test

4b3c2a5

nchaulet requested a review from jen-huang November 25, 2024 21:07

jen-huang approved these changes Nov 25, 2024

View reviewed changes

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

5a71934

… src/core/server/integration_tests/ci_checks'

nchaulet enabled auto-merge (squash) November 25, 2024 23:41

SiddharthMantri approved these changes Nov 26, 2024

View reviewed changes

Merge branch 'fix-agent-policies-space-aware-mappings' of github.com:…

d2d77a5

…nchaulet/kibana into fix-agent-policies-space-aware-mappings

rudolf requested changes Nov 27, 2024

View reviewed changes

change agent policy mappings to dynamic:false

fbacaea

nchaulet requested a review from rudolf November 27, 2024 15:21

nchaulet and others added 3 commits November 27, 2024 10:41

fix mappings

d27b010

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

1660392

…te --fix'

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

7e83e00

… src/core/server/integration_tests/ci_checks'

Merge branch 'main' into fix-agent-policies-space-aware-mappings

c78a9d1

rudolf reviewed Nov 28, 2024

View reviewed changes

rudolf approved these changes Nov 28, 2024

View reviewed changes

nchaulet merged commit 721b4be into elastic:main Nov 28, 2024
9 checks passed

kibanamachine added the v9.0.0 label Nov 28, 2024

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 28, 2024

[Fleet] Fix agent policy mappings for space awareness (elastic#201689)

b548684

(cherry picked from commit 721b4be)

kibanamachine mentioned this pull request Nov 28, 2024

[8.16] [Fleet] Fix agent policy mappings for space awareness (#201689) #202239

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 28, 2024

[Fleet] Fix agent policy mappings for space awareness (elastic#201689)

78490b2

(cherry picked from commit 721b4be)

kibanamachine mentioned this pull request Nov 28, 2024

[8.17] [Fleet] Fix agent policy mappings for space awareness (#201689) #202240

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 28, 2024

[Fleet] Fix agent policy mappings for space awareness (elastic#201689)

83f08d8

(cherry picked from commit 721b4be)

kibanamachine mentioned this pull request Nov 28, 2024

[8.x] [Fleet] Fix agent policy mappings for space awareness (#201689) #202241

Merged

kibanamachine added the v8.18.0 label Nov 28, 2024

CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024

[Fleet] Fix agent policy mappings for space awareness (elastic#201689)

c23e744

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Fix agent policy mappings for space awareness #201689

[Fleet] Fix agent policy mappings for space awareness #201689

nchaulet commented Nov 25, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Nov 25, 2024

jen-huang left a comment

jen-huang Nov 25, 2024

rudolf left a comment

nchaulet commented Nov 27, 2024

rudolf commented Nov 27, 2024

nchaulet commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

nchaulet commented Nov 27, 2024

rudolf Nov 28, 2024

nchaulet Nov 28, 2024 •

edited

Loading

rudolf Nov 28, 2024

rudolf Nov 28, 2024

rudolf commented Nov 28, 2024

nchaulet commented Nov 28, 2024

rudolf Nov 28, 2024

kibanamachine commented Nov 28, 2024

kibanamachine commented Nov 28, 2024

[Fleet] Fix agent policy mappings for space awareness #201689

[Fleet] Fix agent policy mappings for space awareness #201689

Conversation

nchaulet commented Nov 25, 2024 • edited by kibanamachine Loading

Summary

How to reproduce

elasticmachine commented Nov 25, 2024

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Nov 25, 2024

Choose a reason for hiding this comment

rudolf left a comment

Choose a reason for hiding this comment

nchaulet commented Nov 27, 2024

rudolf commented Nov 27, 2024

nchaulet commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

⏳ Build in-progress, with failures

Failed CI Steps

History

nchaulet commented Nov 27, 2024

rudolf Nov 28, 2024

Choose a reason for hiding this comment

nchaulet Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

rudolf Nov 28, 2024

Choose a reason for hiding this comment

rudolf Nov 28, 2024

Choose a reason for hiding this comment

rudolf commented Nov 28, 2024

nchaulet commented Nov 28, 2024

rudolf Nov 28, 2024

Choose a reason for hiding this comment

kibanamachine commented Nov 28, 2024

kibanamachine commented Nov 28, 2024

💚 All backports created successfully

Questions ?

nchaulet commented Nov 25, 2024 •

edited by kibanamachine

Loading

nchaulet Nov 28, 2024 •

edited

Loading