Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Store integration policy secrets in the .fleet-secrets index #154731

Closed
13 of 14 tasks
hop-dev opened this issue Apr 11, 2023 · 2 comments · Fixed by #157176
Closed
13 of 14 tasks

[Fleet] Store integration policy secrets in the .fleet-secrets index #154731

hop-dev opened this issue Apr 11, 2023 · 2 comments · Fixed by #157176
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@hop-dev
Copy link
Contributor

hop-dev commented Apr 11, 2023 •
edited by juliaElastic
Loading

Part of #154715

Using the index created in elastic/elasticsearch#95143 store integration policy secrets separate from the policy its self.

Implementation list:

  • Create feature flag

Package Policy API

  • Create package policy API should take plain text secrets and add them to the secret store, replacing them with secret references
    • On creating an integration policy, if a field is secret kibana will have to extract it, write the secret to the .secrets index and then replace the value with a reference in the .fleet-policies and .kibana index.
  • Edit package policy API should accept existing secret references and not return validation errors
  • Edit package policy API should allow new secrets to be provided in plain text and update them in the secret store, replacing them with secret references
  • Upgrade APIs to support moving a value from .fleet-policies to .secrets when a field moves to being secret
  • On package policy delete, kibana should delete associated secrets too that are only used by that policy
  • Add new top level secret_refs field to policy saved object
    Agent Policy
  • The get full agent policy API will return the secret references as-is, it will be up to the frontend to add the placeholder values (no change needed)
  • new top level secret_refs field in policies index with all secrets used in the agent policy
  • clone policy API should retain references to secrets

Integration Policy Editor

  • When creating a new policy should allow secret fields to be input as they currently are (no change needed)
  • When editing a policy, secret fields should be obscured but still edit-able (All secrets will be basic text inputs so this should just be one new element)

Agent Policy YAML Viewer

  • The UI consumes the full policy and converts it to YAML here, we would just need to replace secret refs with a standard value.
    We could use a static string e.g ${YOUR_SECRET} or we could infer something more descriptive from the field name itself. E.g “- name: password” could use ${PASSWORD} as its replacement
@hop-dev hop-dev added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 11, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@hop-dev hop-dev mentioned this issue Apr 11, 2023
Closed
@hop-dev hop-dev changed the title [Fleet] Store integration policy secrets in the .secrets index [Fleet] Store integration policy secrets in the .fleet-secrets index May 10, 2023
This was referenced May 10, 2023
Merged
Merged
@juliaElastic juliaElastic mentioned this issue May 11, 2023
Merged
9 tasks
hop-dev added a commit that referenced this issue May 16, 2023
@hop-dev
[Fleet] Update secret values (API only) (#156806)
9d5c1cb 
## Summary

Part of #154731

Allow secrets to be updated via the API. When a secret value is updated,
the secret reference is replaced with a "raw" value we detect this on
the API and create a new secret document.

Once a secret reference is updated, we clean up the old secret document
if it is not in use by another policy. This check is a simple lookup of
the secret_references array on policies.

API integration tests updated.
@juliaElastic
Copy link
Contributor

juliaElastic commented May 17, 2023 •
edited
Loading

The work in this is already tracked in #154715, closing as duplicate.

jasonrhodes pushed a commit that referenced this issue May 17, 2023
@hop-dev @jasonrhodes
[Fleet] Update secret values (API only) (#156806)
1f90bd1 
## Summary

Part of #154731

Allow secrets to be updated via the API. When a secret value is updated,
the secret reference is replaced with a "raw" value we detect this on
the API and create a new secret document.

Once a secret reference is updated, we clean up the old secret document
if it is not in use by another policy. This check is a simple lookup of
the secret_references array on policies.

API integration tests updated.
jillguyonnet added a commit that referenced this issue Jun 26, 2023
@hop-dev @jillguyonnet @kibanamachine
[Fleet] View & Edit Secret Variables (#157176)
e72e055 
## Summary
Closes #154731

Related to #154715

Adds a new component for viewing and editing secret values in the policy
editor. Secrets can be written on policy create, when editing a policy,
if a secret is filled then we give the user the option to replace the
value only.

Also when viewing the agent policy yaml, hide secret references and
replace them with incremental env var names, and add a callout which
tells the user they need to replace the vars.

I have kept the UI un-opinionated about which type of field can be a
secret.

**Secret variable in package policy editor:**


https://github.com/elastic/kibana/assets/3315046/b452668f-196e-4ace-a170-be45df135834



Replaced secrets and new callout in policy yaml viewer:


https://github.com/elastic/kibana/assets/3315046/75d684ba-d76e-454b-96bc-9093ee2afbfc



Testing

- start a docker package registry 
```
docker run -p 8080:8080 -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/markhopkin/dev/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main
```
- point your kibana to the registry and use the `secrets-1.0.0` package
to test
- create a package policy with secrets (should be successful, should
show 'normal' input components)
- edit a package policy with secrets (should be successful, should show
"replace value" panel for filled values)
- view agent policy yaml with secrets (should not show secret references
in yaml, should show ${SECRET_0} etc)

### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Jill Guyonnet <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
Co-authored-by: jillguyonnet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] View & Edit Secret Variables hop-dev/kibana
3 participants
@hop-dev @elasticmachine @juliaElastic