diff --git a/docs/en/integrations/build-integration.asciidoc b/docs/en/integrations/build-integration.asciidoc index 42643ea44e..8d29c625f5 100644 --- a/docs/en/integrations/build-integration.asciidoc +++ b/docs/en/integrations/build-integration.asciidoc @@ -7,6 +7,7 @@ Ready to monitor, ingest, and visualize something? Let's get started. * <> * <> * <> +* <> * <> * <> * <> @@ -25,7 +26,8 @@ Before building an integration, you should have an understanding of the followin * {stack} concepts, like data streams, ingest pipelines, and mappings * The <> -In addition, you must have <> installed on your machine. +In addition, you must have <> installed on your machine. +Using `elastic-package` is recommended for integration maintainers as it provides crucial utilities and scripts for building out integrations. [[build-spin-stack]] == Spin up the {stack} @@ -105,6 +107,7 @@ These assets are loaded into {es} when a user installs an integration using the A data stream also defines a policy template. Policy templates include variables that allow users to configure the data stream using the {fleet} UI in {kib}. Then, the {agent} interprets the resulting policy to collect relevant information from the product or service being observed. +Policy templates can also define an integration's supported <>. See {fleet-guide}/data-streams.html[data streams] for more information. **** @@ -128,6 +131,101 @@ Next, manually adjust the data stream: * define ingest pipeline definitions (if necessary) * update the {agent}'s stream configuration +[[define-deployment-modes]] +== Define deployment modes + +Some integrations can be deployed on fully managed agents. +These integrations are known as "agentless" integrations. +Define the deployment mode of an integration with the <> property and display/hide variables +in different deployment modes with the <> property. + +[discrete] +[[deployment_modes]] +=== `deployment_modes` + +Policy templates can indicate which deployment modes they support. +Use the `deployment_modes` property in the policy template schema to define the supported deployment modes. +Options are `default` and `agentless`. A policy template can support both modes. + +Example policy template declaration: + +[source,yaml] +---- +format_version: 3.2.0 +name: aws +title: AWS +version: 2.13.1 +... +policy_templates: + - name: billing + title: AWS Billing + description: Collect billing metrics with Elastic Agent + deployment_modes: <1> + default: + enabled: false <2> + agentless: + enabled: true <3> + data_streams: + - billing + ... +---- +<1> Defines the supported deployment modes +<2> Disables agent deployment support +<3> Enables agentless deployment support + +[discrete] +[[hide_in_deployment_modes]] +=== `hide_in_deployment_modes` + +Variables can be hidden in certain deployment modes. +Use the `hide_in_deployment_modes` property to opt variables in or out of being displayed in default or agentless mode. +This property works at any manifest level. + +Example variable declaration: + +[source,yaml] +---- +streams: + - input: logfile + vars: + - name: paths + type: text + title: Paths + multi: true + required: true + show_user: true + default: + - /var/log/my-package/*.log + - name: agentless_only + type: text + title: Agentless only variable + multi: false + required: false + show_user: true + hide_in_deployment_modes: <1> + - default + - name: hidden_in_agentless + type: text + title: Hidden in agentless variable + multi: false + required: false + show_user: true + hide_in_deployment_modes: <2> + - agentless +---- +<1> Disables visibility of the variable in agent deployment mode +<2> Disables visibility of the variable in agentless deployment mode + +For more information on variable property definitions, refer to <>. + +[discrete] +[[agentless-capabilities]] +=== Agentless capabilities + +The capabilities feature protects agentless deployments from allowing undesired inputs to run. +A static `capabilities.yml` file defines these allowed and disallowed inputs and is passed to deployed agents. +To determine which capabilities are currently allowed on Agentless, refer to https://github.com/elastic/agentless-controller/blob/main/controllers/config/capabilities.yml[`capabilities.yml`]. + [[edit-ingest-pipeline]] == Edit ingest pipelines @@ -571,6 +669,7 @@ To see how to use template functions, for example {{fields "data-stream-name"}}, === Review artifacts +[[define-variable-properties]] === Define variable properties The variable properties customize visualization of configuration options in the {kib} UI. Make sure they're defined in all manifest files. @@ -585,6 +684,8 @@ vars: description: Paths to the nginx access log file. <4> type: text <5> multi: true <6> + hide_in_deployment_modes: <7> + - agentless default: - /var/log/nginx/access.log* ---- @@ -593,7 +694,8 @@ vars: <3> human readable variable name <4> variable description (may contain some details) <5> field type (according to the reference: text, password, bool, integer) -<6> the field has multiple values. +<6> the field has multiple values +<7> hides the variable in agentless mode (see <> for more information) // === Add sample events diff --git a/docs/en/integrations/elastic-package.asciidoc b/docs/en/integrations/elastic-package.asciidoc index 1630642be9..f8c4ad72a7 100644 --- a/docs/en/integrations/elastic-package.asciidoc +++ b/docs/en/integrations/elastic-package.asciidoc @@ -118,11 +118,78 @@ Use this command to format the package files. The formatter supports JSON and YAML format and skips "ingest_pipeline" directories as it's hard to correctly format Handlebars template files. As a result, formatted files are overwritten. [discrete] +[[elastic-package-install]] === `elastic-package install` _Context: package_ -Use this command to install the package in {kib}. +Use this command to upload and install a package in {kib}. + +Starting with Kibana version `8.7.0`, packages do not need to be exposed in the Package Registry to be installed. +Instead, they can be upload as zip files built using the `elastic-package build` command. + +1. Ensure you've validated your package. Before building, validate the package by running the `elastic-package check` command. +2. Use either the `--zip` parameter to install a specific zip file or the `install` command to build the package and upload the built zip file to Kibana. + +[discrete] +==== Install with `--zip` + +Install a zipped package. This method relies on Package Registry. + +[source,shell] +---- +elastic-package stack up -d +elastic-package install --zip /home/user/Coding/work/integrations/build/packages/elastic_package_registry-0.0.6.zip -v +---- + +[discrete] +==== Install with `elastic-package install` + +Build and upload a zipped package without relying on Package Registry. + +[source,shell] +---- +elastic-package stack up -v -d +elastic-package install -v +---- + +[discrete] +==== Customization + +Package installation can be customized to be installed in other Kibana instances with the following variables: + +* `ELASTIC_PACKAGE_KIBANA_HOST` +* `ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME` +* `ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD` +* `ELASTIC_PACKAGE_CA_CERT` + +For example: + +[source,bash] +---- +export ELASTIC_PACKAGE_KIBANA_HOST="https://test-installation.kibana.test:9243" +export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME="elastic" +export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD="xxx" +# if it is a public instance, this variable should not be needed +export ELASTIC_PACKAGE_CA_CERT="" + +elastic-package install --zip elastic_package_registry-0.0.6.zip -v +---- + +[discrete] +==== Older versions + +For versions of Kibana `<8.7.0`, the package must be exposed via the Package Registry. +In case of development, this means that the package should be built previously and then the Elastic stack must be started. +Or, at least, the `package-registry` service needs to be restarted in the Elastic stack: + +[source,terminal] +---- +elastic-package build -v +elastic-package stack up -v -d # elastic-package stack up -v -d --services package-registry +elastic-package install -v +---- + To install the package in {kib}, the command uses {kib} API. The package must be exposed via the {package-registry}. diff --git a/docs/en/integrations/index.asciidoc b/docs/en/integrations/index.asciidoc index b678a2a403..d6b2cc79bf 100644 --- a/docs/en/integrations/index.asciidoc +++ b/docs/en/integrations/index.asciidoc @@ -14,6 +14,8 @@ include::what-is-an-integration.asciidoc[leveloffset=+1] include::build-integration.asciidoc[leveloffset=+1] +include::upload-integration.asciidoc[leveloffset=+1] + include::testing.asciidoc[leveloffset=+1] include::publish-integration.asciidoc[leveloffset=+1] diff --git a/docs/en/integrations/upload-integration.asciidoc b/docs/en/integrations/upload-integration.asciidoc new file mode 100644 index 0000000000..0003458c03 --- /dev/null +++ b/docs/en/integrations/upload-integration.asciidoc @@ -0,0 +1,51 @@ +[[upload-a-new-integration]] += Upload an integration to Kibana + +++++ +Upload an integration +++++ + +{fleet} supports integration installation through direct upload as a means to support integration developers +or users who have created custom integrations that they don't want to commit upstream back to the https://github.com/elastic/integrations[Elastic Integrations repository]. + +Direct upload can also be useful in air-gapped environments, +by providing a way to update integrations without needing to update a self-hosted package registry. + +[discrete] +[[upload-integration-local]] +== Local development + +If you've followed the local development steps in <>, upload your integration to Kibana with the following command: + +[source,terminal] +---- +elastic-package install --zip /path/to/my/custom-integration +---- + +For more information, see <>. + +[discrete] +[[upload-integration-production]] +== Production deployment + +To upload your integration to a production deployment, first zip the package: + +[source,terminal] +---- +$ cd /path/to/my/custom-integration +$ elastic-package build +---- + +You can now use the Kibana API to upload your integration: + +[source,terminal] +---- +$ curl -XPOST \ + -H 'content-type: application/zip' \ + -H 'kbn-xsrf: true' \ + http://your.kibana.host/api/fleet/epm/packages \ + -u {username}:{password} \ + --data-binary @my-custom-integration.zip +---- + +More information on this endpoint is available in the {fleet-guide}/fleet-apis.html[Fleet API Reference].