Making security a first class citizen of Kibana Core

[pgayvallet]

Context

We've been discussing this for a while now, but we never officialized it by opening an issue. I guess it's now done.

Security in Core, why?

Over the past few years, we've been encountering more and more scenarios and use cases where Core not being aware of the security concepts and primitives (authc, authz, user, principal...) and not able to use the security API is problematic.

Historically (and to summarize), the reasons behind this hard separation was xpack and the difference of license between the oss and xpack parts of the code, which drove the decision to have the full security implementation being under the xpack license/folder. However, things changed regarding our code's licensing, and also, there are way to keep the implementation under x-pack while having a proper API exposed from Core.

Functionally, it has always been an issue, but we worked around by various ways (exposing hapihooks so that security can plug its authc within the http lifecycle directly, having the SOR security wrapper, then the SOR security extension, and other inversions of control such as the elasticsearch.setUnauthorizedErrorHandler API to perform reauthentication during ES requests).

However, we now have a strong feeling that now would be a good time to finally tackle this technical debt and have security be more directly and fully integrated within Core.

This would unblock a few significant improvements currently blocked by that bad integration of security (e.g the new Kibana DI system that is planning to introduce the concept of user/project scoped services), and is also a prerequisite if we eventually want to go multitenant or multiproject (as the new DI system is also a prerequisite for going into that direction anyway).

It would also cleanup some tech debt, by having a better integration / handling of security within Core. For example, Core would no longer need to expose the HAPI lifecycle hooks directly used by security to plug itself, the http security logic could then remain an implementation detail of Core, that would just directly it's security API (backed up by the implementation provided by security).

Last but not least, it would also allow to address some cyclic dependencies issues between the security plugin and other plugins (issues we started encountering recently), given security consumers would now use Core APIs directly for this, without having to known (or depends on) which plugin provides the underlying implementation.

What would change for public security API

The details will need to be discussed, but the (hopefully not so naive) way I was seeing it was to "simply" use the current security plugin's contract as the public contract that would be exposed by the new core.security service.

That way, consumers currently using the security plugin would be able to easily transition to using the core service instead.

E.g

would become something like

Given the contract would be the same, we could, during the transition period, simply have the security plugin still expose the same contract, so that the same API would be accessible both via the security plugin contract and via the core.security service contract.

How would that work for internal security APIs

I assume we will also need the security plugin to expose/register to Core more than just the public contract we'll re-expose.

For example, I guess we will need the security plugin to expose the logic used to perform the http authentication (what's currently being registered via the HAPI hooks)

For that, (the way I currently imagine it at least), I think we could just re-use the concept of internal/public service contracts we've been using in Core since the new platform rewrite: services internally expose internal contracts, that are supersets of the public ones. This allows our services to expose APIs that would only be used internally within Core. Then, when injecting the contract to plugins, we build the public version of the contract from the internal one (99% of the time by just removing the internal APIs, the 1% remaining is by injecting the pluginID as an additional parameter only present on the internal version of the contract).

E.g

kibana/packages/core/plugins/core-plugins-server-internal/src/plugin_context.ts

Lines 205 to 210 in c713b91

	customBranding: {
	register: (fetchFn) => {
	deps.customBranding.register(plugin.name, fetchFn);
	},
	getBrandingFor: deps.customBranding.getBrandingFor,
	},

So we would have 2 versions of both the setup and start contracts of the security API:

And the security plugin would register the private version of those contracts during the setup and start phases of the plugin:

very naive:

class SecurityPlugin {
  setup(core) {
    // do the normal things
    core.security.setSecuritySetupContract(internalSecuritySetup);
  }
  start(core) {
    // do the normal things
    core.security.setSecurityStartContract(internalSecurityStart);
  }
}

Do we need to do that for the full set of the security plugin's API

We at least need to register/expose authc and authz that way.

Note sure about the rest, but the security team should be able to help here:

• exposed from the setup service:
  • license
  • audit (audit logging)
  • privilegeDeprecationsService
• exposed from the start service:
  • userProfiles

Questions / things to figure out

Before going further, we need both our teams, @elastic/kibana-security and @elastic/kibana-core, to align on the following questions:

• Do we think this initiative makes sense?

• Do we think the suggested technical approach described in that issue makes sense (this is probably the most important question)?

• Are there parts of the current security plugin contract we can avoid bridging into core (e.g userProfiles for example)?

• What internal security APIs does Core need in addition to just the public security contract?

Phase 1 Tasks DONE

Preview Give feedback

1. Moving authc APIs to Core [Security into Core] expose authc.getCurrentUser from core.security #177976
2. Moving userProfile APIs to Core [security-in-core] Expose userProfile APIs from Core  #178932
3. Moving auditLogging APIs to Core [security-in-core] Expose auditLogging APIs from Core #178934
4. Adapting userSettingsService to no longer depends on the security plugin adapt Core userSettings service to no longer depends on the security plugin #181538

Phase 2 Tasks target End Aug 2024

Preview Give feedback

1. Expose API Key service under core.security.authc [Security-in-core] Expose API Key service under core.security.authc #184764
2. Core licensing service (discussion & plan) [Security in Core] Core licensing service #187296
3. Migration examples Making security a first class citizen of Kibana Core examples #186574

Phase 3 Tasks target End Oct 2024

Preview Give feedback

1. expose authz APIs from Core [security-in-core] Expose authz APIs from Core #183849
2. Core licensing service (implementation Blocked pending solution to single vs dual licensed code problem) [Security in Core] Core licensing service #187296
3. Move security plugin's config to core [Security in Core] Move security plugin's config to core #186863
4. Migration examples Making security a first class citizen of Kibana Core examples #186574

Other Tasks: target End December

Preview Give feedback

1. [Security-in-core] First-class route authorization #184674
   Feature:Hardening Feature:Security/Authorization Team:Core Team:Security enhancement +5
   

Tech debt: target End Feb 2025

Preview Give feedback

1. Cleaning up the (authentication to get user info) hooks between the security plugin and Core (Authentication / http & ES re-authentication) [Security in Core]: Cleaning up the hooks between the security plugin and Core #187211
2. Cleaning up the hooks between the security plugin and Core (Saved Objects extensions & http.registerOnPreResponse) Clean up hooks between the security plugin and Core that have dependencies on other services #187418

Important changes to the plan:

July, 2024

The Core Encryption Service has evolved into a much larger initiative than we originally planned for. It will be handled separately.

August, 2024

Scope changes for Phase 2 and 3 based on Platform plugins audit

[pgayvallet]

After thinking about it a bit more, I just realized that using a simple API registration pattern for the security plugin to register its API during its respective setup and start lifecycle methods won't probably work here, given the security plugin's lifecycle will be executed at the same time as other plugins (meaning that if another plugin tries to access core.security.someApi() before it has been registered, it won't be available.

That makes me wonder if we won't need, after all, to move the implementation to some package and have core use it directly (instead of depending on the security plugin as I initially suggested)...

[pgayvallet]

(continuing to think out loud)

Looking at the current setup contract of the security plugin, I see that we're exposing (even if they are deprecated) most of the authc and authz service.

kibana/x-pack/plugins/security/server/plugin.ts

Lines 73 to 85 in f7fa846

	/**
	* Describes public Security plugin contract returned at the `setup` stage.
	*/
	export interface SecurityPluginSetup extends SecurityPluginSetupWithoutDeprecatedMembers {
	/**
	* @deprecated Use `authc` methods from the `SecurityServiceStart` contract instead.
	*/
	authc: { getCurrentUser: (request: KibanaRequest) => AuthenticatedUser | null };
	/**
	* @deprecated Use `authz` methods from the `SecurityServiceStart` contract instead.
	*/
	authz: AuthorizationServiceSetup;
	}

If we get to the conclusion that those 2 services are sufficient to be injected into Core (meaning that they are the only relevant parts than we need to be able to use in Core and to eventually re-expose to plugin), an alternative, that would require less work, would be to have the security plugin register this part of its setup contract during setup, and have core exposes it to plugins only via its start contract.

Given most (all?) of the authc / authz APIs takes a KibanaRequest as parameter, having those APIs only usable inside core during start (and not during setup) seems sufficient.

This is less of a refactor, as we basically only introduce a new registration API in Core to be used by the security plugin, but it would still solve most of what we're trying to achieve here, so I wonder if it wouldn't be a pragmatic quick win?

[azasypkin]

Do we think this initiative makes sense?

It totally does, thanks for filing this issue!

Do we think the suggested technical approach described in that issue makes sense (this is probably the most important question)?

On the surface, it sounds like the easiest approach that doesn't require any significant changes for either side, which makes sense to me.

After thinking about it a bit more, I just realized that using a simple API registration pattern for the security plugin to register its API during its respective setup and start lifecycle methods won't probably work here, given the security plugin's lifecycle will be executed at the same time as other plugins (meaning that if another plugin tries to access core.security.someApi() before it has been registered, it won't be available.

As you mentioned in the last comment, most of the security logic that other plugins rely on should be accessed after setup through the start contract and asynchronously (at least when it comes to authc and userProfiles), so we can technically pass everything the Core might need during setup. We won't expose anything that is either deprecated or soon to be deprecated via Core contracts. Having said that, our team will still need to take a closer look at our setup contract to see if there are any other potential roadblocks, especially in authz.

Are there parts of the current security plugin contract we can avoid bridging into core (e.g userProfiles for example)?

Initially, we might skip certain parts, I think. But eventually, most of the security stuff would make sense to have in the Core, especially userProfiles since the user profile ID is supposed to be our main unique user/principal identifier. As I already said previously, I'd start with authc and the server-side part of userProfiles to test the ground, and then begin approaching authz once we feel comfortable with the general approach.

All in all, I think this project is already actionable and worth pursuing.

[pgayvallet]

FYI, I started working on the first part of the project, exposing getCurrentUser from core.security, in #177976

[pgayvallet]

#177976 was just merged.

The next step would be to decide which security API(s) we expose next from Core, and if we can just re-use the exact same signatures or if we want to take this opportunity to clean / change it.

cc @TinaHeiligers @azasypkin

[pgayvallet]

Summary of yesterday's meeting with @TinaHeiligers and @azasypkin:

What's next?

We identified 4 areas

User profile

Migrating the userProfile APIs to re-expose them from Core

• We want to expose it from it's dedicated service (so core.userProfile and not core.security.userProfile)
• The public API doesn't need to change
• For the internal API, we will need to add at least update that is currently non-exposed on the server-side
• There is a stateless package about UI components for user profile, but we don't need to do anything about it
• Given we have only few consumers of those APIs, we could even adapt the existing consumers to use the new Core APIs and then fully remove the old APIs from the security plugin’s contract.

Internal hooks

This area is about cleaning the various hooks between the security plugin and core, by exposing internals APIs that Core could consume instead.

• Authentication / http

  • kibana/x-pack/plugins/security/server/authentication/authentication_service.ts

    Line 113 in de84466

    http.registerAuth(async (request, response, t) => {

  • kibana/x-pack/plugins/security/server/authentication/authentication_service.ts

    Line 189 in de84466

    http.registerOnPreResponse(async (request, preResponse, toolkit) => {

  • kibana/x-pack/plugins/security/server/authorization/authorization_service.tsx

    Line 172 in de84466

    http.registerOnPreResponse(async (request, preResponse, toolkit) => {

• ES re-authentication
  

  kibana/x-pack/plugins/security/server/authentication/authentication_service.ts

  Line 241 in de84466

  elasticsearch.setUnauthorizedErrorHandler(async ({ error, request }, toolkit) => {

There is no consumer-facing APIs to move here, it's mostly about taking this opportunity to (finally) remove tech debt around those hooks.

Authorization

Migrating the authorization APIs to re-expose them from Core.

• We do know that we want to change the shape of the API, but we're still not sure how exactly
  • This is an action point for the @elastic/kibana-security team
  • First step would be to audit to decide what we want to re-expose
  • Next step would be to decide on the shape of the new authz APIs

Audit logging

Migrating the auditLogging APIs to re-expose it from Core

• Still need to decide if we want to expose it from its dedicated service (core.auditLogging) or from the security service core.security.auditLogging)
• Some APIs may be exposed from the setup contract atm (need to check, as we won't be able to do the same
• Given we have only few consumers of those APIs, we could even adapt the existing consumers to use the new Core APIs and then fully remove the old APIs from the security plugin’s contract.

Order

1. User profile
2. Audit logging
3. Authz
4. Hooks

[pgayvallet]

I created #178932 and #178934 and updated the current issue's description to link to them.

[TinaHeiligers]

@pgayvallet we need to add an item to update all the README's for the core packages, as discussed in our latest sync. Something along the lines of:
"'this' package exposes the 'insert security service sub-domain' from core. Any issues with delegation of the underlying service should be reported to '#kibana-core'. Issues with the implementation itself should be reported with '#kibana-security'"

[pgayvallet]

The authc, userProfile and auditLogging services have been moved / exposed from Core.

The remaining tasks, as listed on this issue's description, are:

• Moving the authz service
• Performing cleanup on the various security/core hooks currently used

The Core team should be autonomous regarding the cleanup tasks. However regarding the authz service, from our last discussion, the Security team wanted to take this opportunity to potentially revisit the API surface.

@azasypkin / @legrego not really a high priority, but do you have any visibility on when you'll be able to work on that part?

[TinaHeiligers]

Performing cleanup on the various security/core hooks currently used

We should probably discuss our approach in an issue.

[legrego]

The authc, userProfile and auditLogging services have been moved / exposed from Core.

The remaining tasks, as listed on this issue's description, are:

• Moving the authz service
• Performing cleanup on the various security/core hooks currently used

The Core team should be autonomous regarding the cleanup tasks. However regarding the authz service, from our last discussion, the Security team wanted to take this opportunity to potentially revisit the API surface.

@azasypkin / @legrego not really a high priority, but do you have any visibility on when you'll be able to work on that part?

@pgayvallet we do not have an estimated date at this point. Our focus has been pulled away recently towards other commitments. The other service we want to add is the Core Encryption Service (#180867) -- this is a new service as opposed to moving an existing one, but between those two, which would you like to see us focus on first?

[TinaHeiligers]

between those two, which would you like to see us focus on first

@legrego it depends on which one will will be less effort and more reward: the new Core Encryption Service or authz.
We can't shy away from authz forever and need to tackle that in time to get it done and verified but it depends on priority from both teams.

[azasypkin]

@legrego it depends on which one will will be less effort and more reward: the new Core Encryption Service or authz.
We can't shy away from authz forever and need to tackle that in time to get it done and verified but it depends on priority from both teams.

If we have consumers eager to use the Core Encryption Service as soon as it's introduced, I'd go with it (high effort, but also high short-term reward). Even if we re-expose authorization APIs from the Core, it will likely take quite some time before anyone commits to migrating to Core APIs from the Security ones (medium effort, but potentially low short-term reward). Both are high-reward initiatives long-term though.

[pgayvallet]

but between those two, which would you like to see us focus on first?

My gut feeling would be that the encryption service may be a higher effort/reward ratio, but the security team may have more visibility than we do on that question

[TinaHeiligers]

@elastic/kibana-security the scope for phase 2 and 3 need to change a bit based on what code is single vs dual licensed.

Here are the results (focus being on Platform only): https://docs.google.com/spreadsheets/d/1-oiYNxiAWNwBP5gnNoJP6GJ-5hs4kozAPyeYsqSC3Jg/edit?usp=sharing

[TinaHeiligers]

Phase 3 on hold, pending 9.0.0

Not on hold:

• promote migration to core.security to reduce dependency on the security plugin (email, contributors' newsletter - pending on Core)
• authz refactor discussion (pending on Platform Security)