[ML] Add ML saved objects to Kibana saved objects management page #205177

qn895 · 2024-12-26T23:02:08Z

Summary

This PR adds the display of saved objects for anomaly detection jobs, data frame analytics jobs and trained models to the Kibana saved objects management page. Changes include:

Add ml-job saved object type into the allowed types. The allowed_type will exclude ml-job and ml-trained-model objects if missing ml canGetJobs capabilities. They are not hyper linked until new navigation routing is implemented.

Spaces without ML permission will not show ML saved objects

Screen.Recording.2025-01-02.at.15.48.27.mov

The Export {count} objects reflects only exportable saved objects.
User that canGetJobs but not canCreateJob can only inspect, and will not be able to share object to spaces

ml-job saved objects are not deletable. The Delete button will be disabled if any ML saved objects is selected, and a hint tooltip message will be shown.
ml-job saved objects are not exportable. The Export button will be disabled if any ML saved objects is selected, and a hint tooltip message will be shown.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

qn895 · 2024-12-26T23:14:38Z

/ci

qn895 · 2024-12-30T17:52:07Z

/ci

qn895 · 2024-12-30T21:21:41Z

/ci

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts

peteharverson · 2025-01-02T15:17:49Z

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts

+        if (obj.id.startsWith(AD_PREFIX)) {
+          const jobId = obj.id.slice(AD_PREFIX.length);
+          return {
+            path: `/app/management/insightsAndAlerting/jobsListLink/${jobId}`,


Passing the jobId in the link is wasted currently as we don't use that in the ML management page to restore a filter to the query bar. But I guess it makes more sense to wait till the new management page is ready and then edit this link to provide a filtered link to that page?

As discussed, I'm removing the links for all ML objects for now until we have the PR for management page ready

peteharverson · 2025-01-02T15:22:37Z

...lugins/saved_objects_management/public/management_section/objects_table/components/table.tsx

+        return (
+          <FormattedMessage
+            id="savedObjectsManagement.objectsTable.table.hasMlObjects.deleteDisabledTooltip"
+            defaultMessage="Machine learning objects can’t be deleted."


Suggest editing this to say something like Machine learning objects can only be deleted in the Machine Learning management page (last part of that message needs some work!).

Updated 1450c34 (#205177)

@szabosteve Follow-up on my slack message, would love to get some feedback for this one liner.

@qn895 @peteharverson This tooltip message works fine. We could maybe phrase it a bit more action-oriented, something along the lines of:

Navigate to the Machine Learning management page to delete machine learning objects.

That text sounds good to me. Thanks @szabosteve

As discussed, until the new management pages are in, let's change this to say Navigate to the Machine Learning pages to delete machine learning objects.

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts

jgowdyelastic · 2025-01-02T15:57:40Z

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts

+          const jobId = obj.id.slice(AD_PREFIX.length);
+          return {
+            path: `/app/management/insightsAndAlerting/jobsListLink/${jobId}`,
+            uiCapabilitiesPath: 'ml.canGetJobs',


Is it possible to add additional capabilities checks to control what actions a user can perform and whether or not the saved objects are listed?

I think should not list the SOs if the user does not have the minimum read permissions and should only be able to edit them if they have write permissions, e.g. for AD SOs:
canGetJobs AD job saved objects are listed but are read only.
canGetJobs && canCreateJobs AD job saved objects are listed and can be edited

Thanks! Added capabilities check here and here

…n-management

qn895 · 2025-01-02T21:57:15Z

/ci

jgowdyelastic · 2025-01-03T09:33:44Z

src/plugins/saved_objects_management/server/routes/get_allowed_types.ts

+          capabilityPath: 'ml.*',
+        });
+        const canSeeMLJobs =
+          mlCapabilities.ml.canGetJobs && mlCapabilities.ml.canGetDataFrameAnalytics;


We can't group the two ML job types together, AD and DFA need to be separate.
In a serverless environment where the DFA feature is disabled, canGetDataFrameAnalytics will always be false. Same for AD feature.

Looking at what is contained inside mlJobType below, I'm not sure if this will be possible to filter these different job types server side.

A possible solution would be to change this check to an || and add some additional filtering on the client side when displaying table. The client side will also have the capabilities available.

jgowdyelastic · 2025-01-03T09:35:36Z

Is it possible to hide this callout with a link to the "copy to spaces" flyout?

…n-management

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts

qn895 · 2025-01-03T22:50:04Z

/ci

qn895 · 2025-01-07T05:44:22Z

/ci

qn895 · 2025-01-07T15:27:57Z

/ci

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx

qn895 · 2025-01-07T18:54:28Z

/ci

qn895 · 2025-01-08T03:38:53Z

/ci

elasticmachine · 2025-01-08T05:13:09Z

Pinging @elastic/ml-ui (:ml)

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx

…n-management

elasticmachine · 2025-01-10T00:55:37Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 885662f

Failed CI Steps

FTR Configs #86

Test Failures

[job] [logs] FTR Configs #86 / alerting api integration security and spaces enabled Alerts - Group 1 alerts backfill rule runs ad hoc backfill task should run all execution sets of a scheduled backfill and correctly generate alerts

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`savedObjectsManagement`	101	102	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`savedObjectsManagement`	81.8KB	83.8KB	+1.9KB
`spaces`	257.0KB	257.1KB	+26.0B
total			+1.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`savedObjectsManagement`	19.8KB	20.3KB	+471.0B

History

💔 Build #265793 failed e40c0ec
💔 Build #265371 failed c8ca28a
💚 Build #264992 succeeded 63a7536
💔 Build #264936 failed c9c404d
💔 Build #264844 failed 51be700
💔 Build #264594 failed 4b00d37

cc @qn895

peteharverson

Latest changes LGTM

afharo

As a general comment, I'm not a big fan of leaking ML-specific requirements into the SOM code-base.

I wonder if there are any options (onExport/onImport hooks, read-only flag, getInAppUrl) that we can leverage to register all these via the API and inside the ML plugin instead.

Also, please, remove the async from the SOM plugin's lifecycle methods 🙏

afharo · 2025-01-10T15:11:55Z

src/platform/plugins/shared/saved_objects_management/server/plugin.ts


  constructor(private readonly context: PluginInitializerContext) {
    this.logger = this.context.logger.get();
  }

-  public setup({ http, capabilities }: CoreSetup) {
+  public async setup({ http, capabilities, getStartServices }: CoreSetup) {


The use of async lifecycle methods in plugins is deprecated and highly discouraged. It looks like it's not necessary anyway.

afharo · 2025-01-10T15:14:52Z

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx

+          'savedObjectsManagement.objectsTable.table.deleteMLJobsAndModelsDisabledTooltip',
+          {
+            defaultMessage:
+              'Navigate to the Machine Learning management pages to delete machine learning jobs and trained models.',


Can we use management.getInAppUrl in the SO definition to take the users to the ML management pages instead?

Yes that's the plan. This is dependent on another PR that we will have merge soon. I'll add a todo to make notes.

jgowdyelastic · 2025-01-10T16:21:00Z

...red/saved_objects_management/public/management_section/objects_table/saved_objects_table.tsx

+        let filteredSavedObjects = resp.saved_objects;
+        const mlCapabilities = this.props.applications.capabilities.ml as Record<string, boolean>;
+
+        const mlFilters: Record<string, boolean> = {


In the original management page, if only the nlp feature is disabled, we do not list pytorch models, but we do list models from dfa jobs.

With this PR, we are still listing pytorch models.

The filtering code is here:

kibana/x-pack/platform/plugins/shared/ml/server/routes/trained_models.ts

Line 40 in de3bd71

export function filterForEnabledFeatureModels<

It would be good to replicate this behaviour as I don't think elser should be listed if nlp is disabled.
However, it will not be easy as we don't have access to the underlying trained models and I'm not sure we should be loading them here.
We also do not currently have a serverless offering where nlp is disabled. So this is low risk.

jgowdyelastic · 2025-01-10T16:40:11Z

src/platform/plugins/shared/saved_objects_management/common/constants/ml_saved_object_types.ts

+ * License v3.0 only", or the "Server Side Public License, v 1".
+ */
+
+export const ML_SAVED_OBJECT_TYPES = new Set(['ml-job', 'ml-trained-model']);


I wonder if we should move the constants declared here to a package and also include this Set.

In this PR ml-job and ml-trained-model are used a few times and it would be safer if they all came from the same place.

jgowdyelastic · 2025-01-10T16:44:52Z

...red/saved_objects_management/public/services/actions/share_saved_objects_to_space_action.tsx

@@ -43,6 +43,15 @@ export class ShareToSpaceSavedObjectsManagementAction extends SavedObjectsManage
        !this.actionContext ||
        !!this.actionContext.capabilities.savedObjectsManagement.shareIntoSpace;
      const { namespaceType, hiddenType } = object.meta;
+
+      if (object?.type && object.type.startsWith('ml-')) {
+        const hasMlShareCapabilities =


this check is only looking at AD capabilities. If I disable AD in my yml file with xpack.ml.ad.enabled: false I can no longer share trained models or DFA jobs.

jgowdyelastic · 2025-01-10T16:48:19Z

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx

    const anySelected = selectedSavedObjects.length > 0;
    const allHidden =
      anySelected && selectedSavedObjects.every(({ meta: { hiddenType } }) => hiddenType);
+
+    const deleteTooltip = () => {


There's a sneaky delete button at the top of the inspect saved object flyout.

jeramysoucy · 2025-01-10T17:10:54Z

...red/spaces/public/share_saved_objects_to_space/components/share_to_space_flyout_internal.tsx

@@ -378,7 +378,8 @@ export const ShareToSpaceFlyoutInternal = (props: ShareToSpaceFlyoutProps) => {
        enableCreateCopyCallout &&
        spaces.length > 1 &&
        savedObjectTarget.namespaces.length === 1 &&
-        !arraysAreEqual(savedObjectTarget.namespaces, [ALL_SPACES_ID]);
+        !arraysAreEqual(savedObjectTarget.namespaces, [ALL_SPACES_ID]) &&
+        !savedObjectTarget.type.startsWith('ml');


Nit: The only concern I have here is that we're string matching to catch all ML SO types. This appears to work, but I am unfamiliar with all ML SO types. Could we make this more explicit, something like an "isMLType" utility function that we can ensure covers only, but all of, the ML SO types explicitly?

Add ML saved objects to management page

85b1e57

qn895 self-assigned this Dec 26, 2024

qn895 requested review from alvarezmelissa87 and peteharverson December 26, 2024 23:02

qn895 added v9.0.0 release_note:feature Makes this part of the condensed release notes :ml labels Dec 26, 2024

qn895 added 2 commits December 30, 2024 14:42

Update UI

349b842

Remove console

31c5c56

qn895 force-pushed the ml-kbn-saved-objs-in-management branch from 2543291 to 31c5c56 Compare December 30, 2024 21:13

peteharverson reviewed Jan 2, 2025

View reviewed changes

jgowdyelastic reviewed Jan 2, 2025

View reviewed changes

qn895 added 5 commits January 2, 2025 11:16

Merge remote-tracking branch 'upstream/main' into ml-kbn-saved-objs-i…

bc3c7b4

…n-management

Add capabilities check within allowed_types

1450c34

Add capabilities check

ffc1890

Update types

d78992f

Update translation

80ca324

[CI] Auto-commit changed files from 'node scripts/notice'

25964a4

jgowdyelastic reviewed Jan 3, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into ml-kbn-saved-objs-i…

63044c1

…n-management

peteharverson reviewed Jan 3, 2025

View reviewed changes

x-pack/platform/plugins/shared/ml/server/saved_objects/saved_objects.ts Outdated Show resolved Hide resolved

qn895 added 3 commits January 3, 2025 11:54

Disable flyout

6a83915

Update atexts

5c911c6

Fix display name

0211426

Update tests

51be700

Fix test & update text

602c73f

peteharverson reviewed Jan 7, 2025

View reviewed changes

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx Show resolved Hide resolved

peteharverson reviewed Jan 7, 2025

View reviewed changes

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx Outdated Show resolved Hide resolved

Update texts for deleting and exporting

c9c404d

Update tests

63a7536

qn895 marked this pull request as ready for review January 8, 2025 05:13

qn895 requested review from a team as code owners January 8, 2025 05:13

peteharverson reviewed Jan 8, 2025

View reviewed changes

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx Outdated Show resolved Hide resolved

...shared/saved_objects_management/public/management_section/objects_table/components/table.tsx Outdated Show resolved Hide resolved

Fix export all

c8ca28a

jeramysoucy self-requested a review January 9, 2025 18:31

qn895 added the backport:version Backport to applied version labels label Jan 9, 2025

qn895 added 3 commits January 9, 2025 13:32

Update delete texts for deletion

ab6f7fd

Merge remote-tracking branch 'upstream/main' into ml-kbn-saved-objs-i…

e40c0ec

…n-management

Update tests

885662f

peteharverson approved these changes Jan 10, 2025

View reviewed changes

afharo requested changes Jan 10, 2025

View reviewed changes

jgowdyelastic reviewed Jan 10, 2025

View reviewed changes

jeramysoucy reviewed Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Add ML saved objects to Kibana saved objects management page #205177

[ML] Add ML saved objects to Kibana saved objects management page #205177

qn895 commented Dec 26, 2024 •

edited

Loading

qn895 commented Dec 26, 2024

qn895 commented Dec 30, 2024

qn895 commented Dec 30, 2024

peteharverson Jan 2, 2025

qn895 Jan 7, 2025

peteharverson Jan 2, 2025

qn895 Jan 2, 2025

szabosteve Jan 3, 2025

peteharverson Jan 3, 2025

peteharverson Jan 7, 2025

jgowdyelastic Jan 2, 2025

qn895 Jan 2, 2025

qn895 commented Jan 2, 2025

jgowdyelastic Jan 3, 2025 •

edited

Loading

jgowdyelastic commented Jan 3, 2025

qn895 commented Jan 3, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 8, 2025

elasticmachine commented Jan 8, 2025

elasticmachine commented Jan 10, 2025

peteharverson left a comment

afharo left a comment •

edited

Loading

afharo Jan 10, 2025

afharo Jan 10, 2025

qn895 Jan 13, 2025

jgowdyelastic Jan 10, 2025 •

edited

Loading

jgowdyelastic Jan 10, 2025 •

edited

Loading

jgowdyelastic Jan 10, 2025

jgowdyelastic Jan 10, 2025

jeramysoucy Jan 10, 2025

[ML] Add ML saved objects to Kibana saved objects management page #205177

Are you sure you want to change the base?

[ML] Add ML saved objects to Kibana saved objects management page #205177

Conversation

qn895 commented Dec 26, 2024 • edited Loading

Summary

Checklist

Identify risks

qn895 commented Dec 26, 2024

qn895 commented Dec 30, 2024

qn895 commented Dec 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qn895 commented Jan 2, 2025

jgowdyelastic Jan 3, 2025 • edited Loading

Choose a reason for hiding this comment

jgowdyelastic commented Jan 3, 2025

qn895 commented Jan 3, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 7, 2025

qn895 commented Jan 8, 2025

elasticmachine commented Jan 8, 2025

elasticmachine commented Jan 10, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

peteharverson left a comment

Choose a reason for hiding this comment

afharo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

jgowdyelastic Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qn895 commented Dec 26, 2024 •

edited

Loading

jgowdyelastic Jan 3, 2025 •

edited

Loading

afharo left a comment •

edited

Loading

jgowdyelastic Jan 10, 2025 •

edited

Loading

jgowdyelastic Jan 10, 2025 •

edited

Loading