[DataViews] GetIndices should allow me to get non-hidden indices

[stratoula]

Describe the feature:

Not sure if this is a bug or a feature but is important for ES|QL.

We are suggestiing indices in the from command using this api

const indices = await dataViews.getIndices({
    showAllIndices: false,
    pattern: '*',
    isRollupIndex: () => false,
  });

and then we mark them as hidden if index.name.startsWith('.') but this is wrong. Not all the indices starting with . are hidden so we need a way to receive all the non-hidden (system indices) and remove the wrong check for the dot.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

I created a hidden index

{
  "settings": {
    "index.hidden": true
  },
  "mappings": {
    "properties": {
      "field1": {
        "type": "text"
      },
      "field2": {
        "type": "keyword"
      }
    }
  }
}

POST /my_hidden_index/_doc/1
{
  "field1": "This is some text data",
  "field2": "keyword_value"
}

and removed the index.name.startsWith('.')

when trying out dataViews.getIndices

if was not part of the returned list, so it might be sufficient to remove this check to get all non-hidden indices?

[stratoula]

Hmmm this gives me this list

@ninoslavmiskovic are these indices safe to suggest?

[stratoula]

@kertal these are not being suggested in the Index management page so I am not expecting them to be suggested, right?

[jughosta]

these are not being suggested in the Index management page

They might be on Data Streams tab.

[stratoula]

Not all of them Jul

[kertal]

@kertal these are not being suggested in the Index management page so I am not expecting them to be suggested, right?

I had another look at it, and it seems most of those are tagged alias , e.g. .siem-signals-default

this seems to be an alias for the following index:

.internal.alerts-security.alerts-default-000001

which is appears to be a hidden index?

I'm not sure if this alias should be displayed? In the Index Management, it is not findable?

[stratoula]

It should not be suggested if it is an alias to a hidden index. Can the api work like that? Display only indices / aliases that are not hidden (exactly as the Index management page behaves when the Include hidden indices switch is off)

[kertal]

It should not be suggested if it is an alias to a hidden index. Can the api work like that? Display only indices / aliases that are not hidden (exactly as the Index management page behaves when the Include hidden indices switch is off)

I took a look, but it seems to me, this is currently not supported. If a near term solution is needed a workaround would be for the consumer to filter out those alias to a hidden index. Something like first create a list of not-alias indices, then add alias that point to indices that can be found in the non-alias indices list

[stratoula]

@kertal I would like to avoid this in the client side, the api should handle this for performance reasons. So I will wait for this to be prioritized cc @ninoslavmiskovic

[kertal]

. @kertal I would like to avoid this in the client side, the api should handle this for performance reasons. So I will wait for this to be prioritized cc @ninoslavmiskovic

I do agree that a server side solution is preferable 👍

[ninoslavmiskovic]

++ on not doing it on the client side.

I don't believe it is possible in ES to directly prevent users from creating aliases on hidden indices.

However, as a user, it is possible to be able to control access to hidden indices and alias creation through role-based access control (RBAC). E.g Restrict Index Access:, Aliases Privilege (https://www.elastic.co/guide/en/elasticsearch/reference/current/securing-aliases.html#index-alias-privileges), Custom Roles, Index-Level Security:

Especially the aliases' privileges are interesting

I guess you are a super user @stratoula - perhaps that is why you are able to create aliases with hidden indices.?

[kertal]

@elastic/kibana-management I'm interested in the logic behind "Include hidden indices" on Index Management, so the whole list is returned, and the filter is applied client side, on the "hidden" property, right? how does it work server side. thx!

[sabarasaba]

That is correct, we return all indices or datastreams and the pagination happens client side based on the hidden property as you mentioned. This hidden property is computed based on the settings.index.hidden attribute of that given index, you can find the code that deals with creating that response here.

[kertal]

@sabarasaba thx, this is very helpful!

[kertal]

No 100% sure, but it might be the case that we are already using the same code under the hood, so in theory the hidden property "just" needs to be made available to the consumers in UI

kibana/src/plugins/data_views/public/services/get_indices.ts

Lines 43 to 60 in 628994e

	export const getIndicesViaResolve = async ({
	http,
	pattern,
	showAllIndices,
	isRollupIndex,
	}: {
	http: HttpStart;
	pattern: string;
	showAllIndices: boolean;
	isRollupIndex: (indexName: string) => boolean;
	}) => {
	const encodedPattern = encodeURIComponent(pattern);
	return http
	.get<ResolveIndexResponse>(
	`/internal/index-pattern-management/resolve_index/${encodedPattern}`,
	{
	query: showAllIndices ? { expand_wildcards: 'all' } : undefined,
	}

FYI @mattkime (once back from PTO)

[mattkime]

@kertal @stratoula

It should not be suggested if it is an alias to a hidden index. Can the api work like that?

No, thats not how it works and it seems rather purposeful. Our overall state of appropriately marking indices as hidden isn't very thorough - at this point we should be to rely on being marked hidden instead of the leading . but thats not happening in all instances.

If aliases to hidden indices aren't hidden and should be, then the alias should be fixed. Aliases can be marked hidden independent of their underlying indices.

[kertal]

End of the year, finally time to have a closer look here

I did a quick POC how to simply filter out dot prefixed indices that are alias

#205276

This would enable the selection of dot prefixed indices, that ain't alias, and can be tested here

https://kertal-pr-205276-discover-esql-make-use-of-dot-prefixed-ind.kbndev.co/app/r/s/vFXr6

However this is just a workaround and I agree with @mattkime

If aliases to hidden indices aren't hidden and should be, then the alias should be fixed. Aliases can be marked hidden independent of their underlying indices.

So I think we should remove code that checks index.name.startsWith('.') to determine if an index is hidden. We are using this at many places in Kibana, and it's misleading, since it doesn't mean an index is hidden. And we should cleanup/evaluate indices with . prefix if they were meant to be hidden, and this should be fixed if the intention was so.