[Discover] Summary column should not filter out mapped fields which start with _ in Observability profile

[jughosta]

Originally reported via https://discuss.elastic.co/t/switch-between-the-document-and-summary-columns-in-kibana-discover/372063

Kibana version: 8.16

Notice that Summary column is empty:

To reproduce:

• Enable Observability navigation with xpack.spaces.experimental.forceSolutionVisibility: true and switching to Observability solution view via Space settings
• Add some logs data or run node scripts/synthtrace simple_logs.ts --clean
• Index this sample document:

PUT logs-100
POST logs-100/_doc
{
  "@timestamp": "2024-12-18T02:26:56.033Z",
  "__level": "info"
}

• Create logs-* data view in Discover and observe the issue with Summary column.
  (if it the sample document is not shown, adjust the time range to include it or filter for __level field exists)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[jughosta]

We could handle meta fields better instead of excluding any fields starting with _:

kibana/packages/kbn-discover-utils/src/data_types/logs/constants.ts

Line 50 in 6b63f7f

'_', // Filter fields like '_id', '_score'

or we could remove the whole filtering logic so it works as it was in Discover, before Summary replaced Document:

kibana/packages/kbn-discover-utils/src/data_types/logs/constants.ts

Lines 49 to 63 in 6b63f7f

	export const FILTER_OUT_FIELDS_PREFIXES_FOR_CONTENT = [
	'_', // Filter fields like '_id', '_score'
	'@timestamp',
	'agent.',
	'elastic_agent.',
	'data_stream.',
	'ecs.',
	'host.',
	'container.',
	'cloud.',
	'kubernetes.',
	'orchestrator.',
	'log.',
	'service.',
	];

Which way would be more preferable?

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[gbamparop]

Which way would be more preferable?

We discussed this with the team and decided to go with the first approach to exclude the relevant metadata fields instead of filtering out all the fields starting with underscore.