lens esql generation #196049

ppisljar · 2024-10-14T05:54:52Z

Summary

updates lens so it uses ESQL when requesting data from elasticsearch rather than DSL when that is possible.

nothing should change for the user, everything should work exactly as before

but when going to inspector you will be able to see that in some cases we will send out ESQL request.

the feature is only enabled when the lens.enable_esql feature flag is enabled.

at the moment this covers very few cases, but rather builds the necessary infrastructure so we can keep improving this over time:

user has to be in UTC timezone
user needs to use set of lens operations that are supported

how to test:

put this in your kibana.dev.yaml:

feature_flags.overrides:
  lens.enable_esql: true

go to advanced settings and set timezone to UTC
go to lens, configure a chart (for example a metric chart showing average of products.price
go to inspector -> requests and you can see that we sent out ESQL query
visually nothing should change

todo:

make sure CI is green when the feature flag is enabled

…nsesql

src/plugins/data/common/datatable_utilities/datatable_utilities_service.ts

stratoula

I have started testing this. I find the transitions on date histogram a bit weird. How do the rest feel about it?

.../platform/plugins/shared/lens/public/datasources/form_based/operations/definitions/count.tsx

.../plugins/shared/lens/public/datasources/form_based/operations/definitions/date_histogram.tsx

stratoula

This is very close, found 2 more bugs concerning the date histogram

.../plugins/shared/lens/public/datasources/form_based/operations/definitions/date_histogram.tsx

stratoula

Another one related to filtering

stratoula · 2025-01-15T08:44:44Z

src/platform/plugins/shared/data/public/actions/filters/create_filters_from_value_click.ts

Clicking on a metric with count(*) creates a wrong filter

fixed in c9f594c

fixed now, the click is not enabled if the metric is showing count of records

It should be enabled here though right?

I am clicking the metric but I dont get a filter

markov00

Everything fine for me, thanks Peter for this.

dej611

I've left some code reviews.
Not major issues, but it would be nice to have them addressed. If not, please create a follow up issue to track them.

src/platform/plugins/shared/data/public/actions/filters/create_filters_from_value_click.ts

dej611 · 2025-01-20T11:26:37Z

src/platform/plugins/shared/data/public/actions/filters/create_filters_from_value_click.ts

+    return;
+  }
+  const column = table.columns[columnIndex];
+  const { indexPattern, sourceField, operationType, interval } = table.columns[columnIndex].meta


it would be nice to have a type guard here

x-pack/platform/plugins/shared/lens/public/datasources/form_based/to_esql.ts

Co-authored-by: Marco Liberati <[email protected]>

stratoula · 2025-01-22T09:33:40Z

I see this error in the kibana server Peter

stratoula · 2025-01-22T12:57:52Z

I see an error in the console when I drop Records "field"

yes something goes wrong with the xAxis label

stratoula · 2025-01-22T13:42:44Z

ok console error gone and x-axis working as expected. This is the only thing not working correctly #196049 (comment)

elasticmachine · 2025-01-22T14:52:09Z

💔 Build Failed

Buildkite Build
Commit: 317c93b

Failed CI Steps

Test Failures

[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is a date histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is a date histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is not date histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL handles an event when operation type is not date histogram
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL ignores event when sourceField is missing
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL ignores event when sourceField is missing
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL ignores event when value for rows is not provided
[job] [logs] Jest Tests #8 / createFiltersFromClickEvent createFilterESQL ignores event when value for rows is not provided

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	522	537	+15
`lens`	1754	1755	+1
total			+16

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`data`	2598	2602	+4
`expressions`	1769	1771	+2
total			+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	1.6MB	1.6MB	+6.6KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`data`	25	26	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	419.5KB	423.0KB	+3.5KB
`expressions`	101.4KB	101.7KB	+287.0B
total			+3.8KB

Unknown metric groups

API count

id	before	after	diff
`data`	3213	3218	+5
`expressions`	2241	2243	+2
total			+7

History

💔 Build #269165 failed 8766331
💔 Build #269145 failed bb61e3a
💔 Build #268680 failed 2a44aaa
💚 Build #268348 succeeded cfeaa80
💔 Build #267321 failed c9f594c

ppisljar added 2 commits October 14, 2024 08:17

lens esql generation

6a56775

handle missing options

02b4ce9

ppisljar force-pushed the poc/lensesql branch from ab81587 to 02b4ce9 Compare October 14, 2024 07:39

client side auto calculation

ca29b48

teresaalvarezsoler mentioned this pull request Oct 15, 2024

ESQL: have BUCKET produce empty buckets elastic/elasticsearch#111483

Open

ppisljar added 2 commits October 17, 2024 09:44

more features

6f2d487

cleanup

3e33117

markov00 mentioned this pull request Oct 17, 2024

Lens powered by ES|QL - technical spike #192075

Open

ppisljar added 6 commits October 22, 2024 10:35

add advanced setting and basic tests

cd2908e

basic input sanitizaton

31620f7

fix error

e181929

update kibana auto bucketing strategy

5c12195

use ENABLE_ESQL setting

6428efc

update sanitization

a097d5d

ppisljar force-pushed the poc/lensesql branch from a152a67 to a097d5d Compare October 24, 2024 07:11

ppisljar added 9 commits October 24, 2024 09:13

Merge branch 'main' into poc/lensesql

9854348

fix sanitization

1c310e8

Merge branch 'poc/lensesql' of github.com:ppisljar/kibana into poc/le…

8733660

…nsesql

field formatters

1ed9574

Merge branch 'main' into poc/lensesql

7ae6eb7

Merge branch 'main' into poc/lensesql

5d2b96d

labels and formatters

8c3258b

add feature flag

f07ab43

cleanup

e1e57b8

ppisljar added Team:Visualizations Visualization editors, elastic-charts and infrastructure release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Nov 12, 2024

ppisljar commented Nov 12, 2024

View reviewed changes

src/plugins/data/common/datatable_utilities/datatable_utilities_service.ts Outdated Show resolved Hide resolved

cleanup

3dcccc5

ppisljar force-pushed the poc/lensesql branch from 5243363 to 3dcccc5 Compare November 12, 2024 10:36

ppisljar added 2 commits January 9, 2025 10:01

fix for timezone

b5bb6f4

add test

3dfd811

stratoula reviewed Jan 9, 2025

View reviewed changes

.../platform/plugins/shared/lens/public/datasources/form_based/operations/definitions/count.tsx Outdated Show resolved Hide resolved

.../plugins/shared/lens/public/datasources/form_based/operations/definitions/date_histogram.tsx Show resolved Hide resolved

ppisljar added 2 commits January 9, 2025 11:16

disable for ranges

9fae536

fix bug with datehistogram

fadb2f8

ppisljar force-pushed the poc/lensesql branch from a886883 to fadb2f8 Compare January 13, 2025 10:46

ppisljar added 2 commits January 13, 2025 11:46

Merge branch 'main' into poc/lensesql

895a92d

fix test

7dfebfc

markov00 self-requested a review January 14, 2025 08:36

stratoula reviewed Jan 15, 2025

View reviewed changes

.../plugins/shared/lens/public/datasources/form_based/operations/definitions/date_histogram.tsx Show resolved Hide resolved

stratoula reviewed Jan 15, 2025

View reviewed changes

ppisljar added 4 commits January 15, 2025 12:24

pass used_interval

45f221d

fix filtering

c9f594c

fix filtering

6933945

Merge branch 'main' into poc/lensesql

cfeaa80

markov00 approved these changes Jan 20, 2025

View reviewed changes

dej611 reviewed Jan 20, 2025

View reviewed changes

ppisljar and others added 2 commits January 21, 2025 09:02

Apply suggestions from code review

2a44aaa

Co-authored-by: Marco Liberati <[email protected]>

apply review feedback

d0d76e8

botelastic bot added the Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) label Jan 22, 2025

dej611 approved these changes Jan 22, 2025

View reviewed changes

apply review feedback

c780d9f

apply review feedback

aa58127

ppisljar force-pushed the poc/lensesql branch from bb61e3a to aa58127 Compare January 22, 2025 10:08

Merge branch 'main' into poc/lensesql

8766331

apply review feedback

317c93b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lens esql generation #196049

lens esql generation #196049

ppisljar commented Oct 14, 2024 •

edited

Loading

stratoula left a comment

stratoula left a comment

stratoula left a comment

stratoula Jan 15, 2025

ppisljar Jan 15, 2025

markov00 Jan 17, 2025

stratoula Jan 22, 2025

markov00 left a comment

dej611 left a comment

dej611 Jan 20, 2025

stratoula commented Jan 22, 2025

stratoula commented Jan 22, 2025 •

edited

Loading

stratoula commented Jan 22, 2025

elasticmachine commented Jan 22, 2025 •

edited

Loading

API count

lens esql generation #196049

Are you sure you want to change the base?

lens esql generation #196049

Conversation

ppisljar commented Oct 14, 2024 • edited Loading

Summary

how to test:

todo:

stratoula left a comment

Choose a reason for hiding this comment

stratoula left a comment

Choose a reason for hiding this comment

stratoula left a comment

Choose a reason for hiding this comment

stratoula Jan 15, 2025

Choose a reason for hiding this comment

ppisljar Jan 15, 2025

Choose a reason for hiding this comment

markov00 Jan 17, 2025

Choose a reason for hiding this comment

stratoula Jan 22, 2025

Choose a reason for hiding this comment

markov00 left a comment

Choose a reason for hiding this comment

dej611 left a comment

Choose a reason for hiding this comment

dej611 Jan 20, 2025

Choose a reason for hiding this comment

stratoula commented Jan 22, 2025

stratoula commented Jan 22, 2025 • edited Loading

stratoula commented Jan 22, 2025

elasticmachine commented Jan 22, 2025 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

ppisljar commented Oct 14, 2024 •

edited

Loading

stratoula commented Jan 22, 2025 •

edited

Loading

elasticmachine commented Jan 22, 2025 •

edited

Loading