Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ES|QL] Dashboard variables #202875

Open
wants to merge 187 commits into
base: main
Choose a base branch
from
Open

Conversation

stratoula
Copy link
Contributor

@stratoula stratoula commented Dec 4, 2024

Summary

Closes #203967

Supports dashboard variables in ES|QL charts.

This PR introduces the first phase of ES|QL controls. In this phase:

  • the flow starts from Lens ES|QL editor (and no vice-versa, this will happen on a later phase after we discuss some technical details with ES)
  • it is only available for dashboards (we want to include them in other apps as Discover but this is the next phase driven by the presentation team)
  • it supports variables for intervals, fields and values. I haven't added support for functions. I am going to do it after this PR being merged (there are some business questions I want to answer first)

For more info check this deck

meow

Implementation details

  • There is a new service, the ESQLVariables service that is responsible for ES|QL variables. I isolated this to a new plugin owned by the ES|QL team for cleaner code and for avoiding circular dependencies
  • A new ESQL_CONTROL type got created. It follows the exact same logic as the rest controls. No changes in the architecture here.
  • The creation of the controls (the control forms) have been added in the esql plugin.
  • Lens has small changes:
    • The support of variables in the textBased datasource
    • Two callbacks needed to be called after the creation / cancellation of an ES|QL control

Types of ES|QL variables

We have 2 types:

  • Static Values (the user gives a list of values with his own responsibility). As the flow starts from the editor we can identify what they most possibly want to do and we give the user some options but they have the freedom to do as they want. A basic validation has been added too.
  • Values from an ES|QL query (the user gives an ES|QL query that generates the values). As the flow starts from the editor we can suggest a query for the users but they can always change it as they wish.
image

Example of a control creation from the editor

meow

Release note

ES|QL charts now allow the creation of controls in dashboards. You can control a part of the query such as a field, an interval or a value.

Checklist

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Unit or functional tests were updated or added to match the most common scenarios
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

@stratoula stratoula changed the title Esql controls [ES|QL] Dashboard variables Dec 4, 2024
@stratoula stratoula mentioned this pull request Dec 4, 2024
7 tasks
@stratoula
Copy link
Contributor Author

@ThomThomson @dej611 can you please continue with your review? Thanx!

@ThomThomson
Copy link
Contributor

@nreese will be finishing the review from the Presentation team side!

@nreese
Copy link
Contributor

nreese commented Jan 22, 2025

@nreese will be finishing the review from the Presentation team side!

reviewing today

Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really good. Thank you for moving the ESQLService calls out of controls and dashboard plugins and instead just publishing ESQL variables via a publishing subject.

Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kibana-presentation changes LGTM. ESQL variables is a powerful feature. Can't wait to see how these are used.
code review, tested in chrome

@elastic elastic deleted a comment from elasticmachine Jan 23, 2025
Copy link
Contributor

@dej611 dej611 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did another review pass and left some code comments.
I am a bit concerned about the bundle size increase here:
Screenshot 2025-01-23 at 15 29 00

@@ -519,6 +586,9 @@ export function LensEditConfigurationFlyout({
isDisabled={false}
allowQueryCancellation
isLoading={isVisualizationLoading}
onSaveControl={onSaveControl}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This component is becoming a props beast. Perhaps, as a follow up, would be worth to explore how to use the context for this kind of things and composition to enhance its features.

@@ -58,7 +62,12 @@ function getSearchContext(parentApi: unknown) {
timeRange$: new BehaviorSubject(undefined),
};

const esqlVariables = apiPublishesESQLVariables(parentApi)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move this into internalApi?
Type guards like these are performed at initialization level and the logic here should just use them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marco is this what you had in mind? d3fde2a

Comment on lines 131 to 176
const panel = useMemo(() => {
if (!panelId) {
return;
}
return dashboardPanels?.[panelId] as {
updateAttributes: (attributes: TypedLensSerializedState['attributes']) => void;
onEdit: () => Promise<void>;
};
}, [dashboardPanels, panelId]);

const onSaveControl = useCallback(
async (controlState: Record<string, unknown>, updatedQuery: string) => {
if (!panelId) {
return;
}

// add a new control
controlGroupApi?.addNewPanel({
panelType: 'esqlControl',
initialState: {
...controlState,
id: uuidv4(),
},
});
if (panel && updatedQuery) {
panel.updateAttributes({
...attributes,
state: {
...attributes.state,
query: { esql: updatedQuery },
needsRefresh: true,
},
});
// open the edit flyout to continue editing
await panel.onEdit();
}
},
[attributes, controlGroupApi, panel, panelId]
);

const onCancelControl = useCallback(() => {
closeFlyout?.();
if (panel) {
panel.onEdit();
}
}, [closeFlyout, panel]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would really like to not leak any embeddable specific logic in here, nor dashboard specific one.
It would be great to lift up all this logic outside of the component.
I understand this is already in an advanced phase, so I'd propose to move all this logic into a useESQLVariables(dashboardApi, panelId) hook in another file for now.
As a follow up perhaps all of this should live somewhere else outside of this component.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done d2dbebb

Comment on lines 127 to 136
const onVariableNameChange = useCallback(
(e: { target: { value: React.SetStateAction<string> } }) => {
let text = String(e.target.value).replace('?', '');
if (text.charAt(0) === '_') {
text = text.substring(1);
}
setVariableName(text);
},
[]
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no validation on the variable name and it cause both ESQL exceptions and also unfixable lens configuration until you don't remove completely the control.
If you try to remove the wrong variable from the ESQL query you are not allow to save it and the error is unrecoverable if you don't remove the control from the dashboard first.

This happens for example if you use a variable name that contains white spaces or that starts with a /.
You should harden the validation and blocks or disallow using special characters, or, if possible, escape it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanx Marco, done here 031386f and added a test too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another issue I've found is that if I manually enter a variable name (that is not available, not created before) in the query I get the error but no explanation in the editor at all. Even clicking to View problem doesn't bring me anywhere
Screenshot 2025-01-23 at 20 15 25
Screenshot 2025-01-23 at 20 18 24

Copy link
Contributor Author

@stratoula stratoula Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is irrelavant with this PR (this PR doesnt add variables support, we did that in previous PRs.

I will create an issue to track it on the kibana side, the server validation wrong highlight seems to be a bug, but again irrelevant with this PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done #208142

Copy link
Contributor Author

@stratoula stratoula Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems as a bug in ES, this is why our highlighter doesnt work as expected. I pinged the folks.

Update: Yes it is, I created an issue on their repo elastic/elasticsearch#120778

@stratoula
Copy link
Contributor Author

@dej611 thanx a lot for the review, I addressed everything. About the async bundle I dont see this to create any problems and the sync bundle doesnt have any significant increat but I will investigate in a new PR if we can do something better here

@stratoula stratoula requested a review from markov00 January 24, 2025 09:18
Comment on lines 70 to 72
const esqlVariables$ = apiPublishesESQLVariables(parentApi)
? parentApi.esqlVariables$
: undefined;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect here thanks.
The only change I would ask is this change:

Suggested change
const esqlVariables$ = apiPublishesESQLVariables(parentApi)
? parentApi.esqlVariables$
: undefined;
const esqlVariables$ = buildObservableVariable(apiPublishesESQLVariables(parentApi) ? parentApi.esqlVariables$ : undefined);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to tweak it a bit but here it is 014cba1

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
cloudSecurityPosture 689 774 +85
controls 383 472 +89
dashboard 696 768 +72
data 522 607 +85
dataVisualizer 731 816 +85
discover 1014 1099 +85
esql 122 230 +108
esqlDataGrid 414 499 +85
eventAnnotationListing 682 767 +85
infra 1246 1331 +85
investigateApp 581 666 +85
lens 1754 1842 +88
lists 429 430 +1
logsExplorer 611 696 +85
maps 1276 1361 +85
ml 2166 2229 +63
observability 1150 1235 +85
observabilityAIAssistantApp 419 504 +85
securitySolution 6609 6610 +1
slo 898 983 +85
stackAlerts 203 288 +85
unifiedHistogram 197 282 +85
unifiedSearch 400 401 +1
total +1698

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/es-query 210 211 +1
@kbn/esql-editor 12 14 +2
@kbn/esql-utils 77 82 +5
@kbn/esql-validation-autocomplete 195 205 +10
@kbn/esql-variables-types - 7 +7
@kbn/monaco 135 138 +3
controls 131 133 +2
esql 15 29 +14
expressions 1769 1772 +3
total +47

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
controls 491.3KB 494.9KB +3.6KB
dashboard 622.7KB 622.8KB +152.0B
esql 214.7KB 407.6KB +192.9KB
lens 1.6MB 1.6MB +3.0KB
maps 3.1MB 3.1MB +1.0B
total +199.6KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
esql 0 2 +2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
controls 11.0KB 11.3KB +269.0B
data 419.5KB 420.2KB +738.0B
esql 7.2KB 9.4KB +2.2KB
kbnUiSharedDeps-srcJs 3.5MB 3.5MB +1.7KB
savedSearch 11.6KB 11.6KB +45.0B
total +4.9KB
Unknown metric groups

API count

id before after diff
@kbn/es-query 271 272 +1
@kbn/esql-editor 30 36 +6
@kbn/esql-utils 86 95 +9
@kbn/esql-validation-autocomplete 207 217 +10
@kbn/esql-variables-types - 8 +8
@kbn/monaco 135 138 +3
controls 135 137 +2
esql 32 50 +18
expressions 2241 2244 +3
total +60

async chunk count

id before after diff
esql 4 5 +1

ESLint disabled line counts

id before after diff
controls 5 6 +1

References to deprecated APIs

id before after diff
controls 15 18 +3
esql 1 2 +1
total +4

Total ESLint disabled count

id before after diff
controls 5 6 +1

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels Feature:Dashboard Dashboard related features Feature:ES|QL ES|QL related features in Kibana Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:large Large Level of Effort release_note:feature Makes this part of the condensed release notes Team:ESQL ES|QL related features in Kibana Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ES|QL Controls implementation