Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[elasticsearch] Add the local query parameter when fetching cluster state #36586

Merged
merged 5 commits into from
Sep 22, 2023
Merged

[elasticsearch] Add the local query parameter when fetching cluster state #36586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7912fed
[elasticsearch] Add the local query parameter when fetching cluster s…
miltonhultgren Sep 13, 2023
9fa8d2b
Fix lint issues
miltonhultgren Sep 14, 2023
62a469c
Merge branch 'main' into es-cluster-state-local
miltonhultgren Sep 18, 2023
9ec5dda
Merge branch 'main' into es-cluster-state-local
miltonhultgren Sep 21, 2023
2389a77
Merge branch 'main' into es-cluster-state-local
miltonhultgren Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions metricbeat/module/elasticsearch/elasticsearch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,18 +170,20 @@ func getNodeName(http *helper.HTTP, uri string) (string, error) {
Nodes map[string]interface{} `json:"nodes"`
}{}

json.Unmarshal(content, &nodesStruct)
err = json.Unmarshal(content, &nodesStruct)
if err != nil {
return "", err
}

// _local will only fetch one node info. First entry is node name
for k := range nodesStruct.Nodes {
return k, nil
}
return "", fmt.Errorf("No local node found")
return "", fmt.Errorf("no local node found")
}

func getMasterName(http *helper.HTTP, uri string) (string, error) {
// TODO: evaluate on why when run with ?local=true request does not contain master_node field
content, err := fetchPath(http, uri, "_cluster/state/master_node", "")
content, err := fetchPath(http, uri, "_cluster/state/master_node", "local=true")
if err != nil {
return "", err
}
Expand All @@ -190,7 +192,10 @@ func getMasterName(http *helper.HTTP, uri string) (string, error) {
MasterNode string `json:"master_node"`
}{}

json.Unmarshal(content, &clusterStruct)
err = json.Unmarshal(content, &clusterStruct)
if err != nil {
return "", err
}

return clusterStruct.MasterNode, nil
}
Expand Down Expand Up @@ -236,7 +241,10 @@ func GetNodeInfo(http *helper.HTTP, uri string, nodeID string) (*NodeInfo, error
Nodes map[string]*NodeInfo `json:"nodes"`
}{}

json.Unmarshal(content, &nodesStruct)
err = json.Unmarshal(content, &nodesStruct)
if err != nil {
return nil, err
}

// _local will only fetch one node info. First entry is node name
for k, v := range nodesStruct.Nodes {
Expand Down Expand Up @@ -283,11 +291,11 @@ func GetLicense(http *helper.HTTP, resetURI string) (*License, error) {
// GetClusterState returns cluster state information.
func GetClusterState(http *helper.HTTP, resetURI string, metrics []string) (mapstr.M, error) {
clusterStateURI := "_cluster/state"
if metrics != nil && len(metrics) > 0 {
if len(metrics) > 0 {
clusterStateURI += "/" + strings.Join(metrics, ",")
}

content, err := fetchPath(http, resetURI, clusterStateURI, "")
content, err := fetchPath(http, resetURI, clusterStateURI, "local=true")
if err != nil {
return nil, err
}
Expand All @@ -310,7 +318,7 @@ func GetClusterSettings(http *helper.HTTP, resetURI string, includeDefaults bool
queryParams = append(queryParams, "include_defaults=true")
}

if filterPaths != nil && len(filterPaths) > 0 {
if len(filterPaths) > 0 {
filterPathQueryParam := "filter_path=" + strings.Join(filterPaths, ",")
queryParams = append(queryParams, filterPathQueryParam)
}
Expand Down Expand Up @@ -451,7 +459,7 @@ func GetMasterNodeID(http *helper.HTTP, resetURI string) (string, error) {
return "", err
}

for nodeID, _ := range response.Nodes {
for nodeID := range response.Nodes {
return nodeID, nil
}

Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/elasticsearch/shard/shard.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ func init() {
}

const (
statePath = "/_cluster/state/version,nodes,master_node,routing_table"
statePath = "/_cluster/state/version,nodes,master_node,routing_table?local=true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we miss data in a scope: cluster scenario where we only have one metricbeat process ? i'd expect local=true to be set only when scope: node

Copy link
Contributor Author

@miltonhultgren miltonhultgren Sep 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While developing this, I compared the results of the API with and without this flag and they were the same.
Maybe the result could differ between types of nodes, like a master nodes reply versus a data nodes reply, that I didn't compare. I just looked at if a data nodes reply changed between the versions (with and without flag).

@DaveCTurner Can you confirm if that's always the case?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No (or at least this change doesn't make a meaningful difference in that regard). The ?local=true parameter means "give me your local copy of the entire cluster state rather than forwarding this request to the elected master" but you get the same state either way (in fact sometimes you'll get a slightly fresher state with ?local=true, and sometimes ?local=true will succeed when ?local=false would fail, so this seems like the right change)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"No" as in:

Would we miss data in a scope: cluster scenario where we only have one metricbeat process ?

No

Can you confirm if that's always the case?

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the insight!

)

// MetricSet type defines all fields of the MetricSet
Expand Down