Experiment: Generic ElasticSearch query

[rsevilla87]

Type of change

• Refactor
• New feature
• Bug fix
• Optimization
• Documentation Update

Description

Adding a new generic_search method to retrieve data from arbitrary queries, arbitrary buckets and arbitrary aggregations. Example of result after running a testing script against our aws dev instance

https://gist.github.com/rsevilla87/ea63357d1243e1614a2c276eb09c032c

Some benefits:

• Just one query get data broken down by a metadata field.
• Permits nested buckets/metrics, they can be specified with a dot notation, example metadata.ocpMajorVersion.keyword

It's major downside can be the format of the data returned, it's a single JSON document with lots of nested fields, (one per term/metric aggregation. Good thing is that the value of the metric aggregation is returned at the lowest level and the key names are predictable, in this case:

 "aggregations": {
    "metadata.platform.keyword": {
      "buckets": [
        {
          "key": "BareMetal",
          "metadata.sdnType.keyword": {
            "buckets": [
              {
                "key": "OVNKubernetes",
                "metadata.ocpMajorVersion.keyword": {
                  "buckets": [
                    {
                      "key": "4.14",
                      "avg-latency": {
                        "value": 2260.4151167980954
                      }
                    },
                    {
                      "key": "4.13",
                      "avg-latency": {
                        "value": 11727.016764266851
                      }
                    }
                  ]
                }
              }
            ]
          }
        },

Thoughts??

[rsevilla87]

cc: @shashank-boyapally @jtaleric

[jtaleric]

curious about this change?

[shashank-boyapally]

I think elasticsearch7 can be installed using elasticsearch==7.x.x, this has been a previous issue for me too.

[rsevilla87]

yeah, if you don't specify the module version, the python interpreter just picks the latest one available

[vishnuchalla]

Suggested change

	x = None
	x = None
	for bucket in buckets:
	a = A("terms", field=bucket)
	if x is None:
	x = s.aggs.bucket(bucket, a)
	else:
	x = x.bucket(bucket, a)
	x.bucket(metric_name, A(agg_type, field=field))

[vishnuchalla]

Overall I like this idea. Should be useful in reducing multiple queries to one on a single ES index atleast.