add list_fields api

[PSeitz]

• Retry
• List fields from tantivy
• Fallback to schema if field list does not exist
• {index_id}/_field_caps URL
• /_field_caps URL
• fields filter (my-index-000001/_field_caps?fields=rating)
• API tests
• Caching (very simple list field cache)
• Parallel fetch of field list file

➜  quickwit-indices curl -XGET http://localhost:7280/api/v1/_elastic/hdfs/_field_caps -H 'Content-Type: application/json' -d ''
{
  "indices": [
    "hdfs"
  ],
  "fields": {
    "_dynamic.attributes.class": {
      "keyword": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "hdfs"
        ]
      },
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "hdfs"
        ]
      }
    },
    "_field_presence": {
      "long": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [
          "hdfs"
        ]
      }
    },
    "body": {
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [
          "hdfs"
        ]
      }
    },
    "timestamp": {
      "date_nanos": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "hdfs"
        ]
      }
    },
    "tenant_id": {
      "long": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [
          "hdfs"
        ]
      }
    },
    "resource.$facet:service": {
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [
          "hdfs"
        ]
      }
    },
    "severity_text": {
      "keyword": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "hdfs"
        ]
      },
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "hdfs"
        ]
      }
    }
  }
}⏎                                                                                                                                                                                                                               

[fulmicoton]

Can you comment on how it is mapped? Why signed int?

Suggested change

	int32 field_type = 2;
	int32 field_type = 2;

[PSeitz]

I added a comment (Type::to_code()) and switched to u32

[fulmicoton]

do you really mean indices in all of this proto, or did you mean split ids?

[PSeitz]

yes indices. There is no use for split ids in the API currently

[fulmicoton]

I don't understand this contract.
So you return io::Result and lose the info of the missing split ids?
What are non_searchable_index_ids ?

[fulmicoton]

When do we return an ioError as an item.

[PSeitz]

Actually I'm not sure what to do with the error, currently we abort and return. This is happening in the merge function. I guess we want to return the failed split_ids for a retry. There's no retry currently.

let entry = entry.map_err(|err| crate::error::SearchError::Internal(format!("{:?}", err)))?; // TODO: No early return on error

non_searchable_index_ids are exemption list of indexes if searchable: true. I took this from the elastic search API.

[fulmicoton]

do we really need this?

[PSeitz]

quickwit-indexing contains the code to write and read the split fields. I could move it, but not sure where to

[fulmicoton]

Can you add the type here?

Suggested change

	let mut non_searchable_index_ids = if searchable {
	let mut non_searchable_index_ids = if searchable {

Also if perf do not matter, BTreeSet<&str> is a simple way to inform users and maintain the contract that this is sorted and unique.

[PSeitz]

This is building the protobuf API response, which requires Vec

[fulmicoton]

how far are we from elastic's api?

[PSeitz]

The most important features are supported, some minor parameters are not implemented yet

[fulmicoton]

if we drop

engines:
- quickwit

which part breaks?

[PSeitz]

it works without, seems to pick quickwit as default

[fulmicoton]

I mean running it against elasticsearch without it.

[PSeitz]

The main differences are:

• Mixed types are not supported in elastic, so this test doesn't work in elastic. We get multiple entries for mixed types. Support elasticsearch field capabilities endpoint #3527 (comment)
• We have different coercion rules (e.g. i64, u64, f64 preference order)
• Mapping of text types. Currently we map text fields with fast to keyword and indexed to text. keyword is not on the same fieldname in elastic but a new one.
• We map our floats to double, but elastic seems to use float when the type is detected

[fulmicoton]

make it a proto enum to remove the dependency in quickwit-serve.

[PSeitz]

I would rather reuse the existing code, tantivy is in the dependency tree anyway

[fulmicoton]

The code goes

• serialize list of fields as a file artifact with code living in quickwit-indexing, and using tantivy's types.
• convert artifact to protobuf

Would using protobuf to begin lead to better code?
We could remove code, and drop the quickwit-search -> quickwit-indexing dependency

[PSeitz]

The code goes

* serialize list of fields as a file artifact with code living in quickwit-indexing, and using tantivy's types.

* convert artifact to protobuf

Would using protobuf to begin lead to better code? We could remove code, and drop the quickwit-search -> quickwit-indexing dependency

I don't think this would be better code, since this would be creating duplicated code. The existing type does exactly what we want and tantivy is already in the dependency tree.
The quickwit-search -> quickwit-indexing dependency has nothing to do with the tantivy types, this is because field de/serialziation is in models/list_fields.rs

[fmassot]

I think we want to remove the _dynamic from the field names as we don't want to leak that to the users.

[PSeitz]

Yes, I think so too. I wasn't completely sure if there's a case where we need it.

[PSeitz]

done