Merge branch 'main' into trinity/bulk-put

README.md

@@ -36,9 +36,9 @@ Cloud-native search engine for observability (logs, traces, and soon metrics!).
 
 ### 🚀 Quickstart
 
-- [Search and analytics on StackOverflow dataset](https://quickwit.io/docs/get-started/quickstart)
-- [Tracing analytics with Grafana](https://quickwit.io/docs/get-started/tutorials/trace-analytics-with-grafana)
-- [Tracing with Jaeger](https://quickwit.io/docs/get-started/tutorials/tutorial-jaeger)
+- [Search and analytics on Stack Overflow dataset](https://quickwit.io/docs/get-started/quickstart)
+- [Trace analytics with Grafana](https://quickwit.io/docs/get-started/tutorials/trace-analytics-with-grafana)
+- [Distributed tracing with Jaeger](https://quickwit.io/docs/get-started/tutorials/tutorial-jaeger)
 
 <br/>
 
@@ -94,34 +94,44 @@ Cloud-native search engine for observability (logs, traces, and soon metrics!).
 
 # 🔮 Roadmap
 
+- Quickwit 0.9 (June 2024)
+  - Indexing and search performance improvements
+  - Index configuration updates (retention policy, indexing and search settings)
+  - Concatenated field
+
+- Quickwit 0.10 (September 2024)
+  - Schema (doc mapping) updates
+  - Native distributed ingestion
+  - Index templates
+
 - [Long-term roadmap](ROADMAP.md)
   - Live tail
   - SQL
   - Security (TLS, authentication, RBAC)
   - Alerting
   - [and more...](ROADMAP.md)
 
-details are also available on the [github projects](https://github.com/orgs/quickwit-oss/projects?query=is%3Aopen)
+details are also available on the [GitHub projects](https://github.com/orgs/quickwit-oss/projects?query=is%3Aopen)
 
 # 🙋 FAQ
 
-### How can I switch from Elasticsearch or Opensearch to Quickwit?
+### How can I switch from Elasticsearch or OpenSearch to Quickwit?
 
-Quickwit supports a large subset of Elasticsearch/Opensearch API.
+Quickwit supports a large subset of Elasticsearch/OpenSearch API.
 
-For instance, it has a ES-compatible ingest API to make it easier to migrate your log shippers (Vector, Fluent Bit, Syslog, ...) to Quickwit.
+For instance, it has an ES-compatible ingest API to make it easier to migrate your log shippers (Vector, Fluent Bit, Syslog, ...) to Quickwit.
 
-On the search-side, the most popular Elasticsearch endpoints, query DSL, and even aggregations are supported.
+On the search side, the most popular Elasticsearch endpoints, query DSL, and even aggregations are supported.
 
 The list of available endpoints and queries is available [here](https://quickwit.io/docs/reference/es_compatible_api), while the list of supported aggregations is available [here](https://quickwit.io/docs/reference/aggregation).
 
 Let us know if part of the API you are using is missing!
 
-If the client you are using is refusing to connect to Quickwit due to missing headers, you can use the `extra_headers` option in the [node configuration](https://quickwit.io/docs/configuration/node-config#rest-configuration) to impersonate any compatible version of Elasticsearch or Opensearch.
+If the client you are using is refusing to connect to Quickwit due to missing headers, you can use the `extra_headers` option in the [node configuration](https://quickwit.io/docs/configuration/node-config#rest-configuration) to impersonate any compatible version of Elasticsearch or OpenSearch.
 
 ### How is Quickwit different from traditional search engines like Elasticsearch or Solr?
 
-The core difference and advantage of Quickwit are its architecture built from the ground to search on cloud storage. We optimized IO paths, revamped the index data structures and made search stateless and sub-second on cloud storage.
+The core difference and advantage of Quickwit is its architecture built from the ground to search on cloud storage. We optimized IO paths, revamped the index data structures and made search stateless and sub-second on cloud storage.
 
 ### How does Quickwit compare to Elastic in terms of cost?
 
@@ -132,7 +142,7 @@ We estimate that Quickwit can be up to 10x cheaper on average than Elastic. To u
 Quickwit is open-source under the GNU Affero General Public License Version 3 - AGPLv3. Fundamentally, this means you are free to use Quickwit for your project if you don't modify Quickwit. However, if you do and you are distributing your modified version to the public, you have to make the modifications public.
 We also provide a commercial license for enterprises to provide support and a voice on our roadmap.
 
-### Is it possible to setup Quickwit for a High Availability (HA)?
+### Is it possible to set up Quickwit for a High Availability (HA)?
 
 HA is available for search, for indexing it's available only with a Kafka source.
 

ROADMAP.md

@@ -9,6 +9,7 @@
 - Metrics
 - Geospatial search
 - Native Go, Java, and Python clients
+- SQL
 - Pipe-based query language
 - Security (TLS, authentication, RBAC)
 - and more...

quickwit/quickwit-control-plane/src/ingest/ingest_controller.rs

@@ -794,7 +794,7 @@ impl IngestController {
         }
         let num_open_shards_per_leader_target = num_open_shards / num_ingesters;
         let num_open_shards_per_leader_threshold = cmp::max(
-            num_open_shards_per_leader_target * 12 / 10,
+            num_open_shards_per_leader_target * 11 / 10,
             num_open_shards_per_leader_target + 1,
         );
         let mut shards_to_move: Vec<&ShardEntry> = Vec::new();

quickwit/quickwit-serve/src/rest.rs

@@ -275,11 +275,6 @@ fn get_status_with_error(rejection: Rejection) -> RestApiError {
             status_code: StatusCode::UNSUPPORTED_MEDIA_TYPE,
             message: error.to_string(),
         }
-    } else if rejection.is_not_found() {
-        RestApiError {
-            status_code: StatusCode::NOT_FOUND,
-            message: "Route not found".to_string(),
-        }
     } else if let Some(error) = rejection.find::<serde_qs::Error>() {
         RestApiError {
             status_code: StatusCode::BAD_REQUEST,
@@ -291,12 +286,6 @@ fn get_status_with_error(rejection: Rejection) -> RestApiError {
             status_code: StatusCode::BAD_REQUEST,
             message: error.0.to_string(),
         }
-    } else if let Some(error) = rejection.find::<InvalidArgument>() {
-        // Happens when the url path or request body contains invalid argument(s).
-        RestApiError {
-            status_code: StatusCode::BAD_REQUEST,
-            message: error.0.to_string(),
-        }
     } else if let Some(error) = rejection.find::<warp::filters::body::BodyDeserializeError>() {
         // Happens when the request body could not be deserialized correctly.
         RestApiError {
@@ -338,11 +327,6 @@ fn get_status_with_error(rejection: Rejection) -> RestApiError {
             status_code: StatusCode::BAD_REQUEST,
             message: error.to_string(),
         }
-    } else if let Some(error) = rejection.find::<warp::reject::MethodNotAllowed>() {
-        RestApiError {
-            status_code: StatusCode::METHOD_NOT_ALLOWED,
-            message: error.to_string(),
-        }
     } else if let Some(error) = rejection.find::<warp::reject::PayloadTooLarge>() {
         RestApiError {
             status_code: StatusCode::PAYLOAD_TOO_LARGE,
@@ -353,6 +337,22 @@ fn get_status_with_error(rejection: Rejection) -> RestApiError {
             status_code: StatusCode::TOO_MANY_REQUESTS,
             message: err.to_string(),
         }
+    } else if let Some(error) = rejection.find::<InvalidArgument>() {
+        // Happens when the url path or request body contains invalid argument(s).
+        RestApiError {
+            status_code: StatusCode::BAD_REQUEST,
+            message: error.0.to_string(),
+        }
+    } else if let Some(error) = rejection.find::<warp::reject::MethodNotAllowed>() {
+        RestApiError {
+            status_code: StatusCode::METHOD_NOT_ALLOWED,
+            message: error.to_string(),
+        }
+    } else if rejection.is_not_found() {
+        RestApiError {
+            status_code: StatusCode::NOT_FOUND,
+            message: "Route not found".to_string(),
+        }
     } else {
         error!("REST server error: {:?}", rejection);
         RestApiError {