[NL-to-ESQL] update internal documentation (elastic#205853)

## Summary

Fix elastic#205606

- Re-generate the internal ES|QL documentation using the generation
script (+ human review)
- Add more scenario to the NL-to-ESQL evaluation suite  
- Some prompt engineering
  - improving the system instructions / functions summary
  - add more examples to the summary
  - adapt a few opinionated examples for some specific functions 

## Evaluation

- average based on 4 runs for each model/branch tuple
- the new tests were locally added to main to run against the same suite
and properly evaluate the difference

| Model  | before (main) | after (PR) | delta |
| ------------- | ------------- | ------------- | ------------- |
| GPT-4o  | 90.9 | 97.74  | + 6.84 |
| Claude 3.5 Sonnet v2  |  88.58 | 96.49 | +7.91 |
| Gemini 1.5-pro-002  | 88.17  | 94.19 | +6.02 |

Overall, the prompt engineering somewhat significantly improved the
generation efficiency.

x-pack/platform/plugins/shared/inference/scripts/load_esql_docs/load_esql_docs.ts

@@ -69,7 +69,7 @@ yargs(process.argv.slice(2))
           });
           log.info(`Using connector ${connector.connectorId}`);
 
-          const chatClient = kibanaClient.createInferenceClient({
+          const inferenceClient = kibanaClient.createInferenceClient({
             connectorId: connector.connectorId,
           });
 
@@ -84,14 +84,14 @@ yargs(process.argv.slice(2))
           log.info(`Retrieving and converting documentation from ${builtDocsDir}...`);
           const extraction = await extractDocEntries({
             builtDocsDir,
-            inferenceClient: chatClient,
+            inferenceClient,
             log,
           });
 
           log.info(`Rewriting documentation...`);
           const docFiles = await generateDoc({
             extraction,
-            inferenceClient: chatClient,
+            inferenceClient,
             log,
           });
 

x-pack/platform/plugins/shared/inference/scripts/util/kibana_client.ts

@@ -212,10 +212,10 @@ export class KibanaClient {
         return this.axios
           .post(
             this.getUrl({
-              pathname: `/internal/inference/chat_complete/stream`,
+              pathname: `/internal/inference/chat_complete`,
             }),
             body,
-            { responseType: 'stream', timeout: NaN }
+            { timeout: NaN }
           )
           .then((response) => {
             return response.data;

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/doc_base/suggestions.ts

@@ -13,6 +13,11 @@ const suggestions: Suggestion[] = [
       return ['BUCKET'];
     }
   },
+  (keywords) => {
+    if (keywords.includes('TO_DATETIME')) {
+      return ['DATE_PARSE'];
+    }
+  },
 ];
 
 /**

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-bit_length.txt

@@ -0,0 +1,24 @@
+# BIT_LENGTH
+
+This function calculates the bit length of a given string.
+
+## Syntax
+
+`BIT_LENGTH(string)`
+
+### Parameters
+
+#### string
+
+This is the string whose bit length you want to calculate. If `null` is provided, the function will return `null`.
+
+**Note**: Strings are in UTF-8 format, which means a single character may occupy multiple bytes.
+
+## Examples
+
+```esql
+FROM airports
+| WHERE country == "India"
+| KEEP city
+| EVAL fn_length = LENGTH(city), fn_bit_length = BIT_LENGTH(city)
+```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-bucket.txt

@@ -30,8 +30,7 @@ BUCKET can operate in two modes:
 - one where the bucket size is computed based on a bucket count recommendation and a range,
 - and another where the bucket size is provided directly.
 
-When the bucket size is provided directly for time interval,
-it is expressed as a *timespan literal*, e.g.
+When the bucket size is provided directly for time interval, it is expressed as a **timespan literal**, e.g.
 - GOOD: `BUCKET(@timestamp, 1 month)`
 - BAD: `BUCKET(@timestamp, "month")`
 
@@ -74,29 +73,40 @@ FROM employees
 
 More examples:
 
+*Regrouping employees in buckets based on salary and counting them*
 ```esql
 FROM employees
 | WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
-| STATS c = COUNT(1) BY b = BUCKET(salary, 5000.)
+| STATS c = COUNT(*) BY b = BUCKET(salary, 5000.)
 | SORT b
 ```
 
+*Group data emitted over the last 24h into 25 buckets*
 ```esql
 FROM sample_data
 | WHERE @timestamp >= NOW() - 1 day and @timestamp < NOW()
 | STATS COUNT(*) BY bucket = BUCKET(@timestamp, 25, NOW() - 1 day, NOW())
 ```
 
+*Similar to previous example but with fixed 1 hour bucket size*
+```esql
+FROM sample_data
+| WHERE @timestamp >= NOW() - 1 day and @timestamp < NOW()
+| STATS COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour)
+```
+
+*Group employees in 20 buckets based on their hire_date and then calculate the average salary for each bucket*
 ```esql
 FROM employees
 | WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
 | STATS AVG(salary) BY bucket = BUCKET(hire_date, 20, "1985-01-01T00:00:00Z", "1986-01-01T00:00:00Z")
 | SORT bucket
 ```
 
+*Similar to previous example but using fixed 1 month buckets size*
 ```esql
 FROM employees
-| STATS s1 = BUCKET(salary / 1000 + 999, 50.) + 2 BY b1 = BUCKET(salary / 100 + 99, 50.), b2 = BUCKET(salary / 1000 + 999, 50.)
-| SORT b1, b2
-| KEEP b1, s1, b2
+| WHERE hire_date >= "1985-01-01T00:00:00Z" AND hire_date < "1986-01-01T00:00:00Z"
+| STATS AVG(salary) BY bucket = BUCKET(hire_date, 1 month)
+| SORT bucket
 ```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-byte_length.txt

@@ -0,0 +1,22 @@
+# BYTE_LENGTH
+
+This function calculates the byte length of a given string.
+
+## Syntax
+
+`BYTE_LENGTH(string)`
+
+### Parameters
+
+#### string
+
+The text string for which the byte length is to be determined. If `null` is provided, the function will return `null`.
+
+## Examples
+
+```esql
+FROM airports
+| WHERE country == "India"
+| KEEP city
+| EVAL fn_length = LENGTH(city), fn_byte_length = BYTE_LENGTH(city)
+```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-case.txt

@@ -4,7 +4,7 @@ The CASE function accepts pairs of conditions and values. It returns the value t
 
 ## Syntax
 
-`CASE(condition, trueValue)`
+`CASE(condition, trueValue, elseValue)`
 
 ### Parameters
 
@@ -16,16 +16,20 @@ A condition to evaluate.
 
 The value that is returned when the corresponding condition is the first to evaluate to `true`. If no condition matches, the default value is returned.
 
+#### elseValue
+
+The value that will be returned when no condition evaluates to `true`.
+
 ## Examples
 
-Determine whether employees are monolingual, bilingual, or polyglot:
+In this example, employees are categorized as monolingual, bilingual, or polyglot depending on how many languages they speak:
 
 ```esql
 FROM employees
 | EVAL type = CASE(
     languages <= 1, "monolingual",
     languages <= 2, "bilingual",
-     "polyglot")
+    "polyglot")
 | KEEP emp_no, languages, type
 ```
 
@@ -46,6 +50,6 @@ Calculate an hourly error rate as a percentage of the total number of log messag
 FROM sample_data
 | EVAL error = CASE(message LIKE "*error*", 1, 0)
 | EVAL hour = DATE_TRUNC(1 hour, @timestamp)
-| STATS error_rate = AVG(error) by hour
+| STATS error_rate = AVG(error) BY hour
 | SORT hour
-```
+```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-categorize.txt

@@ -0,0 +1,30 @@
+# CATEGORIZE
+
+The `CATEGORIZE` function organizes textual data into groups of similar format.
+
+> **Note:** The `CATEGORIZE` function is currently in technical preview and may undergo changes or be removed in future releases.
+
+## Syntax
+
+`CATEGORIZE(field)`
+
+### Parameters
+
+#### field
+
+The expression that is to be categorized.
+
+## Examples
+
+The following example demonstrates how to use `CATEGORIZE` to group server log messages into categories and then aggregate their counts.
+
+```esql
+FROM sample_data
+| STATS count = COUNT() BY category=CATEGORIZE(message)
+```
+
+## Limitations
+
+- `CATEGORIZE` can't be used within other expressions
+- `CATEGORIZE` can't be used with multiple groupings
+- `CATEGORIZE` can't be used or referenced within aggregate functions

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-date_extract.txt

@@ -10,7 +10,7 @@ The DATE_EXTRACT function is used to extract specific parts of a date.
 
 #### datePart
 
-This is the part of the date you want to extract, such as "year", "month" or ""hour_of_day".
+This is the part of the date you want to extract, such as "year", "month" or "hour_of_day".
 
 #### date
 

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-date_parse.txt

@@ -1,6 +1,6 @@
 # DATE_PARSE
 
-The DATE_PARSE function is used to convert a date string into a date format based on the provided pattern.
+The DATE_PARSE function is used to convert a date string into a date based on the provided format pattern.
 
 ## Syntax
 
@@ -23,6 +23,7 @@ ROW date_string = "2022-05-06"
 | EVAL date = DATE_PARSE("yyyy-MM-dd", date_string)
 ```
 
-ROW date_string = "2023-12-25"
+```esql
+FROM logs
 | EVAL date = DATE_PARSE("yyyy-MM-dd", date_string)
 ```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-eval.txt

@@ -65,6 +65,15 @@ FROM employees
 | STATS avg_height_feet = AVG(`height * 3.281`)
 ```
 
+Any number of evaluations can be performed in a single EVAL command
+
+```esql
+FROM triangle
+| EVAL cos = COS(angle), tan = TAN(angle), sin = SIN(angle), acos=ACOS(angle), asin=ASIN(angle)
+| SORT angle DESC
+| LIMIT 10
+```
+
 ### Limitations
 - If a column with the same name already exists, the existing column is dropped.
 - If a column name is used more than once, only the rightmost duplicate creates a column.

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-exp.txt

@@ -21,6 +21,6 @@ ROW d = 5.0
 ```
 
 ```esql
-ROW value = 2.0
-| EVAL result = EXP(value)
+FROM geo
+| EVAL exp = EXP(x)
 ```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-hash.txt

@@ -0,0 +1,30 @@
+# HASH
+
+The HASH function computes the hash of a given input using a specified algorithm.
+
+## Syntax
+
+`HASH(algorithm, input)`
+
+### Parameters
+
+#### algorithm
+
+The hash algorithm to be used.
+
+The supported algorithms are:
+- "MD5"
+- "SHA-1"
+- "SHA-256"
+
+#### input
+
+The value to be hashed.
+
+## Examples
+
+```esql
+FROM messages
+| EVAL hashed_content = HASH("SHA-1", content)
+| KEEP message_id, hashed_content
+```

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-hypot.txt

@@ -0,0 +1,28 @@
+# HYPOT
+
+The HYPOT function is used to calculate the hypotenuse of two numbers. 
+
+## Syntax
+
+`HYPOT(number1, number2)`
+
+### Parameters
+
+#### number1
+
+This is a numeric value. If it's `null`, the function will also return `null`.
+
+#### number2
+
+This is also a numeric value. If it's `null`, the function will also return `null`.
+
+## Examples
+
+Check the hypotenuse of two variables through the following example:
+
+```esql
+ROW a = 3.0, b = 4.0
+| EVAL c = HYPOT(a, b)
+```
+
+Note that the HYPOT function returns the hypotenuse in double data type. Besides, if any of the numbers is infinity, the hypotenuse returns `null`.