[Automatic Import] An invalid ingest pipeline config results in blank Review results table

[ebeahan]

Kibana version: 8.17.0

Describe the bug:

In certain situations, Automatic Import will successfully complete its workflow and generate a complete ingest pipeline. However, the pipeline will cause an exception when trying to install (e.g. script_execution due to a compilation error).

Since the pipeline cannot install, the _simulate call to populate the field-to-value mappings fails. the Review results screen displays an empty table.

Steps to reproduce:

1. Complete the prerequisite steps to generate an integration package in Auto Import using a known problem sample.

SecurID Log Sample

[
    {
        "TenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "SourceSystem": "Linux",
        "TimeGenerated [UTC]": "7/12/2021, 8:48:41.630 AM",
        "Computer": "localhost",
        "EventTime [UTC]": "7/12/2021, 8:41:27.000 AM",
        "Facility": "user",
        "HostName": "localhost",
        "SeverityLevel": "notice",
        "SyslogMessage": "41:27,344,,audit.runtime.com.rsa.ims.authn.impl.AuthenticationBrokerImpl,INFO,a6c436a2e2e266a219d3d0a5504eff4a,6ceed1b2e2e266a20801cd23efb75fbd,192.168.1.99,192.168.46.46,AUTHN_LOGIN_EVENT,13002,SUCCESS,AUTHN_METHOD_SUCCESS,a636cfbbe2e266a219d365ddeee982a4-x6l+XQokVhIt,ce3d7b2be2e266a21974b6f5ce9ba2ca,000000000000000000001000d0011000,000000000000000000001000e0011000,U328187,Ericka,Ryptography,2e0ca6e7e2e266a21bc6a77ef43a59e5,000000000000000000001000e0011000,192.168.1.99,mn-sv2-jp-sped-c6-sm2.securitydynamics.com,7,000000000000000000002000f1022000,SecurID_Native,,,AUTHN_LOGIN_EVENT,5,1,000000000000000000001000e0011000,SystemDomain,d83770c3e2e266a21b9dcacfc6c9cd78,SpED,2469ab36e2e266a21a69868bc4b6b808,xxxxxxxx2776,,",
        "ProcessID": "",
        "HostIP": "::1",
        "ProcessName": "2021-07-12",
        "MG": "00000000-0000-0000-0000-000000000002",
        "Type": "Syslog",
        "_ResourceId": ""
    }
]

2. Observe the Review results table after the ingest pipeline is generated displays empty.

Expected behavior:

Ideally, if the pipeline does not install, Automatic Import checks and tries to fix the issue before the workflow completes.

If nothing else, throw some sort of error.

Screenshots (if relevant):

Provide logs and/or server output (if relevant):

Kibana debug level logs:

[2024-12-17T22:17:25.179+00:00][DEBUG][elasticsearch.query.data] 400 - 2.0KB
POST /_ingest/pipeline/_simulate
{"docs":[{"_index":"index","_id":"id","_source":{"message":"{\"TenantId\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"SourceSystem\":\"Linux\",\"TimeGenerated [UTC]\":\"7/12/2021, 8:48:41.630 AM\",\"Computer\":\"localhost\",\"EventTime [UTC]\":\"7/12/2021, 8:41:27.000 AM\",\"Facility\":\"user\",\"HostName\":\"localhost\",\"SeverityLevel\":\"notice\",\"SyslogMessage\":\"41:27,344,,audit.runtime.com.rsa.ims.authn.impl.AuthenticationBrokerImpl,INFO,a6c436a2e2e266a219d3d0a5504eff4a,6ceed1b2e2e266a20801cd23efb75fbd,192.168.1.99,192.168.46.46,AUTHN_LOGIN_EVENT,13002,SUCCESS,AUTHN_METHOD_SUCCESS,a636cfbbe2e266a219d365ddeee982a4-x6l+XQokVhIt,ce3d7b2be2e266a21974b6f5ce9ba2ca,000000000000000000001000d0011000,000000000000000000001000e0011000,U328187,Ericka,Ryptography,2e0ca6e7e2e266a21bc6a77ef43a59e5,000000000000000000001000e0011000,192.168.1.99,mn-sv2-jp-sped-c6-sm2.securitydynamics.com,7,000000000000000000002000f1022000,SecurID_Native,,,AUTHN_LOGIN_EVENT,5,1,000000000000000000001000e0011000,SystemDomain,d83770c3e2e266a21b9dcacfc6c9cd78,SpED,2469ab36e2e266a21a69868bc4b6b808,xxxxxxxx2776,,\",\"ProcessID\":\"\",\"HostIP\":\"::1\",\"ProcessName\":\"2021-07-12\",\"MG\":\"00000000-0000-0000-0000-000000000002\",\"Type\":\"Syslog\",\"_ResourceId\":\"\"}"}},{"_index":"index","_id":"id","_source":{"message":"{\"TenantId\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"SourceSystem\":\"Linux\",\"TimeGenerated [UTC]\":\"7/8/2021, 9:51:22.300 AM\",\"Computer\":\"localhost\",\"EventTime [UTC]\":\"7/8/2021, 9:41:27.000 AM\",\"Facility\":\"user\",\"HostName\":\"localhost\",\"SeverityLevel\":\"notice\",\"SyslogMessage\":\"41:27,344,,audit.runtime.com.rsa.ims.authn.impl.AuthenticationBrokerImpl,INFO,a6c436a2e2e266a219d3d0a5504eff4a,6ceed1b2e2e266a20801cd23efb75fbd,192.168.1.99,192.168.46.46,AUTHN_LOGIN_EVENT,13002,SUCCESS,AUTHN_METHOD_SUCCESS,a636cfbbe2e266a219d365ddeee982a4-x6l+XQokVhIt,ce3d7b2be2e266a21974b6f5ce9ba2ca,000000000000000000001000d0011000,000000000000000000001000e0011000,U328187,Ericka,Ryptography,2e0ca6e7e2e266a21bc6a77ef43a59e5,000000000000000000001000e0011000,192.168.1.99,mn-sv2-jp-sped-c6-sm2.securitydynamics.com,7,000000000000000000002000f1022000,SecurID_Native,,,AUTHN_LOGIN_EVENT,5,1,000000000000000000001000e0011000,SystemDomain,d83770c3e2e266a21b9dcacfc6c9cd78,SpED,2469ab36e2e266a21a69868bc4b6b808,xxxxxxxx2776,,\",\"ProcessID\":\"\",\"HostIP\":\"::1\",\"ProcessName\":\"2021-07-08\",\"MG\":\"00000000-0000-0000-0000-000000000002\",\"Type\":\"Syslog\",\"_ResourceId\":\"\"}"}},{"_index":"index","_id":"id","_source":{"message":"{\"TenantId\":\"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\",\"SourceSystem\":\"Linux\",\"TimeGenerated [UTC]\":\"7/8/2021, 9:47:51.537 AM\",\"Computer\":\"localhost\",\"EventTime [UTC]\":\"9/14/2021, 10:41:27.000 PM\",\"Facility\":\"user\",\"HostName\":\"localhost\",\"SeverityLevel\":\"notice\",\"SyslogMessage\":\"41:27,344, , audit.runtime.com.rsa.ims.authn.impl.AuthenticationBrokerImpl,INFO, a6c436a2e2e266a219d3d0a5504eff4a,6ceed1b2e2e266a20801cd23efb75fbd,192.168.1.99,192.168.46.46,AUTHN_LOGIN_EVENT,13002,SUCCESS,AUTHN_METHOD_SUCCESS,a636cfbbe2e266a219d365ddeee982a4-x6l+XQokVhIt,ce3d7b2be2e266a21974b6f5ce9ba2ca,000000000000000000001000d0011000,000000000000000000001000e0011000,U328187,Ericka,Ryptography,2e0ca6e7e2e266a21bc6a77ef43a59e5,000000000000000000001000e0011000,192.168.1.99,mn-sv2-jp-sped-c6-sm2.securitydynamics.com,7,000000000000000000002000f1022000,SecurID_Native,,,AUTHN_LOGIN_EVENT,5,1,000000000000000000001000e0011000,SystemDomain,d83770c3e2e266a21b9dcacfc6c9cd78,SpED,2469ab36e2e266a21a69868bc4b6b808,xxxxxxxx2776,,\",\"ProcessID\":\"\",\"HostIP\":\"::1\",\"ProcessName\":\"2016-09-15\",\"MG\":\"00000000-0000-0000-0000-000000000002\",\"Type\":\"Syslog\",\"_ResourceId\":\"\"}"}}],"pipeline":{"description":"Pipeline to process securid audit logs","processors":[{"set":{"tag":"set_ecs_version","field":"ecs.version","value":"8.11.0"}},{"set":{"tag":"copy_original_message","field":"originalMessage","copy_from":"message"}},{"rename":{"ignore_missing":true,"if":"ctx.event?.original == null","tag":"rename_message","field":"originalMessage","target_field":"event.original"}},{"remove":{"ignore_missing":true,"if":"ctx.event?.original != null","tag":"remove_copied_message","field":"originalMessage"}},{"remove":{"ignore_missing":true,"tag":"remove_message","field":"message"}},{"json":{"tag":"json_original","field":"event.original","target_field":"securid.audit"}},{"rename":{"ignore_missing":true,"field":"securid.audit.SourceSystem","target_field":"host.os.platform"}},{"rename":{"ignore_missing":true,"field":"securid.audit.Computer","target_field":"host.name"}},{"script":{"tag":"script_convert_array_to_string","description":"Ensures the date processor does not receive an array value.","lang":"painless","source":"if (ctx.securid?.audit?.EventTime [UTC] != null &&\n    ctx.securid.audit.EventTime [UTC] instanceof ArrayList){\n    ctx.securid.audit.EventTime [UTC] = ctx.securid.audit.EventTime [UTC][0];\n}\n"}},{"date":{"if":"ctx.securid?.audit?.EventTime [UTC] != null","tag":"date_processor_securid.audit.EventTime [UTC]","field":"securid.audit.EventTime [UTC]","target_field":"event.start","formats":["M/d/yyyy, h:mm:ss.SSS a"]}},{"rename":{"ignore_missing":true,"field":"securid.audit.Facility","target_field":"log.syslog.facility.name"}},{"rename":{"ignore_missing":true,"field":"securid.audit.HostName","target_field":"host.hostname"}},{"rename":{"ignore_missing":true,"field":"securid.audit.SeverityLevel","target_field":"log.level"}},{"rename":{"ignore_missing":true,"field":"securid.audit.SyslogMessage","target_field":"message"}},{"convert":{"ignore_failure":true,"ignore_missing":true,"field":"securid.audit.HostIP","target_field":"host.ip","type":"ip"}},{"script":{"tag":"script_drop_null_empty_values","description":"Drops null/empty values recursively.","lang":"painless","source":"boolean dropEmptyFields(Object object) {\n  if (object == null || object == \"\") {\n    return true;\n  } else if (object instanceof Map) {\n    ((Map) object).values().removeIf(value -> dropEmptyFields(value));\n    return (((Map) object).size() == 0);\n  } else if (object instanceof List) {\n    ((List) object).removeIf(value -> dropEmptyFields(value));\n    return (((List) object).length == 0);\n  }\n  return false;\n}\ndropEmptyFields(ctx);\n"}},{"geoip":{"ignore_missing":true,"tag":"geoip_source_ip","field":"source.ip","target_field":"source.geo"}},{"geoip":{"ignore_missing":true,"tag":"geoip_source_asn","database_file":"GeoLite2-ASN.mmdb","field":"source.ip","target_field":"source.as","properties":["asn","organization_name"]}},{"rename":{"ignore_missing":true,"tag":"rename_source_as_asn","field":"source.as.asn","target_field":"source.as.number"}},{"rename":{"ignore_missing":true,"tag":"rename_source_as_organization_name","field":"source.as.organization_name","target_field":"source.as.organization.name"}},{"geoip":{"ignore_missing":true,"tag":"geoip_destination_ip","field":"destination.ip","target_field":"destination.geo"}},{"geoip":{"ignore_missing":true,"tag":"geoip_destination_asn","database_file":"GeoLite2-ASN.mmdb","field":"destination.ip","target_field":"destination.as","properties":["asn","organization_name"]}},{"rename":{"ignore_missing":true,"tag":"rename_destination_as_asn","field":"destination.as.asn","target_field":"destination.as.number"}},{"rename":{"ignore_missing":true,"tag":"rename_destination_as_organization_name","field":"destination.as.organization_name","target_field":"destination.as.organization.name"}},{"remove":{"ignore_missing":true,"tag":"remove_fields","field":["securid.audit.HostIP"]}},{"remove":{"ignore_failure":true,"ignore_missing":true,"if":"ctx?.tags == null || !(ctx.tags.contains(\"preserve_original_event\"))","tag":"remove_original_event","field":"event.original"}}],"on_failure":[{"append":{"field":"error.message","value":"Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}"}},{"set":{"field":"event.kind","value":"pipeline_error"}}]}} [script_exception]: compile error
[2024-12-17T22:17:25.180+00:00][DEBUG][plugins.integrationAssistant.apmTracer] onChainStart: run:
{
  "id": "00577e67-5232-4349-a82c-5acb38ebeb86",
  "name": "ChannelWrite<lastExecutedChain,rawSamples,samples,hasTriedOnce,ecs,exAnswer,packageName,dataStreamName,finalized,reviewed,errors,previousError,pipelineResults,currentPipeline,currentProcessors,initialPipeline,results,samplesFormat,handleValidatePipeline>",
  "parent_run_id": "c2484e84-9933-4d0d-9841-4d15293da4cc",
  "start_time": 1734473845180,
  "serialized": {
    "lc": 1,
    "type": "not_implemented",
    "id": [
      "langgraph",
      "ChannelWrite"
    ]
  },
  "events": [
    {
      "name": "start",
      "time": "2024-12-17T22:17:25.180Z"
    }
  ],
  "inputs": {
    "errors": [
      {
        "error": "script_exception\n\tCaused by:\n\t\tillegal_argument_exception: cannot resolve symbol [UTC]\n\tRoot causes:\n\t\tscript_exception: compile error"
      }
    ],
    "previousError": "[\n  {\n    \"error\": \"script_exception\\n\\tCaused by:\\n\\t\\tillegal_argument_exception: cannot resolve symbol [UTC]\\n\\tRoot causes:\\n\\t\\tscript_exception: compile error\"\n  }\n]",
    "previousPipelineResults": [],
    "pipelineResults": [],
    "lastExecutedChain": "validate_pipeline"
  },
  "execution_order": 16,
  "child_execution_order": 16,
  "run_type": "chain",
  "child_runs": [],
  "extra": {
    "metadata": {
      "langgraph_step": 4,
      "langgraph_node": "handleValidatePipeline",
      "langgraph_triggers": [
        "handleRelated"
      ],
      "langgraph_path": [
        "__pregel_pull",
        "handleValidatePipeline"
      ],
      "langgraph_checkpoint_ns": "handleValidatePipeline:487e6370-5afd-593c-9edc-c79e9e841d54",
      "__pregel_resuming": false,
      "__pregel_task_id": "487e6370-5afd-593c-9edc-c79e9e841d54",
      "checkpoint_ns": "handleValidatePipeline:487e6370-5afd-593c-9edc-c79e9e841d54"
    }
  },
  "tags": [
    "langsmith:hidden"
  ],
  "trace_id": "3021d68f-3fff-4beb-9fbe-b2ff3aceb1b7",
  "dotted_order": "20241217T221724633001Z3021d68f-3fff-4beb-9fbe-b2ff3aceb1b7.20241217T221725172015Zc2484e84-9933-4d0d-9841-4d15293da4cc.20241217T221725180016Z00577e67-5232-4349-a82c-5acb38ebeb86"
}

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[haetamoudi]

Another type of error:
testPipeline step failure:

e.g: pipeline generated:

...
{
      "date": {
        "if": "ctx.a?.a?.@timestamp != null",
        "tag": "date_processor_a.a.@timestamp",
        "field": "a.a.@timestamp",
        "target_field": "@timestamp",
        "formats": null
      }
    },
...

will trigger the following error:

{
  pipelineResults: [],
  errors: [
    {
      error: 'parse_exception\n' +
        '\tRoot causes:\n' +
        '\t\tparse_exception: [formats] required property is missing'
    }
  ]
}

Docs won't be added to pipelineResults and nothing will be displayed on the Review Integration step