Unmapped Proposal

[rroupski]

Proposal 1: UTF-8 encoded key/value pairs (the current format)

• Unmapped fields will need to be UTF-8 encoded.
• Original field names will be preserved, though UTF-8 encoded, potentially violating OCSF guidelines and grammar
• There will be no limitation on nesting, though implementation may impose a limitation.
• Embedded arrays and objects are supported.
• Values may use different types.

Example

{
  "unmapped": {
    "EventType": 42,
    "TaskCategory": "Logon",
    "Network Information": {
      "Workstation Name": "ENGCSEAPEX01"
    },
    "SourceName": "Microsoft Windows security auditing.",
    "Accesses": ["DELETE", "READ_CONTROL", "WRITE_DAC", "WRITE_OWNER", "AddMember", "RemoveMember", "ListMembers", "ReadInformation", "WriteAccount", "Undefined Access (no effect) Bit 5", "Undefined Access (no effect) Bit 6", "Undefined Access (no effect) Bit 7", "Undefined Access (no effect) Bit 8"],
    "Detailed Authentication Information": {
      "Package Name (NTLM only)": "-",
      "Transited Services": "-",
      "Key Length": 22
    },
    "Failure Information": {
      "Sub Status": "0xc0000064"
    },
    "OpCode": "Info",
    "ComputerName": "WSDCKAS1.emn.com"
  }
}

Cons

• Potential onslaught of fields. Strict typing difficulties since these fields are not defined by OCSF.

Proposal 2: Similar to Proposal 1 (values are strings)

• Unmapped fields will need to be UTF-8 encoded.
• Original field names will be preserved, though UTF-8 encoded, potentially violating OCSF guidelines and grammar
• There will be no limitation on nesting, though implementation may impose a limitation.
• All values will be cast to strings to prevent type collisions from disparate logs that may use different types.
• All arrays should be flattened.

Example

{
  "unmapped": {
    "EventType": "42",
    "TaskCategory": "Logon",
    "Network Information": {
      "Workstation Name": "ENGCSEAPEX01"
    },
    "SourceName": "Microsoft Windows security auditing.",
    "Accesses": "DELETE, READ_CONTROL, WRITE_DAC, WRITE_OWNER, AddMember, RemoveMember, ListMembers, ReadInformation, WriteAccount, Undefined Access (no effect) Bit 5, Undefined Access (no effect) Bit 6, Undefined Access (no effect) Bit 7, Undefined Access (no effect) Bit 8"],
    "Detailed Authentication Information": {
      "Package Name (NTLM only)": "-",
      "Transited Services": "-",
      "Key Length": "22"
    },
    "Failure Information": {
      "Sub Status": "0xc0000064"
    },
    "OpCode": "Info",
    "ComputerName": "WSDCKAS1.emn.com"
  }
}

Cons

• Potential onslaught of fields. Strict typing difficulties since these fields are not defined by OCSF.
• Requires some extra cpu work to flatten the arrays and convert the data to strings.
• How an array of objects will encoded? For example, an array of file objects?

Proposal 3: Binary blob

• The unmapped field will reference a JSON binary blob
• Nested fields will be preserved, along with types in the original encoding

Example

{
  "unmapped": "{\"EventType\": 42, \"TaskCategory\": \"Logon\", \"Network Information\": {\"Workstation Name\": \"ENGCSEAPEX01\"}, \"SourceName\": \"Microsoft Windows security auditing.\", \"Accesses\": [\"DELETE\", \"READ_CONTROL\", \"WRITE_DAC\", \"WRITE_OWNER\", \"AddMember\", \"RemoveMember\", \"ListMembers\", \"ReadInformation\", \"WriteAccount\", \"Undefined Access (no effect) Bit 5\", \"Undefined Access (no effect) Bit 6\", \"Undefined Access (no effect) Bit 7\", \"Undefined Access (no effect) Bit 8\"], \"Detailed Authentication Information\": {\"Package Name (NTLM only)\": \"-\", \"Transited Services\": \"-\", \"Key Length\": 22 }, \"Failure Information\": {\"Sub Status\": \"0xc0000064\"}, \"OpCode\": \"Info\", \"ComputerName\": \"WSDCKAS1.emn.com\"}"
}

Cons

• Potential implementation difficulties in utilizing a different encoding within the schema, as JSON doesn’t natively support binary.
• Field names cannot be natively “discovered” as with Proposal 1/2, where they would be listed via conventions such as the parquet column metadata.