String Sibling Normalization

[paveljos]

"sibling" strings, e.g, severity as it relates to severity_id, are currently defined as relating to the event source. Again, for severity, "The event severity, as defined by the event source."

This has a few potential problems:

1. Inconsistency with how OCSF defines the utilization of ref_ fields; i.e., severity should be renamed ref_severity
2. No offer of normalization for applications that utilize or prefer the strings for things like captions or visualizations. e.g, Informational and informational and INFORMATIONAL may be grouped independently for poorly written queries.
3. Cases that do not fit into the pre-defined categorization get lumped into "Other" with no further assistance offered by the schema for data taxonomy.

I would recommend the following:
For "sibling" fields, the should be understood and defined as enum+. The strings will match the predefined cases as defined by the OCSF definition for their integer sibling, with the exception of "Other", where the original value is preserved and offers data consumers an opportunity to further understand and utilize the data.

For example,

severity:
0	Unknown
1	Informational	
2	Low	
3	Medium	
4	High	
5	Critical	
6	Fatal	
99	Other

If someone had the severity as "Blocker", the int enum severity_id would be 99 and the string sibling severity would be Other:Blocker

I will set up a poll for voting and additional feedback.

[paveljos]

My bad; ref_ fields are no longer in scope for OCSF. I had just re-read https://github.com/ocsf/ocsf-docs/blob/main/Understanding%20OCSF.pdf and so they were on my mind. As a follow-up, we should update the documentation on ref_ fields.

[pagbabian-splunk]

Yes, I think I had removed the ref_ convention but I may have missed it somewhere or other. I am updating the WP to match our current RC this weekend (already made a few changes) but I haven't pushed it yet. I will double check.

[rroupski]

It is highly desirable to normalize the data and use the predefined enum values as much as possible. However, there will be cases when that's going to be not possible and forcing rules on the text makes the schema too restrictive. It is very important to keep the free form text of the enum text because it allows using values that are defined by other standards.

[floydtree]

Are you suggesting that the sibling string values should always be the original values as provided by the source event?

[jp-harvey]

Can't those producers use 99 - Other:3:AUTOMAGIC (for example) if they don't want to / can't map to OCSF directly? It may as well be a free-text field if we're going to allow any id-string mapping.

[rroupski]

yes, if the value is other(99), then sibling string is provided by the source event.

[pagbabian-splunk]

I don't think I understand the reasoning on Other. What convention are we proposing for a value that is unknown to us a priori? Is it to prefix the event source literal with "Other:"? What is the convention being proposed?

[paveljos]

The convention for "other" isn't the primary concern for this proposal, though to be complete we will have to discuss and decide on whatever convention.

The primary concern is that as it currently stands, the sibling may be un-normalized. For instance, for a network connection, we could see in the sibling "inbound", "INBOUND", "Ingress", and "INGRESS" all with the same integer sibling. I believe there is more value if we can define the sibling to be set to a normalized string, and in the exceptional case of "other", we can have a convention. Potentially allowing 4 variations of the string sibling does not render the field very useful for anything, and it overlaps with the intent of "raw_data".

In cases where there is a truly new value that does not fit cleanly into the OCSF-defined enum, the value should be able to be utilized cleanly and consistently (I'd advocate for a convention that doesn't require parsing; so upon reflection, I would argue that a prefixe of "Other" isn't ideal. Maybe the original value of the mapped field, or to go one step further, a re-interpretation of the original field to match some convention - i.e., no spaces).

[pagbabian-splunk]

I think I get you now. The sibling string should be exactly the enum label. If Other, the sibling string should be exactly what the source produced. Not prefixed by 'Other:' and the siblings should not be prefixed by 'ref_' because for standard enum values, they aren't references to an external non-normalized value.

[floydtree]

Side Note - A similar proposal was submitted in October as well - #313

[pagbabian-splunk]

One more point- the above proposal didn't make mention of whether the enum label is required in the sibling or whether it is optional. The siblings themselves are all marked as Optional, but would effectively be silently promoted to Required if Other was the enum value.

[JasonKeirstead]

I voted for free-form text, because the assertion in the issue description I do not agree with.

"No offer of normalization for applications that utilize or prefer the strings for things like captions or visualizations" - when taking data and rendering it in a user interface, one has to determine whether the data element has to support I18N. If it does, then displaying a string as-is, is not possible. One has to use the normalized field - in this case, the integer - then look it up for translation. If the field on the other hand, does not need to be translated, then one should preserve the original information the sender is actually sending - if they send it in all caps, it should be shown in all caps, not corrupted/ manipulated.

The TL;DR is - if the string siblings have to be normalized as well, then why have them at all in the first place?