Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a warning banner when no EventId and/or EventTime is found #5

Open
itsnotapt opened this issue Feb 8, 2023 · 1 comment
Open
Assignees
Labels
enhancement New feature or request

Comments

@itsnotapt
Copy link
Collaborator

The banner would be a yellow warning that informs the user that EventId and/or EventTime are missing, and therefore tagging will not enabled for this query. We can point them to docs on tagging.

The banner should be closeable as well.

@itsnotapt itsnotapt added the enhancement New feature or request label Feb 8, 2023
@itsnotapt itsnotapt self-assigned this Feb 8, 2023
@ramen0x3f
Copy link
Contributor

If you would like a KQL layer to attempt to safely resolve missing columns, the following function lets you define commonly used alternate column names that should be mapped to EventId and EventTime. It also handles columns with the right name but wrong datatype.

It can be used for automated normalization or be manually invoked in queries:

let ValidateColumns = (T:(*), FailOnError:bool=true, 
        CustomIncoming:dynamic=dynamic({"ReportId":"EventId", "ReportTime":"EventTime", "Timestamp":"EventTime"}), //Map incoming to required output columns
        RequiredOutgoing:dynamic=dynamic({"EventId":"string", "EventTime":"datetime"})) { //Required for TIM tags
    let fixed = (T //Check and rename fields as needed
        | extend Field_List=pack_all(true)
        | mv-apply x=bag_keys(Field_List) to typeof(string) on (
            extend k=iff(bag_has_key(RequiredOutgoing, x), x, //Extract key if it's relevant (default or custom map)
                        iff(bag_has_key(CustomIncoming, x) and bag_has_key(RequiredOutgoing, tostring(CustomIncoming[x])), CustomIncoming[x], "")) 
            | summarize Field_Relevant=bag_merge(make_bag_if(bag_pack(k, Field_List[x]), isnotempty(k)), dynamic({"":""})), //Keep required values
                Field_WrongType=strcat("Incorrect type: ", strcat_array(make_set_if(k, isnotempty(k) and gettype(Field_List[x]) != RequiredOutgoing[k]), ", "), "."),
                Field_Missing=strcat("Missing: ", strcat_array(set_difference(bag_keys(RequiredOutgoing), make_set_if(k, isnotempty(k))), ", "), ".")
            | extend ColumnValidationErrors=strcat_array(set_difference(pack_array(Field_Missing, Field_WrongType), pack_array("Missing: .", "Incorrect type: .")), " ") //Format
        )
        | evaluate bag_unpack(Field_Relevant, columnsConflict="keep_source") //Unpack relevant fields (in case something was renamed)
        | project-away Field_*, _Empty //_Empty column is because we merge an empty bag so the bag_unpack doesn't fail.
    );
    let err = toscalar(fixed | take 1 | project isnotempty(ColumnValidationErrors)); //Assert requires a scalar, so we have to store as vars like this.
    fixed | where assert(not(err) or not(FailOnError), "You had missing and/or invalid fields. Call this function with FailOnError=false for details.")
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants