Azure AD SignIn Logs Not Parsed

[dsplice]

When attempting to ingest AzureAD SignIn logs downloaded from the portal in JSON, the files (EG "InteractiveSignIns_2023-05-08_2023-05-09.json") are not ingested at all.

When attempting to ingest AzureAD SignIn logs downloaded using powershell (eg "Get-AzureADAuditSignInLogs -Filter "userPrincipalName eq '[email protected]'" | ConvertTo-Json | Out-File -FilePath C:\Temp\test\dereklogin.json") the logs are ingested but without any parsing or enrichment.

The data appears different from the data released in FOR509. Not sure if they need a different way of downloading, or if the format has changed.

[Pierre450]

@dsplice The problem with getting logs from PowerShell commands is that Microsoft keeps making changes and keeps threatening to deprecate these commands to force everyone to use Graph API. For Azure logs, it's much easier to send the logs to a storage account (which is what we use in FOR509).

[dsplice]

@Pierre450 Understood. Doesn't help much with my current engagement but will be trying the storage account method in the future.

[Pierre450]

@dsplice Take a look at this project: for509.com/invictus-suite
It's from Korstiaan Stam who is a FOR509 instructor. He has done a lot of work to make sure to extract the logs in a way that can be imported into SOF-ELK.
In the example you gave, it's highly likely that "ConvertTo-Json" cmdlet is doing some strange things. When I was testing with Korstiaan, we had to make sure to force UTF8 or the files wouldn't import.

[dsplice]

I gave Invictus-Suite a try. It uses the command I listed in the original issue. I tried importing the resulting file, and the same outcome. It would only parse it generically. No fields brought out, timestamp was all set to the extraction time. So I do not think it is the ConvertTo-Json that is causing the issues, but will continue to dig.

[hutspot79]

I have the same problem. All my Microsoft 365 logs, both in CSV and JSON format, are only processed globally. The content log fields are not processed and displayed in Kibana. The date and time were set to the moment of adding them to SOF-ELK. I used the Invictus extractor suite to export the logs via Powershell. I have written the logs directly as JSON files. Are there any options left to try?

[dsplice]

I saw that the files I was trying to play with were in UTF-8 when the signin logs from Invictus were in UTF-16. But since the parser is forcing UTF-8, I doubt that should be an issue. I did notice a very different amount of data coming from Invictus vs other methods, plus key names in the file appear to have different capitalizations. I might look at seeing what it will take to create a logstash config to parse it through, just need time.

[philhagen]

I suspect this is related to #285. If you can send me a sample of a problem file, I can dig in

[mdenzel]

I can confirm this issue. Unfortunately, I cannot provide sample data as it is not my data. However, I tried all of the following and importing always failed:

• used Invictus-Suite (json output)
• used the Powershell commands of Invictus-Suite manually (json output)
• used Powershell commands to output csv
• converted the json output into UTF-8
• converted the json output into UTF-16

[spivjohn]

@philhagen I see other people are having the same issue that I experienced today. I have exported the Azure SignIn logs using the Invictus-Suite Get-ADSignInLogsGraph. This as you know creates output files in JSON format but when they are ingested into Sof-ElK they are not parsed correctly. Is this still a known issue with no fix as yet?

[dabltrabl]

Hello all do we have some updates on this issue?
@philhagen Ive tested the parsers from last image and still have the same problems
Im interested in writing this parses on my own in future on additional log samples so if you're available to share some details and thoughts or some more information on these it would be largely beneficial. Even the time you would work on new one would be excellent to see.
Ive seen your comments on currently available parsers and they give some insight but its hard to start from 0 without some more context. Ill start tomorrow so i hope i will share the successful result soon...
I can share a sample file just contact me, i have made a lab for this use case.
BR

[philhagen]

Apologies. Been a very busy few weeks but after the new VM released, I should have some bandwidth to look at this again.

Couple steps:

• Have you tested these files on the new ECS release? (v2024-10-11)
• Can you provide sample files (via email is fine - they will not be shared publicly) that are not being parsed?

With answers to those, I can start to dig in. However, in discussion with @Pierre450, and looking at FOR509 lab material in particular, we've had trouble reproducing it.

[philhagen]

I've been working with @invictus-korstiaan on this quite a bit and we think we have a good idea of what the two main problems are (need to get multiline JSON --> ndjson and a field naming capitalization issue). We're working together on the best approach, as well as aligning that approach to data pulled via methods other than the Microsoft Extractor-Suite tools, and the ECS naming scheme. It'll be some work but I wanted to confirm you know it is being actively worked!

[invictus-korstiaan]

@JoeyInvictus tagging for visibility, let's see if we can change UAL output to NDJSON

[JoeyInvictus]

Hi, thanks for bringing this issue to my attention. In the future, feel free to create an issue on the MES page so we can address it as quickly as possible. We will likely add a SofElk parameter to all the Unified Audit Log functionalities of the tool to write the output in the required NDJSON format. I’ll experiment with this and try to get it working in SOF-ELK. This feature will be included in our next update.