Microsoft ETW (Event Tracing for Windows) Listener. Look for a Beats to deal with Service Fabric Logging!

[MonDeveloper]

The Microsoft Service Fabric platform (a full system from microsoft to develop & run microservices) uses the Microsoft ETW subsystem to manage both Infrastructure and Application Logging, in order to ship those logs into Elastic without implementing esoteric & ad-hoc configuration we are wondering if Beats could implement a ETW message receiver so everything will run according the official architectural idea.

[geeko76]

I double quote the request, it would be very very very helpful !!
Thank you for support.

[Rofos0574]

It would be a great improvement. I really need it and it would be a great functionality for all the people who want to use elastic search functionalities to process the logs coming from service fabric.

[rosfisi]

It would be very interesting and helpful! A great help! Thx!

[ruflin]

Thanks for the request. As I'm not familiar with Microsoft ETW it would be nice if you could provide some links to read up the details on how the infrastructure and application logging works and how a beat could hook into this.

[andrewkroh]

There are two other requests for ETW on the forums.

https://discuss.elastic.co/t/windows-dns-diagnostic-logs/53344
https://discuss.elastic.co/t/does-winlogbeat-work-with-the-semantic-logging-application-block-slab/47524/2

[MonDeveloper]

@ruflin here [https://msdn.microsoft.com/en-us/library/dn775006.aspx] you can find an overall description of the SLAB (Semantic Logging Application Block) "https://msdn.microsoft.com/en-us/library/dn775006.aspx"; it describes 2 scenarios: "In Process" & "Out Of Process", what we are looking for is a Beat to replace the poorly developed "Out-of-process windows service" to intercept the logs at OS level (here it deals with the ETW part of the SLAB) and ship them to the store (Elastic or Logstash in our case).

Looking at this [https://technet.microsoft.com/en-us/library/jj714799.aspx] description of ETW, the role we are wondering to have implemented with Beats is the "ETW Consumer" which can be feed live through "real time delivery messages" and/or via bulk reading the "logged events files (.etl)" (the "bulk" way is a nice to have :-)).

Here [https://slab.codeplex.com/releases/view/125681] you can find the project used for the ETW Consumer role written in .NET, maybe you could take inspiration from it.

[elvarb]

This could be an inspiration https://azure.microsoft.com/en-us/documentation/articles/service-fabric-diagnostic-how-to-use-elasticsearch/

[EmFl]

Could someone confirm that this is still unsupported and if there's any hope of seeing it integrated in the near future ?
Thanks

[ruflin]

@EmFl There is no implementation for this yet, but it would be really great if someone that runs such a system could take a stab at it. By now also the inputs in Filebeat have become much more modular which makes this easier to add in case this would be an input / prospector.

[nareshkhatri81]

are there any updates when this enhancement will go ?
are there other alternatives to forwards ETW logs to ELK Stack if beats doesnt consume ETW logs ?

[neu5ron]

++++++1,001
ETW gives the ability to provide additional context and information/logs in addition to native windows event logs and even sysmon. In fact, the logs/information are so rich, that there is a developer for Google who uses it for profiling and such:
https://twitter.com/BruceDawson0xB/status/940747236614574080

ETW also gives the ability to provide equal to and a lot of times greater than what many EDR products cost. For example, there was a large "EDR" vendor that used sysmon up until just last 4-6 months. They now heavily rely on ETW. Implementing ETW drastically reduces the cost of relying on third party vendors and code, to provide what is just a built-in capability into windows already.
It provides WMI tracing that is far superior to sysmon WMI, dll loads, (true) dns debug logging, etc..

If I am not mistaken, windows powershell (debug) logging (ie: 4103,4104, etc...) started off as an ETW that eventually became supported by Microsoft through GPO's and such.

reading material / insights / info:

• https://randomascii.wordpress.com/2015/09/24/etw-central/
• https://docs.google.com/presentation/d/1dkrldTTlN3La-OjWtkWJBb4hVk6vfsSMBFBERs6R8zA/mobilepresent?slide=id.g26b44710fe_0_216
• https://www.blackhat.com/docs/us-15/materials/us-15-Graeber-Abusing-Windows-Management-Instrumentation-WMI-To-Build-A-Persistent%20Asynchronous-And-Fileless-Backdoor-wp.pdf
• https://github.com/Microsoft/krabsetw

more elastic reference:
https://discuss.elastic.co/t/trace-event-logs/75537

[RamblingCookieMonster]

fwiw, someone has this working / almost ready for Splunk : ) https://twitter.com/vector_sec/status/1170140453603336192 : )

[maekee]

Need to collect the DNS Analytical logs for threat hunting, any news on winlogbeat supporting this?
Or do anyone have a workaround solution in place? I use logstash/winlogbeat.

Thanks

[jdcc2]

+1

[neu5ron]

Need to collect the DNS Analytical logs for threat hunting, any news on winlogbeat supporting this?
Or do anyone have a workaround solution in place? I use logstash/winlogbeat.

Thanks

you can use something like SilkETW, powershell service that collects and stores the ETW into an event log. Here is a blog describing the whole process all the way to sending the logs to ELK.
https://medium.com/threat-hunters-forge/threat-hunting-with-etw-events-and-helk-part-1-installing-silketw-6eb74815e4a0

[cmiscloni]

+1

[jamallmahmoudi]

Thank you for supportt

[9r00t-z]

+1

[Zalgo2462]

PacketBeat could also take advantage of ETW for its packet listening on Windows, removing the need to pay for a license for NPCAP.

[Ripcurl99983]

Will there be any solutions to ingest ETW logs any time soon? I need PowerShell and DNS .etl logs ingested in my stack. I sold my boss on spinning up an ELK stack a while back. He keeps sending me use cases, and I simply can’t deliver without building crazy pipelines that he won’t give me time to build. I’m a one man ELK show, and I suspect there are many in my position. Paired with a lack of support for FMC logs, I have more losses than wins on the ole’ Elasticsearch scorecard. It’s only a matter of time before our stack is dismantled. I know this is a tall order, but ETW logs aren’t exactly a new technology anymore. I love Elasticsearch, would like continue building our stack!

[felix-lessoer]

@andrewkroh @ruflin
Any update on this? Already open for multiple years..

[g3ppyT]

Any news on this?

[vinnimin]

+1 Updates? This functionality would be awesome...

[Ruuucker]

Hello folks, there is a dirty workaround to read logs from ETW providers in, at least, Winlogbeat.
Winlogbeat could read pure .etl extension files, so in my case I just create a ETW session with providers and write logs to an .etl file. After that you can set the log file in a config like this:
winlogbeat.event_logs:

• name: C:\Windows\System32\winevt\Logs\ADtest.etl

And, that's it. If you have trouble with the "The template for an event definition cannot be found in the resource." error, I suggest you to include XML of the event like this:
winlogbeat.event_logs:

• name: C:\Windows\System32\winevt\Logs\ADtest.etl
  include_xml: true

Hope this helps someone.

EDIT: Winlogbeat can't read continuously .evtx nor .etl files, so this is only about reading one part of logs by each one launch of winlogbeat.

[peas100]

Hi @Ruuucker ,
I tried this option but still it read once winlogbeat starts. Once the file is read, it doesn't give me any logs(i.e. it doesn't stream continuously.)

What settings did you use?
Mine are as below

- name: C:\Windows\System32\winevt\Logs\ADtest.etl
# no_more_events: stop -> This I have deliberately commented out, in a hope that it wouldn't stop and read file continuously

#winlogbeat.shutdown_timeout: 30s
winlogbeat.registry_file: evtx-registry.yml

[Ruuucker]

Hello @peas100,
I am sorry, this is a mistake of mine, I had to double check it. Winlogbeat can't read continuously .evtx nor .etl files as it said by people in other topics, so this is only about reading one part of logs by each one launch of winlogbeat.
By now I have thoughts about using tracerpt utility in scheduler or something like this to repeatedly convert .etl logs into something filebeat could read without problems and than send in to elastic.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[Flexxi51]

Any news?

[LMey]

Is there any chance that the WMI functionality from Logstash could be moved into metricbeats?

Reasons:
As I understand logstash is a central function. So using this requires remote calls and security configuration
metric beat is where metrics are collate and set through to elastic. So why go somewhere else for your metrics.

It is also requested here: #29072 and here: #8614

[franklymrshankley]

There are some good libraries available for golang these days which allow the reading of events live from ETW, rather than having to export events to a file and then parse.

Perhaps these would be a good addition to winlogbeat to enable the real-time pulling of ETW logs from various providers?

Links:

• https://github.com/0xrawsec/golang-etw
• https://github.com/bi-zone/etw

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[franklymrshankley]

This is still relevant and needed functionality, as far as I'm concerned.

[RamblingCookieMonster]

This is like winlogbeat - You enabled collecting Windows Event Logs, and thus, enabled collecting data from every service that uses the Windows Event Logs.

ETW might not be as widely used, but it is the same idea. If you enable collecting this data, you will enable logging from a variety of services that happen to use ETW, which you do not handle today. Definitely still needed IMHO.

[jamiehynds]

FYI @chemamartinez

[jamiehynds]

@RamblingCookieMonster @franklymrshankley we are currently working on support to ingest logs via ETW (PR linked above). Our initial focus will be around supporting DNS Analytical events via ETW. Curious if there are other events you're interested in ingesting via ETW? Thanks!

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[jamiehynds]

Closing as we now have added a new ETW reader: #36914