Support to access raw HTTP headers

[jimdigriz]

I have a requirement to inspect the ordering and case-sensitive-ness of the HTTP headers in the request for User-Agent identification. As cowboy only supports normalised header access, I wired in a headers_raw boolean option to make the request object contain headers_raw:

#{bindings => #{},body_length => 0,cert => undefined,
             has_body => false,
             headers =>
                 #{<<"accept">> => <<"*/*">>,
                   <<"host">> => <<"localhost:8080">>,
                   <<"user-agent">> => <<"curl/7.52.1">>},
             headers_raw =>
                 <<"Host: localhost:8080\r\nUser-Agent: curl/7.52.1\r\nAccept: */*\r\n">>,
             host => <<"localhost">>,host_info => undefined, 
             method => <<"GET">>,path => <<"/">>,path_info => undefined,
             peer => {{127,0,0,1},47606}, 
             pid => <0.124.0>,port => 8080,qs => <<>>,ref => test,
             scheme => <<"http">>,
             sock => {{127,0,0,1},8080},
             streamid => 1,version => 'HTTP/1.1'}

My use case is then to then access this via middleware to do browser identification.

Before making this PR 'worthy', at this point I really looking for feedback on your appetite for this sort of thing, and if this is the best way to plumb it in and maybe you have some suggestions and/or recommendations?

[essen]

Not terribly fond of special features to help track users especially when they're not compatible with HTTP/2. But let's leave it open for now and see if it could benefit other use cases.

[jimdigriz]

Though everything can be used for Evil(tm), my personal use is when the User-Agent claims to be Windows/Chrome but is actually Linux/PhantomJS. The idea is that combined with other signals then if, for example, they are lying about the User-Agent and the IP is from a data center then they probably are up to no good and should be blocked.

Thanks for the feedback.

[jimdigriz]

FYI, raw headers (capturing the header ordering and case-sensitively of the name) provide no benefit for tracking users. There is a N:1 relationship of browsers (where N is a group of same family and range of versions) to a particular signature.

This gleans you no more information about the user that you could obtain just from parsing the value of the User-Agent its-self which is more unique (I see user id's in there...the internet is awful...).

[essen]

Sure but that doesn't change things much, field order has no meaning in the protocol (except for fields of the same name) and you risk burning yourself if/when legitimate clients end up doing the same. In Cowboy I would prefer to stick as close as possible to the protocols.

Still, maybe someone has a need for this for other reasons so let's wait a few months and see if it can be useful.

[essen]

You're not alone wanting this golang/go#24375

Also https://www.sans.org/reading-room/whitepapers/detection/paper/34460

I like the principle behind the implementation of the PR though. So if there's more demand I'm not against including.

[jimdigriz]

I'm looking into doing the HTTP/2 version as I need it for my use case too. I'll submit those changes once I have them. Thanks

…

On 31 October 2018 16:21:46 Loïc Hoguin ***@***.***> wrote: You're not alone wanting this golang/go#24375 Also https://www.sans.org/reading-room/whitepapers/detection/paper/34460 I like the principle behind the implementation of the PR though. So if there's more demand I'm not against including. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[essen]

Hmm are you going to keep raw data also, perhaps just the HEADERS block? Just curious.

[jimdigriz]

Depends on how much I need. For HTTP 1.x I only need the header order and not the values, no idea what I need to do for HTTP/2 fingerprinting, but I am currently busy with SSL fingerprinting.

…

On 31 October 2018 16:57:18 Loïc Hoguin ***@***.***> wrote: Hmm are you going to keep raw data also, perhaps just the HEADERS block? Just curious. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[essen]

At IETF103 it was mentioned it could become a requirement for intermediaries to send headers without concatenating them, so it's possible a more general solution will be required in the future. If that happens I'll favor a mode to select whether Cowboy should operate using maps or lists of headers, with the caveat of course that lists will be slower to manipulate.

[jimdigriz]

Attached is a the HTTP/2 supporting updated implementation that I use in my passive browser fingerprinting library.

Thoughts?

I'll write some unit tests if this looks to be an acceptable approach.

[essen]

Hm I would rather have a more consistent approach, from what I understand HTTP/1.1 gives a binary while HTTP/2 gives the parsed list of headers. Is there any reason for HTTP/1.1 to be a binary?

[jimdigriz]

For my use case, no as in my project I actually convert the binary to an KV list anyway. :)

As this is 'raw' I thought for the generic case (as those users have particular needs and processing ideas) it was maybe best to expose what ever was closest to the wire representation for full headers (otherwise maybe I should just return the frame?). Mostly as some may for HTTP/1.1 decide they want to use a binary match to search for leading white space in values without having to iterate over a list.

Happy to add something to the HTTP/1.x side to fix up the returned value into something more resembling cow_http:headers(); I suppose those who want the binary can just fold it back up into a binary if they wanted.