Use layer4 as a listener wrapper OR pipe layer4 connection to http FOR ocserv multiplexing

[vnxme]

Hi,

My intention is to multiplex ocserv (OpenConnect VPN server, it could be treated as a TLS-capable non-HTTP application) and caddy on a single port. What I would like to achieve is the following logic: caddy listens on :443, if it receives a TLS connection with SNI ocserv.example.com, it passes it through without terminating TLS to another host where ocserv runs, otherwise it terminates TLS on its own and serves example.com and its subdomains other than ocserv.example.com. An additional point is I would like to use caddyfile adapter to configure caddy for both layer4 and http/tls, but I understand that's another issue.

It's possible (as of [email protected] and caddy-l4@ca3e2f3) to make the layer4 app listen on :443 and the http app listen on another port (e.g. :444), and proxy non-ocserv.example.com from :443 to :444. For reference, see below the relevant part of autosave.json. However, given both layer4 and http apps are inside caddy, I would like to find a way to do the same more efficiently without proxying.

autosave.json, layer4 proxy, working

"layer4": {
    "servers": {
  	  "example": {
  		  "listen": [":443"],
  		  "routes": [
  			  {
  				  "match": [
  					  {
  						  "tls": {"sni": ["ocserv.example.com"]}
  					  }
  				  ],
  				  "handle": [
  					  {
  						  "handler": "proxy",
  						  "upstreams": [
  							  {"dial": ["ocserv.address.local:443"]}
  						  ]
  					  }
  				  ]
  			  },
  			  {
  				  "match": [
  					  {
  						  "not": [
  							  {
  								  "tls": {"sni": ["ocserv.example.com"]}
  							  }
  						  ]
  					  }
  				  ],
  				  "handle": [
  					  {
  						  "handler": "proxy",
  						  "upstreams": [
  							  {"dial": ["localhost:444"]}
  						  ]
  					  }
  				  ]
  			  }
  		  ]
  	  }
    }
},

As far as I understand, this is a topic which has been discussed a few times so far, and there are a number of relevant issues/PRs:

• core: Wrap listeners with type that can pipe #5040 As described by @mholt, the layer4 app could hand off a connection to the http app without having to proxy it, which is very efficient. However, it is still a draft PR, and it lacks documentation (no sample config at all), so I can't even try it.

• Pass non terminated (SNI matched) TLS connection to http app mholt/caddy-l4#70 Here @coolaj86 mentioned that he managed to multiplex SSH with caddy thanks to the listener wrapper functionality implemeted by @WeidiDeng and further discussed here (implement listener wrapper for use with http app mholt/caddy-l4#78). Although it lacks documentation, I tried a similar config, which you can also see below, without success. May also be related to Non blocking matchers & matching timeout breaking l4 as a listener wrapper mholt/caddy-l4#207.

  autosave.json, layer4 listener wrapper, not working

  The second wrapper - TLS - seems to never get a connection, so this way only ocserv.example.com works, other domains won't work with ERR_CONNECTION_CLOSED in Chrome.

  "listener_wrappers": [
      {
    	  "wrapper": "layer4",
    	  "routes": [
    		  {
    			  "match": [
    				  {
    					  "tls": {"sni": ["ocserv.example.com"]}
    				  }
    			  ],
    			  "handle": [
    				  {
    					  "handler": "proxy",
    					  "upstreams": [
    						  {"dial": ["ocserv.address.local:443"]}
    					  ]
    				  }
    			  ]
    		  }
    	  ]
      },
      {
    	  "wrapper": "tls"
      }
  ],
  

• Feature request: invoke handler mholt/caddy-l4#211 Here @legobeat suggested another approach to solve the problem and collected a couple of extra links to the relevant topics.

Thus, my question is whether there is any way to make ocserv multiplexing work without proxying other subdomains (any mistake in layer4 listener wrapper config above?) or implement such a functionality in caddy and/or caddy-l4?

Regards,

[vnxme]

Seems like there is a related PR battle between mholt/caddy-l4#208 and mholt/caddy-l4#210. Hope they will soon find a mutually acceptable solution to restore the old behaviour (fallback next handler), and the layer4 listener wrapper config above will probably work.

Nevertheless, it would be great if #5040 could be finalised and some documentation/examples provided. Generally it might be an even better solution than listen wrappers.

[mholt]

Yeah, layer4 plugin is currently undergoing some revisions.

To serve TLS without terminating it, you'll need the layer4 module. Caddy's HTTP server can only terminate TLS. Generally speaking, any and all protocol multiplexing will need the layer4 module.

[vnxme]

Following @ydylla's config samples, namely the one named "listener-wrapper-matcher-with-no-handler.json" (see mholt/caddy-l4#209), I've managed to make my layer4 listener wrapper config work using the suggested workaround.

autosave.json, layer4 listener wrapper, working

"listener_wrappers": [
  {
  	"wrapper": "layer4",
  	"routes": [
  		{
  			"match": [
  				{
  					"tls": {"sni": ["ocserv.example.com"]}
  				}
  			],
  			"handle": [
  				{
  					"handler": "proxy",
  					"upstreams": [
  						{"dial": ["ocserv.address.local:443"]}
  					]
  				}
  			]
  		},
  		{
  			"match": [
  				{
  					"not": [
  						{
  							"tls": {"sni": ["ocserv.example.com"]}
  						}
  					]
  				}
  			]
  		}
  	]
  },
  {
  	"wrapper": "tls"
  }
],

[vnxme]

@mholt Thank you for responding. The layer4 module is absolutely amazing! Really appreciate your time and effort spent on it.

Do you have any plans to merge it into caddy core? Or at least make it configurable the way caddy works? I mean the caddyfile adapter.

And what do you actually think about #5040?

[mholt]

Do you have any plans to merge it into caddy core? Or at least make it configurable the way caddy works? I mean the caddyfile adapter.

We've talked about it. We need to resolve some of the glaring issues with it first, obviously. But it's a good candidate as it seems to be getting quite popular / in higher demand as time goes on. It'll be a lot of work, especially that Caddyfile support.

And what do you actually think about #5040?

I like that change still, but I have yet to find time to test it end-to-end. If someone who is savvy with Go wants to try wiring up the Layer4 module to support the exported Pipe() method, then we can probably make some progress on it sooner. I've just been very behind 😅