Adding support for bind before connect.

[zakattacktwitter]

We use http to get 1000s of files per second and maintain that rate for hours. This leaves a lot of sockets in the TIME_WAIT state and will eventually cause the kernel to run out of ephemeral ports to assign. The solution is to bind before connect and keep trying to do so until an ephemeral port is found.

You can read about the solution here

https://idea.popcount.org/2014-04-03-bind-before-connect/

This is a very hard condition to unit test, but we test it empirically all the time. Thanks!

[diegonehab]

I understand the problem you are trying to solve. Is there any reason why this can't be solved without modifying LuaSocket? For example, you can use the "create" field in the request table to give you complete control over the socket object. You can, for example, return a socket that is already bound.

[zakattacktwitter]

I think I can do it via the optional create function. I thought it would be nice for others, as its a non-trivial scenario to debug (need high load on a machine with a very fast network to the http server). But if you are not interested in it, then I can close this PR.

[diegonehab]

I am definitely interested in the issue, as it is not something I had considered before. The use case for the "create" field has always been non-blocking I/O, as per the "check-links.lua" program in etc/. It would be nice to see if it can be used to solve your problem as well. I believe it would. In fact, there is a pull request I am ashamed not to have gone through yet that would simplify this even further, by factoring out and exposing the url parsing that goes on to create the request table from the url. I tend to prefer general solutions to specific ones.

[zakattacktwitter]

Thanks, yeah, understood, my goal was just to provide future users with an
"industrial strength” solution so they can just use http as is and not have
to worry about TIME_WAIT. Maybe I can move this to a create function and
then provide an example create implementation somewhere, maybe in etc/,
sound fair?

On Thu, May 28, 2015 at 9:40 AM, Diego Nehab [email protected]
wrote:

I am definitely interested in the issue, as it is not something I had
considered before. The use case for the "create" field has always been
non-blocking I/O, as per the "check-links.lua" program in etc/. It would be
nice to see if it can be used to solve your problem as well. I believe it
would. In fact, there is a pull request I am ashamed not to have gone
through yet that would simplify this even further, by factoring out and
exposing the url parsing that goes on to create the request table from the
url. I tend to prefer general solutions to specific ones.

—
Reply to this email directly or view it on GitHub
#140 (comment).

[diegonehab]

Certainly. It will look even better once I merge the factoring of url parsing.

[diegonehab]

Did we reach an agreement here?

[ewestbrook]

Does using create() as suggested solve the problem for you? Does anything still need to be done here?

We're reviewing open items in preparation for a release. If action is needed here, please add a comment. Otherwise, this issue will be closed on or after 24-Feb-2019.

Thanks!