Coroutines are not threads! A.K.A. lightweight async/await support

[IS4Code]

This is an amazing project from what I've seen, especially the DLR interaction, but I am a bit concerned by the coroutine implementation.

As a target introduction, coroutines in Lua should essentially allow you to effortlessly do something like this:

local url = "http://example.org/file"
local text = http.fetch(url)
print(text)

The core part is that in C# you would have to use await to continue the http.fetch call with code that operates on the result of the task, but as a consequence you have to mark every method as async. Lua does not have this limitation ‒ if you have a global scheduler that can spawn coroutines as needed, a single thread can switch between execution of active coroutines and implement I/O, waiting etc. using simple, non-blocking functions. This is why Lua is amazing for embedded applications and games or other event-based environments especially.

The point is that Lua coroutines are not (CPU/OS) threads, they are fibers and they use cooperative multitasking. This is why they should be amazing for interaction with TPL in .NET, and you could actually convert between coroutines and tasks with the help of a scheduler (a coroutine can have its TaskCompletionSource<LuaResult> and you can call coroutine.resume through Task.ContinueWith). They are also a generalized form of iterators, so a coroutine should also be an IEnumerator<LuaResult>, and calling MoveNext should call coroutine.resume.

My issue here (and please correct me if I am wrong) is that based on the LuaThread implementation, you seem to be creating actual .NET threads (through Task.Factory.StartNew) which obviously take resources, and based on your recommendation of using .Wait() to call async methods, you are blocking those threads from doing anything useful. This is terrible! The async/await APIs exist in the first place to move away from blocking to event-based API while keeping the convenient usage without cumbersome continuation-passing and closures. It's true that you can implement the coroutine API using threads and you can mimic fibers using threads, but that leaves you with all the heaviness of threads but with less capability.

I suggest writing a completely new code that actually does use state machines to implement this API, but I understand it may not be easy ‒ every call to a function, every indexing, every operator, and even every instruction (if hooked by debug.sethook) in Lua may yield, and I assume it would be very hard to compile that to IL; in essence turn every code into a state machine similarly to what iterators in C# do. However, making this would make it possible to have thousands of coroutines without depleting system resources, which would work even in single-threaded environments (WebAssembly), and interact with other systems (TPL or Unity's coroutine), which is definitely worth the effort.

[neolithos]

Full ack.

I started this project in the year 2012. And async/await didnot exists. That was the reason why I used threads in the first place. To implement coroutine.yield. I am also not happy with the solution. That is the reason, why but a simple async/await implemention on top in our "company" projects (this is partly open in the DES-project). But this is only a bad fix for it.

To support real async/await/cooperative multitasking, there is a need of a state machine. To keep it fast. A first problem is the syntax tree, that I avoided in the current implementation. This tree makes the split in the state parts easier. Because you need to keep track of the code flow and lifted local variables. If this will sometimes implemented, the rest will be easy with an own implemention of SynchronizationContext.

Bad is, there is no help within the Linq Framework (as I know). So, the complex logic that the c# team did, must be redone for NeoLua. Or do you know some library?

[IS4Code]

I am afraid I don't know of a usable library for that, but I wish you good luck on solving this!

[Matoneto]

I am interested in using NeoLua as a scripting language for a game. For this I will need to have many coroutines running at the same time that need to be pretty lightweight. I am talking about a few hundred, maybe event thousands at the same time.

Is this possible with the current implementation? If not, are you considering improving the implementation like @IS4Code suggested?

I really hope this works out, because I also tested other lua implementations for .net and even other scripting languages and NeoLua was way faster than all others.

[int2e]

I need to start thousands or even 10K coroutines
Do you have any plans to make changes?
@neolithos

[int2e]

I am interested in using NeoLua as a scripting language for a game. For this I will need to have many coroutines running at the same time that need to be pretty lightweight. I am talking about a few hundred, maybe event thousands at the same time.

Is this possible with the current implementation? If not, are you considering improving the implementation like @IS4Code suggested?

I really hope this works out, because I also tested other lua implementations for .net and even other scripting languages and NeoLua was way faster than all others.

I have the same needs as you, but neolua cannot be used in our project until the coroutine changes.

[neolithos]

I have a lot open issues in my payed job. So, this will take a while... Hopefully, over x-mas I will have a little bit free time.

I have already tested small parts, but I need to rewrite the whole parser and syntax tree.

Is there a nice async/await framework for linq somewhere?