Headless Support + Scaleable API / Tasks

[kashalls]

This was brought up in the community discord today, I just wanted it to be brought up for discussion.

I am interested in adding a scalable backend api to zipline to allow horizontal scaling. As the project stands, there is no pheasable way once you max out your one instance. Zipline has an option to disable the website, but as @diced said "it still runs nextjs which is kind of dumb".

@ItzDerock mentioned that it would be possible to disable tasks and make it somewhat scalable.

---

Suggested Solution

The api should be separated from nextjs. It should be in its own container that can be replicated as resources are allowed. Any cron-based tasks should also be in its own container.

This shouldn't impact users who use yarn build to run zipline as it will be implemented as monorepo. Users who use docker-compose should not necessarily be affected either.

---

I currently run my instance of zipline in my k3s home cluster with what I would consider pretty generous resource limits. I added this to show how I am currently using it.