New package for inclusion - vsivsi:job-collection

[mitar]

Package/project name & description

job-collection is a powerful and easy to use job manager designed and built for Meteor.js. It solves the following problems (and more):

• Schedule jobs to run (and repeat) in the future, persisting across server restarts
• Create repeating jobs with complex schedules using Later.js. See demo here
• Move work out of Meteor's single threaded event-loop
• Enable work on computationally expensive jobs to run anywhere, on any number of machines
• Track jobs and their progress, and automatically retry failed jobs
• Easily build an admin UI to manage all of the above using Meteor's reactivity and UI good

Links

• Atmosphere
• Github repository

Current status of the project

• Abandoned

Reasoning

I think this is the best background task package for Meteor, supporting many use cases.

Progress

• Discussion finished
• Moved / Forked
• Team Created
• Meteor Org Created and set as maintainer
• communitypackages org added as a maintainer on Atmosphere

[StorytellerCZ]

Have you looked on jobs from Max Savin? https://github.com/msavin/SteveJobs

[StorytellerCZ]

Also I think at this point we need to get things moving on packages that we have move into the org before admitting any new packages.

[mitar]

We have people interested in maintaining, and they are already doing that in their fork, so not sure why making it more official and organized would be a bad thing?

[LarsBuur]

vsivsi:job-collection is mature and heavily battle-tested, also already written in CoffeeScript.

[sebakerckhof]

@StorytellerCZ Last time I looked steve jobs package was lacking some features compared to jobs collection. Most notably jobs collection allows you to have work distributed amongst multiple servers and to have workers outside of Meteor. There's a package for pure nodejs workers or you can even have workers in any other language using a rest api to fetch jobs.

[vsivsi]

job-collection author here. I fully support transitioning this package into community support. I'm no longer doing any active Meteor development, so this just makes sense given that others are still actively using it and willing to keep it going.

I would hope to be able to similarly transition its sister package file-collection.

Sorry for being late to the discussion, I've been out of action with a terrible case of the flu for a couple of weeks. Get those shots, it's awful!

[vsivsi]

A couple of wrinkles... The job-collection package has dependencies on some more generic node-js libs.

1. https://github.com/vsivsi/meteor-job is my own base library that enables pure node-js code to play with job-collection. But job-collection itself is also built on this, so it needs to somehow come along, even though it's not a pure Meteor package.

2. https://github.com/oortcloud/node-ddp-client The pure node-js DDP client (and everything it depends on, eg. https://github.com/oortcloud/ejson) is required for pure node-js worker authentication and interoperability.

3. https://github.com/vsivsi/ddp-login Which streamlines robust worker authorization flow.

There are probably more. The point being that just maintaining job-collection but neglecting these other pieces will probably lead to difficulty down the line, so be aware.

[SimonSimCity]

Any update on this? I've already created my own personal fork of this and am running it in production nowadays (https://atmospherejs.com/simonsimcity/job-collection). I'd like to get it back into a non-forked maintained stream.

I'm also on @StorytellerCZ that we might have to do some research here to investigate on where the different job packages are good in and where the individual implementation falls short in. I'm all in for diversity, but it should be reasonable when to join which path.

[sebakerckhof]

We are also using the fork from @SimonSimCity in our main product for a while now.

[wildhart]

Just a note on this - I am using my own fork/rewrite of msavin/sjobs, mainly due to this significant inefficiency in sjobs which is WONTFIX (every new job type/queue you create results in a 3 second setInterval which hits the db - my app currently has 20 different job queues/types!). Also, despite claiming to be multi-server, job queues cannot be paused across servers and the list of paused jobs do not survive a server reboot.

My own version is much more efficient as it uses a single observer on the job queue instead of 20 db lookups every 3 seconds! Mine can also persists paused job queues across multiple servers.

I haven't used job-collection and it looks awesome, but I personally would not recommend endorsing msavin/sjobs. Having said that, I wouldn't endorse mine either because I haven't written tests for it!

Edit: it looks like jobs-collection uses polling as well, which in my case could be very inefficient. The readme says:

job-collection must poll for this operation because it is time that is changing, not the contents of the database, so there are no database updates to listen for

My solution to that is to use an observer to watch for changes to a query looking for the next job across all queues, and within that observer set a timeout for the next due job. One mostly idle observer, one mostly idle timeout, for as many different job queues as you want. Simple, efficient and effective.

[copleykj]

I think we can move ahead with this. Does someone want to take the lead on getting the repo transferred and setting up orgs, teams, and permissions?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mitar]

Where are we about this one?

[copleykj]

@mitar If we have maintainers in place, feel free to move forward.

[mitar]

I think maybe we could share it between @SimonSimCity and me? I am still interested in it, but at the moment a bit low on time.

[SimonSimCity]

@mitar I could help with the initial setup, but I'm no longer working on a project requiring it, so I'm not all too eager investing much time into it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mitar]

Still planing. :-)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mitar]

Unstale.

[holmrenser]

@mitar, I am using a copied version of job-collection as a local package at the moment, but a community-hosted package would have my preference. I can help out from time to time.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[holmrenser]

Not stale from my perspective

[StorytellerCZ]

The setup is no problem. @mitar and @holmrenser if you want to become the maintainers on this then that is enough for me to get this started.

[holmrenser]

Sure thing! Let me know what I can do!

[mitar]

I am still interested in this (using it in some of my legacy apps) but it seems I will not have time to do anything concrete about this anytime soon. :-(

[holmrenser]

Proposal: @mitar and myself become maintainers, we put some (limited) effort into migrating this to meteor-community-packages as is, which I can probably mostly take care of, but might want to have some feedback from @mitar on.
Afterwards we can think about updates/changes and review pull requests. I have a few modifications that might be interesting to include, but we can think about that once this is safely incorporated into meteor-community-packages.

@StorytellerCZ what do you think?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[holmrenser]

Not stale. Pinging @mitar and @StorytellerCZ

[holmrenser]

@StorytellerCZ Reminder

[StorytellerCZ]

Alright, first step is to add communitypackages to the package on Atmosphere and then we can start on the migration.

[StorytellerCZ]

On another note, we can now also migrate in: https://github.com/percolatestudio/meteor-synced-cron

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.