ITVR - task queue creates too many database connections

[tim738745]

Describe the Bug
From time to time, the task queue service may create and leave open too many database connections; this impacts the ability for the backend to connect to the database.

Expected Behaviour
This should not happen.

Actual Behaviour
This does happen.

Implications
The backend may not be able to connect to the database, and the task queue itself is unable to make new connections to the database.

Steps To Reproduce
This is hard to reproduce, because it does not happen deterministically. The task queue has 4 worker processes, and the threads in each process manages its own database connections. While django-q, the library we're using, does close db connections, it only seems to close them at the beginning of each task/job. So it could be the case that a scheduled job that opens some connections executes in process 1, and then in process 2, and then process 3. This would lead to an accumulation of connections. However, if it executes in process 1 the second time around, the connections it created the first time around would be closed. We do not determine in which process a job executes. So to address this issue, we should close db connections at the end of each task/job (see scheduled_jobs.py).

Also, the job that expires rebates runs each expiry action inside a transaction; maybe each transaction opens a new connection?