Restart scheduler on error

[mkindahl]

If the scheduler receives an error, it will never restart again since bgw_restart_time is set to BGW_NEVER_RESTART, which will prevent all jobs from executing.

This commit fixes the issue by adding a GUC that can be set to the restart time for the scheduler, and set the default to 30 seconds. It also adds some additional variables to be able to shutdown the scheduler with a non-zero exit code, which allows the restart functionality to be tested.

Fixes #5091

[codecov]

Codecov Report

Merging #6195 (6791e70) into main (21a3ebd) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #6195      +/-   ##
==========================================
+ Coverage   65.04%   65.07%   +0.03%     
==========================================
  Files         246      246              
  Lines       56955    56957       +2     
  Branches    12621    12621              
==========================================
+ Hits        37045    37064      +19     
  Misses      18042    18042              
+ Partials     1868     1851      -17     

Files	Coverage Δ
src/bgw/scheduler.c	82.60% <100.00%> (+3.06%)	⬆️
src/guc.c	96.77% <100.00%> (+0.03%)	⬆️
src/loader/bgw_launcher.c	84.74% <100.00%> (+1.74%)	⬆️
src/loader/loader.c	82.86% <100.00%> (ø)

... and 55 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[github-actions]

@antekresic, @jnidzwetzki: please review this pull request.

Powered by pull-review

[jnidzwetzki]

@mkindahl

Have you seen the bgw_scheduler_control regression check return an error code when invoked multiple times?

test bgw_scheduler_control        ... ok        50458 ms
test bgw_scheduler_control        ... FAILED (test process exited with exit code 1)     5048 ms

[mkindahl]

@mkindahl

Have you seen the bgw_scheduler_control regression check return an error code when invoked multiple times?

test bgw_scheduler_control        ... ok        50458 ms
test bgw_scheduler_control        ... FAILED (test process exited with exit code 1)     5048 ms

Hmm... I though I'd fixed it.

Managed to fix it locally: let's see if it works now.

[mkindahl]

@mkindahl
Have you seen the bgw_scheduler_control regression check return an error code when invoked multiple times?

test bgw_scheduler_control        ... ok        50458 ms
test bgw_scheduler_control        ... FAILED (test process exited with exit code 1)     5048 ms

Hmm... I though I'd fixed it.

Managed to fix it locally: let's see if it works now.

For some reason. stop_background_workers were not taking effect, so switched to using pg_terminate_backend. It seems stop_background_workers is killing a scheduler for a different database. Not sure why, but that looks like a bug that is unrelated to this pull request.

[timescale-automation]

Automated backport to 2.12.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.12.x
You are currently cherry-picking commit bebd1ab42.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .unreleased/feature_6195
	modified:   src/bgw/scheduler.c
	modified:   src/guc.c
	modified:   src/guc.h
	modified:   src/loader/bgw_launcher.c
	modified:   src/loader/bgw_launcher.h
	modified:   src/loader/loader.c
	new file:   tsl/test/expected/bgw_scheduler_restart.out
	new file:   tsl/test/sql/bgw_scheduler_restart.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   tsl/test/sql/CMakeLists.txt

Job log