fix(meta): fix receiving scheduled command on blocked database

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Discovered in https://buildkite.com/risingwavelabs/main-cron/builds/4257#_

Get the following panic in madsim test.

thread '<unnamed>' panicked at src/meta/src/barrier/checkpoint/control.rs:453:17:
should be at running: should not have command when not running

The assertion is based on the assumption that when a database is in recovery, it should not receive command. We have marked the database as Blocked when it enters recovery, so normal command won't be able to add to the command queue. However, we still allow DropXxx command to be enqueued, and since we didn't prevent polling the queue of the recovering database, we will still receive this command and cause panic in global barrier manager.

In this PR, we will change to avoid polling command queue of any blocked database. When the database is marked ready, we will notify on any change subscriber if the database queue is not empty.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

• My PR needs documentation updates.

Release note

[gru-agent]

This pull request has been modified. If you want me to regenerate unit test for any of the files related, please find the file in "Files Changed" tab and add a comment @gru-agent. (The github "Comment on this file" feature is in the upper right corner of each file in "Files Changed" tab.)

[kwannoel]

Does this mean we cannot drop stream jobs until db recovered?

[wenym1]

Does this mean we cannot drop stream jobs until db recovered?

No. The stream job will have been removed from catalog before we enqueue the command. The command can still be enqueue, and will take affect either by pre_apply_drop_cancel_scheduled during recovery, or by the normal handling logic after db has recovered and resumed running.