From a689e0bb5541191ea1647e404f65f13191cd0533 Mon Sep 17 00:00:00 2001 From: Jannik Zschiesche Date: Wed, 19 Jun 2024 11:38:31 +0200 Subject: [PATCH 1/5] Add docs for new task manager --- docs/php/symfony/task-manager/index.mdx | 168 ++++++++++++++++++++---- 1 file changed, 145 insertions(+), 23 deletions(-) diff --git a/docs/php/symfony/task-manager/index.mdx b/docs/php/symfony/task-manager/index.mdx index 0580370..e0462b1 100644 --- a/docs/php/symfony/task-manager/index.mdx +++ b/docs/php/symfony/task-manager/index.mdx @@ -7,6 +7,12 @@ import {LinkList} from "../../../../src/components/Link/LinkList"; packagist="https://packagist.org/packages/21torr/task-manager" /> +The task manager is a bundle that builds in top of [Symfony Messenger] to add additional convenience functionality for running, logging and queuing tasks. + +:::note +As this bundle adds quite a lot of functionality on top of Symfonys messenger system, we use the names "task" instead of "message" and "task handler" instead of "message handler" to make clear, that these are the *extended versions*. +::: + ## Installation @@ -16,9 +22,9 @@ First install this bundle: composer require 21torr/task-manager ``` -Then configure your queues in `config/packages/task_manager.yaml`: +Then configure your queues: -```yaml +```yaml title="config/packages/task_manager.yaml" task_manager: queues: # queues sorted by priority. Highest priority at the top @@ -30,55 +36,171 @@ task_manager: While this bundle auto-detects all queue names, you should define them manually in your config, as otherwise the priority between these queues might be wrong. -## Registering new Tasks +## Defining Tasks -The bundle provides a `TaskManager`, that you should use to register tasks to the queue. +You have to define your tasks by creating a new task class extending the `Task` base class: ```php -public function example (TaskManager $taskManager) +use Torr\TaskManager\Task\Task; + +class readonly PimImportTask extends Task { - $someJob = new JobMessage(); - $taskManager->enqueue($someJob); + public function __construct ( + private string $locale, + ) + { + parent::__construct(); + } + + /** + * + */ + public function getMetaData () : TaskMetaData + { + return new TaskMetaData( + label: "PIM Import: {$this->locale}", + group: "PIM", + uniqueTaskId: "pim.import.{$this->locale}", + ); + } } ``` +:::caution +Your task object needs to be serializable. Avoid unnecessary state, e.g. you should always generate the metadata object on-the-fly. +::: + +### Metadata + +The task has to define some metadata, to have a convenient integration into automated tools. + + +#### Label + +An identifying label for the task and it's configuration. You can use some or all parameters here, to improve readability. + + +#### Group + +An optional group label. Used for grouping related tasks when building UI (like the CLI mentioned below). + +#### Unique Task Id + +The unique task id is used for uniquely identifying the type of certain tasks, to [avoid registering them multiple times](#unique-tasks). + + + +### Unique Tasks + +By default, tasks can be registered even if the same task is already added in the queue. For a lot of tasks this is quite wasteful, as running the same task consecutively multiple times will have the exact same result. + +> Example: you register a task to regenerate all image caches. If you change multiple images, you might want to register this task multiple times, once after every image change. By using correct task priority and putting the cache regeneration after all "adding images" tasks, you only need to run the task once. + +Every task can optionally define a unique task id that is used to identify the same tasks: + +```php +return new TaskMetaData( + // ... + uniqueTaskId: "pim.import.{$this->locale}", +); +``` + +When queueing the task, the task manager will first loop through all registered tasks in all queues and look for a task with the same unique task id. If one is found, the new task is not queued. + +:::tip +You can and should use config state of the task object to have proper identifying task ids. +::: + +In the PIM import example, you should use the locale to generate the task id — otherwise the EN and FR pim imports might use the same task id, and you would erroneously assume you don't have to run the task. -## Unique Tasks -The task manager has a feature to only add tasks if there isn't the same task already registered. For that you need to give the task a unique name. -You can provide a name either by passing it to `enqueue()`: +## Queueing Tasks + +You can then queue these tasks using the `TaskManager`: ```php -$taskManager->enqueue($message, "unique.message.key"); +use Torr\TaskManager\Manager\TaskManager; + +public function example (TaskManager $taskManager) +{ + $englishPimImport = new PimImportTask("en"); + $taskManager->enqueue($englishPimImport); +} ``` -But it is recommended to add a name to the task itself: +You can also queue tasks via the CLI, [see below](#registering-tasks). + + +## Task Handlers + +As this bundle builds on top of Symfony messages, you define your task handler as message handler: ```php -class ImportDataTask implements UniqueMessageInterface +use Symfony\Component\Messenger\Attribute\AsMessageHandler; +use Torr\TaskManager\Director\TaskDirector; + +final readonly class PimImportMessageHandler { public function __construct ( - private string $locale, + private TaskDirector $taskDirector, ) {} - /** - * @inheritDoc - */ - public function getJobId () : ?string + #[AsMessageHandler] + public function onPimImport (PimImportMessage $message) : void { - return "import-data.{$this->locale}"; + // start the run + $run = $this->taskDirector->startRun($message); + + // you can get the IO from the run director + $io = $run->getIo(); + + // ... do your thing ... + + // at the end you mark your task as finished and indicate, whether the task was handled successfully + $run->finish(success: true); } } ``` +When your task failed, you have two options: you can either throw an exception, to mark this run as failed (and let the retry functionality kick in) or you can finish handling the task, but mark it as `success: false`. + +You use the task director to start a run for the given task. This will return a run director, that helps you with working through your run: + +- you can get IO to display information on the console +- the IO output will automatically be logged as well +- you can mark the run as success / failure + :::best-practice -To ensure consistent job ids, you should always use the `UniqueMessageInterface` in favor of inline job keys. +Will not explicitly `->finish()`ed tasks will be handled properly, you should always `finish` all your task runs. ::: -## Priorities +## Task Log + +The task and run directors automatically log the output of your tasks, whether they succeeded and some metadata. This includes the task object itself, so it must be serializable. + + +## Running the Message Handler + +Internally the task manager uses [Symfony Messenger], so to run the tasks, you just consume the messages: + +```shell +bin/console messenger:consume queue1 queue2 +``` + + +### Priorities + +The task manager also supports mirroring your configured priorities internally. + +You have to implicitly define the priority of the queues in Symfony by ordering the queues in your CLI call: + +```php +bin/console messenger:consume queue1 queue2 +``` + +The earlier the queue name, the higher the priority, so in this case `queue1` has a higher priority than `queue2`. -The task manager also supports mirroring your priorities internally. -You can implicitly define the priority of the queues in Symfony by ordering the queues in your `bin/console messenger:consume queue1 queue2` call. +[Symfony Messenger]: https://symfony.com/doc/current/messenger.html From 9547d1b9651c23943275c9ceba0186db0f3e6e63 Mon Sep 17 00:00:00 2001 From: Jannik Zschiesche Date: Wed, 19 Jun 2024 17:16:42 +0200 Subject: [PATCH 2/5] Finalize task docs --- docs/php/symfony/task-manager/index.mdx | 62 +++++++++++++++++++++---- 1 file changed, 54 insertions(+), 8 deletions(-) diff --git a/docs/php/symfony/task-manager/index.mdx b/docs/php/symfony/task-manager/index.mdx index e0462b1..29e7956 100644 --- a/docs/php/symfony/task-manager/index.mdx +++ b/docs/php/symfony/task-manager/index.mdx @@ -114,6 +114,46 @@ You can and should use config state of the task object to have proper identifyin In the PIM import example, you should use the locale to generate the task id — otherwise the EN and FR pim imports might use the same task id, and you would erroneously assume you don't have to run the task. +## Registering Tasks + +The bundle provides a way to register your tasks in a global registry, that then can be used to build a UI around automatically queueing tasks. + +You can integrate your existing task classes by registering them in the event: + +```php +use Symfony\Component\EventDispatcher\Attribute\AsEventListener; +use Torr\TaskManager\Event\RegisterTasksEvent; + +class RegisterTasksListener +{ + #[AsEventListener] + public function onRegisterTasks (RegisterTasksEvent $event) : void + { + $event->register(new PimImportTask("de")); + } +} +``` + +By default, your tasks are not auto-detected, so you can have "internal" tasks that are not manually selectable. + +You can view all registered tasks in the debug command: + +```shell +bin/console task-manager:debug +``` + +You can automatically queue registered tasks via the CLI: + +```shell +bin/console task-manager:queue +``` + +You can either interactively select the tasks or directly pass the task IDs to the command: + +```shell +bin/console task-manager:queue task-id-1 task-id-2 +``` + ## Queueing Tasks @@ -172,7 +212,7 @@ You use the task director to start a run for the given task. This will return a - you can mark the run as success / failure :::best-practice -Will not explicitly `->finish()`ed tasks will be handled properly, you should always `finish` all your task runs. +While not explicitly `->finish()`ed tasks will be handled properly, you should always `finish` all your task runs. ::: @@ -180,27 +220,33 @@ Will not explicitly `->finish()`ed tasks will be handled properly, you should al The task and run directors automatically log the output of your tasks, whether they succeeded and some metadata. This includes the task object itself, so it must be serializable. +You can view the task log via the CLI: + +```shell +bin/console task-manager:log +``` + ## Running the Message Handler Internally the task manager uses [Symfony Messenger], so to run the tasks, you just consume the messages: ```shell -bin/console messenger:consume queue1 queue2 +bin/console messenger:consume $(bin/console task-manager:messenger:queue-names) ``` +This command automatically uses the queues in the correct (by priority in descending) order. +Schedules automatically stay at the top of the priority list. -### Priorities -The task manager also supports mirroring your configured priorities internally. +## Debugging -You have to implicitly define the priority of the queues in Symfony by ordering the queues in your CLI call: +You can debug your detected transports, the priority and the registered task via the CLI: -```php -bin/console messenger:consume queue1 queue2 +```shell +bin/console task-manager:debug ``` -The earlier the queue name, the higher the priority, so in this case `queue1` has a higher priority than `queue2`. [Symfony Messenger]: https://symfony.com/doc/current/messenger.html From a49c291d83618cfddc16ce81cad227ecee5b8378 Mon Sep 17 00:00:00 2001 From: Jannik Zschiesche Date: Wed, 19 Jun 2024 17:19:50 +0200 Subject: [PATCH 3/5] Add note about migrations --- docs/php/symfony/task-manager/index.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/php/symfony/task-manager/index.mdx b/docs/php/symfony/task-manager/index.mdx index 29e7956..3087c51 100644 --- a/docs/php/symfony/task-manager/index.mdx +++ b/docs/php/symfony/task-manager/index.mdx @@ -226,6 +226,10 @@ You can view the task log via the CLI: bin/console task-manager:log ``` +:::note +The bundle only contains the entities, no migrations. You might need to generate the migrations yourself, if you use the doctrine migrations bundle. +::: + ## Running the Message Handler From f6e39ea841149a76e5c9e58b3c0c7a3fd6090469 Mon Sep 17 00:00:00 2001 From: Jannik Zschiesche Date: Wed, 19 Jun 2024 17:27:24 +0200 Subject: [PATCH 4/5] Add note about stamps --- docs/php/symfony/task-manager/index.mdx | 42 +++++++++++++++++-------- 1 file changed, 29 insertions(+), 13 deletions(-) diff --git a/docs/php/symfony/task-manager/index.mdx b/docs/php/symfony/task-manager/index.mdx index 3087c51..2a375be 100644 --- a/docs/php/symfony/task-manager/index.mdx +++ b/docs/php/symfony/task-manager/index.mdx @@ -142,18 +142,6 @@ You can view all registered tasks in the debug command: bin/console task-manager:debug ``` -You can automatically queue registered tasks via the CLI: - -```shell -bin/console task-manager:queue -``` - -You can either interactively select the tasks or directly pass the task IDs to the command: - -```shell -bin/console task-manager:queue task-id-1 task-id-2 -``` - ## Queueing Tasks @@ -169,7 +157,35 @@ public function example (TaskManager $taskManager) } ``` -You can also queue tasks via the CLI, [see below](#registering-tasks). +If you need to pass stamps, you can do that in the second parameter: + +```php +use Symfony\Component\Messenger\Stamp\DispatchAfterCurrentBusStamp; + +$taskManager->enqueue(new PimImportTask("en"), [ + new DispatchAfterCurrentBusStamp(), +]); +```` + + +:::best-practice +When queueing tasks inside a task handler, you should always add the `DispatchAfterCurrentBusStamp`. +::: + + +You can also queue tasks via the CLI: + +You can automatically queue registered tasks via the CLI: + +```shell +bin/console task-manager:queue +``` + +You can either interactively select the tasks or directly pass the task IDs to the command: + +```shell +bin/console task-manager:queue task-id-1 task-id-2 +``` ## Task Handlers From a5726cad82be943c8dac086449b8323819807b88 Mon Sep 17 00:00:00 2001 From: Jannik Date: Thu, 20 Jun 2024 12:59:23 +0200 Subject: [PATCH 5/5] Add oxford comma Co-authored-by: Kai Eichinger --- docs/php/symfony/task-manager/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/php/symfony/task-manager/index.mdx b/docs/php/symfony/task-manager/index.mdx index 2a375be..b7b11ea 100644 --- a/docs/php/symfony/task-manager/index.mdx +++ b/docs/php/symfony/task-manager/index.mdx @@ -10,7 +10,7 @@ import {LinkList} from "../../../../src/components/Link/LinkList"; The task manager is a bundle that builds in top of [Symfony Messenger] to add additional convenience functionality for running, logging and queuing tasks. :::note -As this bundle adds quite a lot of functionality on top of Symfonys messenger system, we use the names "task" instead of "message" and "task handler" instead of "message handler" to make clear, that these are the *extended versions*. +As this bundle adds quite a lot of functionality on top of Symfonys messenger system, we use the names "task" instead of "message", and "task handler" instead of "message handler" to make clear, that these are the *extended versions*. :::