Skip to content

Latest commit

 

History

History
92 lines (70 loc) · 3.2 KB

README.md

File metadata and controls

92 lines (70 loc) · 3.2 KB
Raw

Kafka for Drupal 8

Features

  • Drupal 8 Queue API backend. Queue names map to Kafka topics.
    • Standard Drush commands to consume the queue: drush queue-list / drush queue-run
    • Code design documented in DESIGN.md
  • (planned): PSR-3 Log appender. Logger channels map to Kafka topics.
  • Drush commands to
    • check proper operation of the Queue API adapter
      • drush kqpd: producer demo via Queue API
      • drush kqcd: consumer demo via Queue API
      • Existing core drush queue-run command works
    • check proper operation of the high-level consumer, low-level consumer, and producer in relation with a Kafka instance, e.g. check IP / port / credentials:
      • drush kpd: producer demo, extension-level
      • drush khlcd: high-level consumer demo, extension-level
      • drush kllcd: low-level consumer demo, extension-level
    • check available topics drush kafka-topics.

Based on php_rdkafka / librdkafka.

License

Like any Drupal plugin, the module is licensed under the General Public License, version 2.0 or later (SPDX: GPL-2.0+).

Install

  • Install the Drupal 8 site without the module.

  • Get IP, port, and list of topics names for the Kafka 0.10 broker you will be using.

  • Install the librdkafka client library.

  • Install the php-rdkafka PHP extension version 1 or 2 (see "Requirements" for details).

  • Install the module as usual.

  • Configure Drupal settings.php to expose the Kafka queue, either as the default queue backend, or for specific queue names, and point the services to the chosen Kafka brokers.

      $settings['queue_default'] = 'queue.kafka';
  $settings['queue_service_{queue_name}'] = 'queue.kafka';
  $settings['kafka'] = [
    'consumer' => [
      'brokers' => ['127.0.0.1']
    ],
    'producer' => [
      'brokers' => ['127.0.0.1'],
    ],
  ];

Requirements

  • Drupal 8.2.0 or later
  • Apache Kafka 0.10.1.0 or later
  • libradkafka
  • php-rdkafka master 2016-11-01 or more recent
    • 2.0.0 or later for PHP 7.x
    • 1.0.0 or later for PHP 5.6.x
    • 0.9.1 is NOT usable
    • In 2016-11, this means building the extension from sources.
  • PHP 5.6.27 or more recent. Slightly earlier versions might work but have not been evaluated

Limitations

At this point:

  • The module does not provide a "reliable" queue service in the Drupal sense (ReliableQueueInterface). See Kafka documentation about why this is so: the exact reliability guarantees provided depend on the Kafka topic configuration.
  • Using the module during site installation is not supported.

Topics handling:

  • In production instances, automatic topic creation/deletion is usually not enabled, for various reasons, so they need to be created and deleted out of band.
  • On development instances, you may configure automatic topic creation/deletion.