Recovery support multi hosts and ports of external kafka cluster

[MemberIT]

Fix issue #1584

[MemberIT]

Hi @Mokto
Sorry, but i can't understand how to fix this

[Mokto]

Doesn't matter, no worries.

[patsevanton]

@MemberIT Do you have ready terraform kafka code for yandex cloud ? If not, okay.

[nadecancode]

I think this looks right.

Is it possible to move the configuration somewhere else? Moving kafka hosts & ports to cluster will break the existing configurations.

[MemberIT]

@MemberIT Do you have ready terraform kafka code for yandex cloud ? If not, okay.

@patsevanton, I don't have any ready-made terraform code to quickly test this.

[MemberIT]

I think this looks right.

Is it possible to move the configuration somewhere else? Moving kafka hosts & ports to cluster will break the existing configurations.

Actually, until version sentry-v25.19.0, externalKafka could be either a hash or an array, which is very strange and prevents the introduction of additional values (in the future, we can add support for SSL authentication for kafka). While preparing the kafka sasl auth feature for sentry-v25.19.0, I missed that externalKafka can be an array because it was documented only in the README.md.

I could restore backward compatibility in this PR to how it was before sentry-v25.19.0 by combining variables, which seems logical:

externalKafka:
  compression:
    type:  # 'gzip', 'snappy', 'lz4', 'zstd'
  message:
    max:
      bytes: 50000000
  socket:
    timeout:
      ms: 1000

and

sentry:
  kafka:
    message:
      max:
        bytes: 50000000
    compression:
      type:  # 'gzip', 'snappy', 'lz4', 'zstd'
    socket:
      timeout:
        ms: 1000

But these variables need to be separated:

externalKafka:
  sasl:
    mechanism: None  # PLAIN, SCRAM-256, SCRAM-512
    username: None
    password: None
  security:
    protocol: plaintext  # 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL', and 'SSL'

like:

externalKafkaSASL:
  mechanism: None  # PLAIN, SCRAM-256, SCRAM-512
  username: None
  password: None
externalKafkaSecurity:
  protocol: plaintext  # 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL', and 'SSL'

Is it worth making these changes to guarantee backward compatibility?

It seems normal to me that the structure of values passed into the chart can change from version to version.

I am open to any suggestions on the PR.

[patsevanton]

I'm leaning towards

externalKafka:
  sasl:
    mechanism: None  # PLAIN, SCRAM-256, SCRAM-512
    username: None
    password: None
  security:
    protocol: plaintext  # 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL', and 'SSL'

[patsevanton]

@MemberIT Hello! i created k8s and kafka by repository https://github.com/patsevanton/sentry-with-external-kafka

and set values.yaml

externalKafka:
  cluster:
    - host: "rc1a-ah3jsas127q8ng11.mdb.yandexcloud.net"
      port: 9092
    - host: "rc1b-c4pv8ou9atkl61b4.mdb.yandexcloud.net"
      port: 9092
    - host: "rc1d-h2gh8089paovc814.mdb.yandexcloud.net"
      port: 9092
  compression:
    type:  # 'gzip', 'snappy', 'lz4', 'zstd'
  message:
    max:
      bytes: 50000000
  sasl:
    mechanism: SCRAM-SHA-512
    username: sentry
    password: your_password_here
  security:
    protocol: SASL_PLAINTEXT
  socket:
    timeout:
      ms: 1000

Work!
and
values.yaml

externalKafka:
  host: "rc1a-ah3jsas127q8ng11.mdb.yandexcloud.net"
  port: 9092
  compression:
    type:  # 'gzip', 'snappy', 'lz4', 'zstd'
  message:
    max:
      bytes: 50000000
  sasl:
    mechanism: SCRAM-SHA-512
    username: sentry
    password: your_password_here
  security:
    protocol: SASL_PLAINTEXT
  socket:
    timeout:
      ms: 1000

work!

FYI @Mokto

[Lattenike]

This issue isn't add to tag 26.4.0? When will you add this issue to tag?

[Lattenike]

Now my sentry version is 26.3.0, I can't use multi hosts and ports of external kafka ,and I'm meeting error "snuba.migrations.errors.MigrationInProgress: events_analytics_platform: 0009_drop_index_attribute_key_buckets_1_19", if this issue affect sentry-snuba-migrate.

[Lattenike]

Now I just do "kubectl delete job sentry-snuba-migrate" to skip snuba-migrate

[Lattenike]

Just now , my UI service and other service is still working healthly , if snuba-migrate is not prerequisite.

[nadecancode]

Now my sentry version is 26.3.0, I can't use multi hosts and ports of external kafka ,and I'm meeting error "snuba.migrations.errors.MigrationInProgress: events_analytics_platform: 0009_drop_index_attribute_key_buckets_1_19", if this issue affect sentry-snuba-migrate.

It got stuck but I don't think it's related to this PR. More likely your clickhouse had some issues and I would retry this migration by creating a snuba debug container, exec into it and reverse this migration.

[patsevanton]

@Mokto is it possible to merge this pull request ?

[iscultas]

@MemberIT @Mokto is it possible to backport these changes into v25? Because it is impossible to upgrade to v26 when config is broken in v25.19.0

[Mokto]

Sorry but that would be super annoying to do with the current build system.

Why don't you do it manually in your project?

[iscultas]

Sorry but that would be super annoying to do with the current build system.

Why don't you do it manually in your project?

Already did, but it is a nasty hack regarding that the chart is set up in Terraform as a part of a module.

[MemberIT]

@MemberIT @Mokto is it possible to backport these changes into v25? Because it is impossible to upgrade to v26 when config is broken in v25.19.0

@Lattenike, I reviewed the changes in helm chart from sentry-v25.18.0 to sentry-v26.5.0, and it seems that it might be possible to use the chart from sentry-v26.5.0 specifying Sentry version as 24.8.0 in helm chart to upgrade Sentry to 24.8.0, and then use it as is to upgrade Sentry to 24.9.0

[Mokto]

That's a good idea.

[MemberIT]

That's a good idea.

@Mokto, I added a note in CHANGELOG.md. If this workaround proves effective, we can merge this PR so that the workaround is documented

[MemberIT]

@MemberIT @Mokto today tried to upgrade to 26.5.0 and faced with Error: execution error at (sentry/templates/sentry/subscription-consumer/transactions/deployment-sentry-subscription-consumer-transactions.yaml:33:33): A valid .Values.externalKafka.host is required Probably required condition must be remove here https://github.com/sentry-kubernetes/charts/blame/ff096da0b7cc94a35e2891550426c8d61cf700d6/charts/sentry/templates/_helper.tpl#L422

@iscultas, You need to change externalKafka.host to externalKafka.cluster
README.md
Before (for sentry-v25.18.0):

externalKafka:
  - host: "233.5.100.28"
    port: 9092
  - host: "kafka-confluent-2"
    port: 9093
  - host: "kafka-confluent-3"
    port: 9094

After (for sentry-v26.5.0):

externalKafka:
  cluster:
    - host: "233.5.100.28"
      port: 9092
    - host: "kafka-confluent-2"
      port: 9093
    - host: "kafka-confluent-3"
      port: 9094

[MemberIT]

@iscultas
It seems to me, this error might occur if there is a typo in the word cluster or if the cluster key was lost due to indentation issues in your custom values.yaml.