GCP Tiered Storage org.jclouds.rest.ResourceNotFoundException - Pulsar 2.6.1

[devinbost]

I have a heartbeat topic that receives messages on the topic persistent://public/default/canary, and I configured retention on this topic to test tiered storage in GCP. When I manually trigger offloading from the topic, I get the error java.util.concurrent.CompletionException: org.jclouds.rest.ResourceNotFoundException: Not Found, as shown below:

bin/pulsar-admin topics offload --size-threshold 10M persistent://public/default/canary

Offload triggered for persistent://public/default/canary for messages before 873151:0:-1

bin/pulsar-admin topics offload-status -w persistent://public/default/canary

Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.rest.ResourceNotFoundException: Not Found

On the broker and bookie containers, I have the GCP JSON key file located at conf/auth/tiered-storage.key

In my broker.conf file, I have the GCS attributes configured as follows:

managedLedgerOffloadDriver=google-cloud-storage
gcsManagedLedgerOffloadRegion=us-west-3
gcsManagedLedgerOffloadBucket=gs://my-bucket-name
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadServiceAccountKeyFile=/pulsar/conf/auth/tiered-storage.key

The tiered-storage.key file looks as follows:

{
  "type": "service_account",
  "project_id": "my-project-id",
  "private_key_id": "c8myid12345",
  "private_key": "-----BEGIN PRIVATE KEY-----\nABCDEFG...XYZ=\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/example-cert-url%40identity-example.iam.gserviceaccount.com"
}

It seems that the key file path is incorrect. However, when I tried altering the key file to contain invalid JSON, I got an error message that the JSON was invalid when I triggered a manual offload of the topic, which implies that Pulsar was able to access the key file.

[devinbost]

FYI @david-streamlio

[devinbost]

Could the bucket path be incorrect? Perhaps it doesn't like the prefix.

[devinbost]

I removed the "gs://" prefix from the bucket name in broker.conf, and the issue was resolved.

[codelipenghui]

@devinbost Seems we should clarify it in the document, are you interested in fix it?

[tisonkun]

cc @momo-jun This can indicate a doc issue.

[momo-jun]

Sorry for my late reply. As discussed with @RobertIndie, we both think the docs are okay by clarifying the attribute (bucket name) and providing an example (pulsar-topic-offload). It would be too much if we add patches to not use the prefix.