[Filebeat][input/aws-s3]: backup_to_bucket_arn and delete_after_backup do not work

[constanca-m]

backup_to_bucket_arn and delete_after_backup options do not work. The filebeat keeps processing files from the bucket_arn over and over again, and they are never deleted and placed in delete_after_backup bucket.

This was tested with version 8.15.3.

Example

Filebeat configuration:

    filebeat.inputs:
      - type: aws-s3
        enabled: true
        session_token: x
        access_key_id: x
        secret_access_key: x
        bucket_arn: arn:aws:s3:::constanca-test-s3-backup
        backup_to_bucket_arn: arn:aws:s3:::constanca-test-s3-backup-delete
        delete_after_backup: true
        number_of_workers: 5
        bucket_list_interval: 20s
        expand_event_list_from_field: Records

Logs that prove filebeat finds the files:

{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.564Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object","file.name":"awss3/s3_objects.go","file.line":129},"message":"Begin S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.673Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"export.ndjson","elapsed_time_ns":108787233,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-11-26T08:44:57.775Z","log.logger":"input.aws-s3.s3","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/awss3.(*s3ObjectProcessor).ProcessS3Object.func1","file.name":"awss3/s3_objects.go","file.line":137},"message":"End S3 object processing.","service.name":"filebeat","id":"6F32504842815259","bucket_arn":"constanca-test-s3-backup","object_key":"gopher.png","elapsed_time_ns":210936613,"ecs.version":"1.6.0"}

However, the file remains in bucket_arn:

And backup_to_bucket_arn remains empty:

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[kaiyan-sheng]

The fix is added back in PR: #41071
But seems like the fix has been removed by a later on PR: #40699

We should add the feature back and add tests to make sure the regression does not happen again.