Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable --expire-all-object-versions #4840

Merged
merged 6 commits into from
Feb 20, 2024

Conversation

shtripat
Copy link
Contributor

@shtripat shtripat commented Feb 2, 2024

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

This PR introduces a new flag --expire-all-object-versions to the command mc ilm rule add to enabled option to create ILM rule to remove all versions of expired objects.

Motivation and Context

Allow setting ILM rule for removing all versions of expired objects from command line.

How to test this PR?

Follow below steps to verify the changes

Step-1: Set your system date to few days older (~3days)

Step-2: Create a MinIO instance, create a bucket and enable versioning for the bucket

Step-3: Set the bucket public using mc anonymous set public ALIAS/BUCKET

Step-4: Set the MinIO scanner speed to fastest using mc admin config set ALIAS scanner speed=fastest

Step-5: Load say 100 objects to the bucket using curl command with older mtime value as below

curl -X PUT http://127.0.0.1:9000/BUCKET/OBJNAME -d @PATH-TO-OBJ-SRC-FILE -H "X-Minio-Source-Mtime: 2023-03-12T16:19:03.115209866-08:00"

Step-6: Now delete these objects using mc rm which would create DEL markers for all the objects. Eventually two versions for all objects

Step-7: Set the date back to current for the system

Step-8: Add bewlo ILM rules for bucket
mc ilm rule add ALIAS/BUCKET --expire-all-object-versions --expire-days 1

Step-9: After few seconds all versions of all objects should be removed and mc ls ALIAS/BUCKET --versions should not return any values.

Step:10: Also verify prometheus metrics values minio_s3_requests_total{api="putobject",server="127.0.0.1:9000"} and minio_node_ilm_action_count_delete_all_versions_action{server="127.0.0.1:9000"}. They should return 100 and 200 respectively as count values.

Note: needs: minio/minio-go#1927 and minio/minio#18954

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

@shtripat shtripat marked this pull request as draft February 5, 2024 12:26
@shtripat shtripat marked this pull request as ready for review February 6, 2024 05:06
go.mod Outdated Show resolved Hide resolved
@shtripat shtripat force-pushed the expired-object-all-versions branch from 03c9dfe to 4bc037c Compare February 15, 2024 04:26
Copy link
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few questions

cmd/ilm-rule-add.go Outdated Show resolved Hide resolved
cmd/ilm/options.go Outdated Show resolved Hide resolved
cmd/ilm/options.go Outdated Show resolved Hide resolved
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat shtripat force-pushed the expired-object-all-versions branch from 4bc037c to c13ab49 Compare February 20, 2024 04:49
@shtripat shtripat changed the title Enable --expired-object-all-versions Enable --expire-all-object-versions Feb 20, 2024
@shtripat shtripat requested a review from vadmeste February 20, 2024 04:52
Copy link
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@prakashsvmx prakashsvmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested . LGTM 👍

@harshavardhana harshavardhana merged commit 8156961 into minio:master Feb 20, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants