Skip to content

Commit

Permalink
Allow specifying an archival policy for buckets
Browse files Browse the repository at this point in the history
Only enables this for openscapes, which is using it as described in
https://github.com/2i2c-org/infrastructure/issues/3562.
  • Loading branch information
yuvipanda committed Jan 2, 2024
1 parent 3fae917 commit e9da67b
Show file tree
Hide file tree
Showing 4 changed files with 37 additions and 4 deletions.
9 changes: 9 additions & 0 deletions docs/howto/features/buckets.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ on why users want this!
},
"bucket2": {
"delete_after": null
},
"bucket3": {
"archival_storageclass_after": 3
}
}
```
Expand All @@ -28,6 +31,12 @@ on why users want this!
very helpful for 'scratch' buckets that are temporary. Set to
`null` to prevent this cleaning up process from happening, e.g., if users want a persistent bucket.

`archival_storageclass_after` (available only for AWS currently) transitions objects
created in this bucket to a cheaper, slower archival class after the number of days
specified in this variable. This is helpful for archiving user home directories or similar
use cases, where data needs to be kept for a long time but rarely accessed. This should
not be used for frequently accessed or publicly accessible data.

2. Enable access to these buckets from the hub or make them publicly accessible from outside
by [editing `hub_cloud_permissions`](howto:features:cloud-access:access-perms)
in the same `.tfvars` file. Follow all the steps listed there - this
Expand Down
12 changes: 12 additions & 0 deletions terraform/aws/buckets.tf
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,18 @@ resource "aws_s3_bucket_lifecycle_configuration" "user_bucket_expiry" {
days = each.value.delete_after
}
}

rule {
id = "archival-storageclass"
status = each.value.delete_after != null ? "Enabled" : "Disabled"

transition {
# Transition this to much cheaper object storage after a few days
days = each.value.archival_storageclass_after
# Glacier Instant is fast enough while also being pretty cheap
storage_class = "GLACIER_IR"
}
}
}

locals {
Expand Down
3 changes: 3 additions & 0 deletions terraform/aws/projects/openscapes.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ user_buckets = {
"scratch" : {
"delete_after" : 7
},
"prod-homedirs-archive" : {
"archival_storageclass_after" : 3
}
}


Expand Down
17 changes: 13 additions & 4 deletions terraform/aws/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,26 @@ variable "cluster_nodes_location" {
}

variable "user_buckets" {
type = map(object({ delete_after : number }))
type = map(
object({
delete_after : optional(number, null),
archival_storageclass_after : optional(number, null)
})
)
default = {}
description = <<-EOT
S3 Buckets to be created.
The key for each entry will be prefixed with {var.prefix}- to form
the name of the bucket.
The value is a map, with 'delete_after' the only accepted key in that
map - it lists the number of days after which any content in the
bucket will be deleted. Set to null to not delete data.
The value is a map, with the following accepted keys:
1. `delete_after` - number of days after *creation* an object in this
bucket will be automatically deleted. Set to null to not delete data.
2. `archival_storageclass_after` - number of days after *creation* an
object in this bucket will be automatically transitioned to a cheaper,
slower storageclass for cost savings. Set to null to not transition.
EOT
}

Expand Down

0 comments on commit e9da67b

Please sign in to comment.