Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

registering a file into a tiered resource should provide tiered behavior for that data object #280

Open
trel opened this issue Dec 3, 2024 · 1 comment

Comments

@trel
Copy link
Member

trel commented Dec 3, 2024

With resources t0, t1, and t2 configured in example_group_1...

Registering a file in-place, which is already physically on the t2 volume... should result in the data object having enough storage_tiering metadata for that data object to be restage-able... just as if it had been placed on that tier by the storage_tiering framework itself.

In practice... this means it needs both irods::access_time information AND irods::storage_tiering::group information in the metadata.

Currently, a newly registered data object has only the irods::access_time AVU and does not restage when iget is issued.

$ imeta ls -R t0
AVUs defined for resource t0:
attribute: irods::storage_tiering::group
value: example_group_1
units: 0
----
attribute: irods::storage_tiering::time
value: 30
units: 
$ imeta ls -R t1
AVUs defined for resource t1:
attribute: irods::storage_tiering::group
value: example_group_1
units: 1
----
attribute: irods::storage_tiering::time
value: 45
units: 
$ imeta ls -R t2
AVUs defined for resource t2:
attribute: irods::storage_tiering::group
value: example_group_1
units: 2

$ ilsresc -l t2 | grep vault
vault: /tmp/t2vault

$ echo "hello" > /tmp/t2vault/home/rods/hello.txt
$ ireg -R t2 /tmp/t2vault/home/rods/hello.txt /tempZone/home/rods/hello.txt
$ imeta ls -d hello.txt
AVUs defined for dataObj /tempZone/home/rods/hello.txt:
attribute: irods::access_time
value: 1733264876
units: 

$ iget hello.txt -
hello
$ ils -L hello.txt
  rods              0 t2            6 2024-12-03.17:27 & hello.txt
        generic    /tmp/t2vault/home/rods/hello.txt
$ iqstat
No delayed rules pending for user rods

With the manual addition of the group and replica AVU, then the data object begins to stage/restage as expected.

$ imeta add -d hello.txt irods::storage_tiering::group example_group_1 0
$ iget hello.txt -
hello
$ iqstat
id     name
23591 {"delay_conditions":"<INST_NAME>irods_rule_engine_plugin-unified_storage_tiering-instance</INST_NAME><EF>60s REPEAT UNTIL SUCCESS OR 5 TIMES</EF><PLUSET>22s</PLUSET>","destination-resource":"t0","group-name":"example_group_1","md5":"445a76f8acd6aad970a568e6b08208fe","object-path":"/tempZone/home/rods/hello.txt","preserve-replicas":false,"rule-engine-instance-name":"irods_rule_engine_plugin-unified_storage_tiering-instance","rule-engine-operation":"irods_policy_data_movement","source-replica-number":"0","source-resource":"t2","user-name":"rods","user-zone":"tempZone","verification-type":"catalog"} 

and then it has successfully been migrated to t0...

$ ils -L hello.txt
  rods              1 t0            6 2024-12-03.17:31 & hello.txt
        generic    /tmp/t0vault/home/rods/hello.txt
$ imeta ls -d hello.txt
AVUs defined for dataObj /tempZone/home/rods/hello.txt:
attribute: irods::access_time
value: 1733265082
units: 
----
attribute: irods::storage_tiering::group
value: example_group_1
units: 1
@trel
Copy link
Member Author

trel commented Dec 3, 2024

The apply_access_time policy is firing on the registration PEP, but we also need to fire the apply_tier_group_metadata_to_object policy on registration.

This might require a little reorganization of the existing code for modularity, but the policies and AVU setting already exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant