diff --git a/README.md b/README.md index 93aeecd..17382fe 100644 --- a/README.md +++ b/README.md @@ -34,12 +34,12 @@ In iRODS terminology, the `attribute` is defined by a **plugin_specific_configur For example: ``` -imeta add -R fast_resc irods::storage_tiering::group example_group 0 -imeta add -R medium_resc irods::storage_tiering::group example_group 1 -imeta add -R slow_resc irods::storage_tiering::group example_group 2 +imeta add -R fast_resc irods::storage_tiering::group example_group_1 0 +imeta add -R medium_resc irods::storage_tiering::group example_group_1 1 +imeta add -R slow_resc irods::storage_tiering::group example_group_1 2 ``` -This example defines three tiers of the group `example_group` where data will flow from tier 0 to tier 2 as it ages. In this example `fast_resc` is a single resource, but it could have been set to `fast_tier_root_resc` and represent the root of a resource hierarchy consisting of many resources. +This example defines three tiers of the group `example_group_1` where data will flow from tier 0 to tier 2 as it ages. In this example `fast_resc` is a single resource, but it could have been set to the root of a resource hierarchy consisting of many resources. ### Setting Tiering Policy @@ -112,7 +112,7 @@ For a default installation the following values are used: "minimum_restage_tier" : "irods::storage_tiering::minimum_restage_tier", "preserve_replicas" : "irods::storage_tiering::preserve_replicas", "object_limit" : "irods::storage_tiering::object_limit", - "default_data_movement_parameters" : "60s DOUBLE UNTIL SUCCESS OR 5 TIMES", + "default_data_movement_parameters" : "60s REPEAT UNTIL SUCCESS OR 5 TIMES", "minumum_delay_time" : "irods::storage_tiering::minimum_delay_time_in_seconds", "maximum_delay_time" : "irods::storage_tiering::maximum_delay_time_in_seconds", "time_check_string" : "TIME_CHECK_STRING", @@ -162,6 +162,8 @@ A tier within a tier group may identify data objects which are in violation by a Data objects which have been labeled via particular metadata, or within a specific collection, owned by a particular user, or belonging to a particular project may be identified through a custom query. The default attribute **irods::storage_tiering::query** is used to hold this custom query. To configure the custom query, attach the query to the root resource of the tier within the tier group. This query will be used in place of the default time-based query for that tier. For efficiency this example query checks for the existence in the root resource's list of leaves by resource ID. Please note that any custom query must return DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM in that order as it is a convention of this rule engine plugin. +**Checking for resources in violating queries is required to prevent erroneous data migrations for replicas on other resources which may represent other tiers in the storage tiering group.** This can be done in the manner shown below (`DATA_RESC_ID in ('10068', '10069')`) or via resource hierarchy (e.g. `DATA_RESC_HIER like 'root_resc;%`), but the query must filter on resources to correctly identify violating objects. + ``` imeta set -R fast_resc irods::storage_tiering::query "SELECT DATA_NAME, COLL_NAME, USER_NAME, USER_ZONE, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < 'TIME_CHECK_STRING' AND DATA_RESC_ID IN ('10068', '10069')" ``` @@ -216,3 +218,49 @@ In order to log the transfer of data objects from one tier to the next, set `dat }, ``` +## Limitations + +There are a few known limitations to the storage tiering plugin which should be noted explicitly for understanding different failure modes which users may experience. + +### A data object should only have replicas in one tiering group + +Any given data object should only have replicas in a single tiering group. Stated negatively, a data object should NOT have replicas in multiple tiering groups. The tiering group for a data object is tracked by an AVU that looks like this: +``` +attribute: irods::storage_tiering::group +value: example_group_1 +units: 1 +``` + +The value is the tiering group to which this object belongs. There should only be one AVU with the attribute `irods::storage_tiering::group` associated with it. Here is an example of the AVUs an object with multiple tiering groups would have: +``` +attribute: irods::storage_tiering::group +value: example_group_1 +units: 1 +--- +attribute: irods::storage_tiering::group +value: example_group_2 +units: 2 +``` + +Notice the different values indicating that the object has replicas in two different tiering groups. + +Support for multi-group data objects may become available in the future, but it should be avoided at this time. + +### Managing replicas and metadata outside of storage tiering should be done with caution + +Replicating, trimming, and manipulating `irods::storage_tiering`-namespaced metadata on data objects which are under management in a tiering group using the storage tiering plugin should be done only when necessary. If any metadata is not in the expected state or replicas are not found in their expected resources, unexpected behavior can and will occur as it relates to storage tiering operations. + +### A resource should only belong to one tier in a given group + +Resources may belong to multiple tiering groups (e.g. a common archive). It is not recommended for a resource to be tagged with multiple tiers for a given group as the storage tiering plugin assumes that a resource only represents a single tier in any given group. In other words, a resource should not have multiple AVUs like this: +``` +attribute: irods::storage_tiering::group +value: example_group_1 +units: 0 +--- +attribute: irods::storage_tiering::group +value: example_group_1 +units: 1 +``` + +The above AVUs indicate that the resource represents tier 0 AND tier 1 in example_group_1. This should not be done. diff --git a/libirods_rule_engine_plugin-unified_storage_tiering.cpp b/libirods_rule_engine_plugin-unified_storage_tiering.cpp index 0b1d8a2..c60958a 100644 --- a/libirods_rule_engine_plugin-unified_storage_tiering.cpp +++ b/libirods_rule_engine_plugin-unified_storage_tiering.cpp @@ -20,6 +20,9 @@ #include #include "exec_as_user.hpp" +#include +#include + #undef LIST // =-=-=-=-=-=-=- @@ -84,12 +87,35 @@ namespace { return std::make_tuple(l1_idx, resource_name); } // get_index_and_resource + auto resource_hierarchy_has_good_replica(RcComm* _comm, + const std::string& _object_path, + const std::string& _root_resource) -> bool + { + namespace fs = irods::experimental::filesystem; + + const auto object_path = fs::path{_object_path}; + + const auto query_string = fmt::format("select DATA_ID where DATA_NAME = '{0}' and COLL_NAME = '{1}' and " + "DATA_RESC_HIER like '{2};%' || = '{2}' and DATA_REPL_STATUS = '1'", + object_path.object_name().c_str(), + object_path.parent_path().c_str(), + _root_resource); + + const auto query = irods::query{_comm, query_string}; + + return query.size() > 0; + } // resource_hierarchy_has_good_replica + void replicate_object_to_resource( rcComm_t* _comm, const std::string& _instance_name, const std::string& _source_resource, const std::string& _destination_resource, const std::string& _object_path) { + // If the destination resource has a good replica of the data object, skip replication. + if (resource_hierarchy_has_good_replica(_comm, _object_path, _destination_resource)) { + return; + } dataObjInp_t data_obj_inp{}; rstrcpy(data_obj_inp.objPath, _object_path.c_str(), MAX_NAME_LEN); diff --git a/packaging/test_plugin_unified_storage_tiering.py b/packaging/test_plugin_unified_storage_tiering.py index 327b954..86b35c5 100644 --- a/packaging/test_plugin_unified_storage_tiering.py +++ b/packaging/test_plugin_unified_storage_tiering.py @@ -213,6 +213,16 @@ def invoke_storage_tiering_rule(): with session.make_session_for_existing_admin() as admin_session: admin_session.assert_icommand(['irule', '-r', rep_instance, '-F', rule_file_path]) +def get_tracked_replica(session, logical_path, group_attribute_name=None): + coll_name = os.path.dirname(logical_path) + data_name = os.path.basename(logical_path) + + tracked_replica_query = \ + "select META_DATA_ATTR_UNITS where DATA_NAME = '{}' and COLL_NAME = '{}' and META_DATA_ATTR_NAME = '{}'".format( + data_name, coll_name, group_attribute_name or 'irods::storage_tiering::group') + + return session.run_icommand(['iquest', '%s', tracked_replica_query])[0].strip() + class TestStorageTieringPlugin(ResourceBase, unittest.TestCase): def setUp(self): @@ -714,6 +724,7 @@ def test_put_and_get(self): finally: admin_session.assert_icommand('irm -f ' + filename) + class TestStorageTieringPluginPreserveReplica(ResourceBase, unittest.TestCase): def setUp(self): super(TestStorageTieringPluginPreserveReplica, self).setUp() @@ -781,42 +792,54 @@ def test_preserve_replicas_works_with_restage_when_replicas_exist_in_multiple_ti IrodsController().restart(test_mode=True) with session.make_session_for_existing_admin() as admin_session: filename = 'test_put_file' + logical_path = '/'.join([admin_session.home_collection, filename]) try: - # make sure replicas stored on tier 2 are preserved + # Replicas should be preserved on ufs1 and ufs2. ufs1 should not be the minimum restage tier to + # ensure that a restage to a lower tier can occur. + admin_session.assert_icommand('imeta rm -R ufs1 irods::storage_tiering::minimum_restage_tier true') + admin_session.assert_icommand('imeta rm -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta add -R ufs1 irods::storage_tiering::preserve_replicas true') admin_session.assert_icommand('imeta add -R ufs2 irods::storage_tiering::preserve_replicas true') - # create test file and upload it + # Create test file and upload it. lib.create_local_testfile(filename) admin_session.assert_icommand('iput -R ufs0 ' + filename) admin_session.assert_icommand('ils -L ', 'STDOUT_SINGLELINE', 'rods') - # stage to tier 1, look for both replicas + # Stage to tier 1 and ensure that the replica on ufs0 was trimmed. sleep(6) invoke_storage_tiering_rule() admin_session.assert_icommand('iqstat', 'STDOUT_SINGLELINE', 'irods_policy_storage_tiering') - delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '0 ufs0') delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '1 ufs1') + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + # Stage to tier 2, look for replica in tier 1 and tier 2. sleep(15) - # stage to tier 2, look for replica in tier 0 and tier 2 invoke_storage_tiering_rule() admin_session.assert_icommand('iqstat', 'STDOUT_SINGLELINE', 'irods_policy_storage_tiering') - delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '0 ufs0') + delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '1 ufs1') delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '2 ufs2') - # test restage to tier 1 using replica from tier 0, look for replica in all tiers afterwards - admin_session.assert_icommand('iget -R ufs0 ' + filename + ' - ', 'STDOUT_SINGLELINE', 'TESTFILE') + # Test restage to tier 0 using replica from tier 1 to ensure that a replica that is not the + # "tracked" replica (the tracked replica is in ufs2) gets restaged. Look for replica in all tiers + # afterwards. + admin_session.assert_icommand('iget -R ufs1 ' + filename + ' - ', 'STDOUT_SINGLELINE', 'TESTFILE') sleep(15) - delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '0 ufs0') - delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '3 ufs1') + delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '3 ufs0') + delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '1 ufs1') delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', '2 ufs2') finally: + admin_session.assert_icommand('imeta rm -R ufs1 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta rm -R ufs2 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta add -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta add -R ufs1 irods::storage_tiering::minimum_restage_tier true') + admin_session.assert_icommand('irm -f ' + filename) def test_preserve_replicas_works_with_restage_when_replica_only_exists_in_last_tier__issue_233(self): @@ -860,6 +883,223 @@ def test_preserve_replicas_works_with_restage_when_replica_only_exists_in_last_t finally: admin_session.assert_icommand('irm -f ' + filename) + def test_tiering_out_with_existing_replica_in_higher_tier__issue_235(self): + with storage_tiering_configured_with_log(): + IrodsController().restart(test_mode=True) + with session.make_session_for_existing_admin() as admin_session: + filename = 'test_put_file' + logical_path = '/'.join([admin_session.home_collection, filename]) + + try: + # Preserve replicas on tier 1 and tier 2, not tier 0. Make tier 1 NOT the minimum restage tier. + admin_session.assert_icommand('imeta rm -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta add -R ufs1 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta rm -R ufs1 irods::storage_tiering::minimum_restage_tier true') + + lib.create_local_testfile(filename) + admin_session.assert_icommand('iput -R ufs0 ' + filename) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Tier out replica from ufs0 to ufs1. + sleep(6) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs1 and not preserved on ufs0. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + + # Ensure that the "tracked" replica updates to replica 1, which is the tiered-out replica on ufs1. + self.assertEqual('1', get_tracked_replica(admin_session, logical_path)) + + # Tier out replica from ufs1 to ufs2. + sleep(15) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs2 and preserved on ufs1, and no replica exists on + # ufs0. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Ensure that the "tracked" replica updates to replica 2, which is the tiered-out replica on ufs2. + self.assertEqual('2', get_tracked_replica(admin_session, logical_path)) + + # Get the data object in order to trigger a restage, targeting replica 2 to ensure it is the replica + # that gets restaged. + admin_session.assert_icommand(['iget', '-n', '2', filename, '-'], 'STDOUT', 'TESTFILE') + + # Ensure that the restage has been scheduled. + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_data_movement') + + # Ensure that the replica is restaged to the minimum restage tier (ufs0) and the replica on ufs2 + # is trimmed. Also make sure ufs1 still has a replica. The restage is signified by the + # trimming of the higher tier replica and an updated group AVU. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2') == False) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + + # Ensure that the "tracked" replica updates to replica 3, which is the replica restaged to ufs0. + self.assertEqual('3', get_tracked_replica(admin_session, logical_path)) + + # Now for the actual test. Tier out replica from ufs0 to ufs1. + sleep(6) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs1 and not preserved on ufs0. We test for the absence + # of a replica on ufs0 because the restage is signified by the trimming of the lower tier replica + # and an updated group AVU. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0') == False) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + + # Ensure that the "tracked" replica updates to replica 1, which is the replica on ufs1 that has not + # moved since it was created. + self.assertEqual('1', get_tracked_replica(admin_session, logical_path)) + + finally: + # Ensure that the preservation policy for each resource is set back to what it was before. + admin_session.run_icommand('imeta add -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.run_icommand('imeta rm -R ufs1 irods::storage_tiering::preserve_replicas true') + admin_session.run_icommand('imeta add -R ufs1 irods::storage_tiering::minimum_restage_tier true') + + admin_session.assert_icommand('irm -f ' + filename) + + def test_restaging_with_existing_replica_in_lower_tier__issue_235(self): + with storage_tiering_configured_with_log(): + IrodsController().restart(test_mode=True) + with session.make_session_for_existing_admin() as admin_session: + filename = 'test_put_file' + logical_path = '/'.join([admin_session.home_collection, filename]) + + try: + # Preserve replicas on tier 1, not tier 0 or tier 2. + admin_session.assert_icommand('imeta rm -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.assert_icommand('imeta add -R ufs1 irods::storage_tiering::preserve_replicas true') + + lib.create_local_testfile(filename) + admin_session.assert_icommand('iput -R ufs0 ' + filename) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Tier out replica from ufs0 to ufs1. + sleep(6) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs1 and not preserved on ufs0. + delay_assert_icommand(admin_session, 'ils -L ' + filename, 'STDOUT_SINGLELINE', 'ufs1') + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + + # Ensure that the "tracked" replica updates to replica 1, which is the tiered-out replica on ufs1. + self.assertEqual('1', get_tracked_replica(admin_session, logical_path)) + + # Tier out replica from ufs1 to ufs2. + sleep(15) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs2 and preserved on ufs1, and no replica exists on + # ufs0. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Ensure that the "tracked" replica updates to replica 2, which is the tiered-out replica on ufs2. + self.assertEqual('2', get_tracked_replica(admin_session, logical_path)) + + # Now for the actual test. Get the data object in order to trigger a restage, targeting replica 2 + # to ensure it is the replica that gets restaged. + admin_session.assert_icommand(['iget', '-n', '2', filename, '-'], 'STDOUT', 'TESTFILE') + + # Ensure that the restage has been scheduled. + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_data_movement') + + # Ensure that the replica is restaged to the minimum restage tier (ufs1) and the replica on ufs2 + # is trimmed. Also make sure ufs0 still does not have a replica. We test for the absence of a + # replica on ufs2 because the restage is signified by the trimming of the higher tier replica and an + # updated group AVU. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2') == False) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + + # Ensure that the "tracked" replica updates to replica 1, which is the replica on ufs1 that has not + # moved since it was created. + self.assertEqual('1', get_tracked_replica(admin_session, logical_path)) + + finally: + # Ensure that the preservation policy for each resource is set back to what it was before. + admin_session.run_icommand('imeta add -R ufs0 irods::storage_tiering::preserve_replicas true') + admin_session.run_icommand('imeta rm -R ufs1 irods::storage_tiering::preserve_replicas true') + + admin_session.assert_icommand('irm -f ' + filename) + + def test_replicas_cannot_be_restaged_to_a_higher_tier__issue_239(self): + with storage_tiering_configured_with_log(): + IrodsController().restart(test_mode=True) + with session.make_session_for_existing_admin() as admin_session: + filename = 'test_put_file' + logical_path = '/'.join([admin_session.home_collection, filename]) + + try: + lib.create_local_testfile(filename) + admin_session.assert_icommand('iput -R ufs0 ' + filename) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Tier out replica from ufs0 to ufs1. + sleep(6) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs1 and preserved on ufs0. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + + # Ensure that the "tracked" replica updates to replica 1, which is the tiered-out replica on ufs1. + self.assertEqual('1', get_tracked_replica(admin_session, logical_path)) + + # Tier out replica from ufs1 to ufs2. + sleep(15) + invoke_storage_tiering_rule() + admin_session.assert_icommand('iqstat', 'STDOUT', 'irods_policy_storage_tiering') + + # Ensure that the replica was tiered out to ufs2 and not preserved on ufs1. + lib.delayAssert(lambda: lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + + # Ensure that the "tracked" replica updates to replica 2, which is the tiered-out replica on ufs2. + self.assertEqual('2', get_tracked_replica(admin_session, logical_path)) + + # Now for the actual test. Get the data object in order to trigger a restage, targeting replica 0 + # to ensure it is the replica that gets restaged. + admin_session.assert_icommand(['iget', '-n', '0', filename, '-'], 'STDOUT', 'TESTFILE') + + # Wait for a bit to ensure that no data movement is happening. Assertions about whether anything + # happened are made below. The sleep time is 1 second, so 15 seconds of sleep is plenty of time to + # let any data migrations (or lack thereof) finish up. This test expects no movement to occur. + sleep(15) + admin_session.assert_icommand_fail('iqstat', 'STDOUT', 'irods_policy_data_movement') + + # Ensure that the replica is NOT restaged to the minimum restage tier (ufs1) and the replica on ufs0 + # is NOT trimmed. This is because the restage is happening from a tier lower than the minimum + # restage tier. Ensure that the replica on ufs2 hasn't moved either because it is not the replica + # which was scheduled for restage. + self.assertFalse(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs1')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs0')) + self.assertTrue(lib.replica_exists_on_resource(admin_session, logical_path, 'ufs2')) + + # Ensure that the "tracked" replica does not update. We assert this because the restage is not + # supposed to happen. + self.assertEqual('2', get_tracked_replica(admin_session, logical_path)) + + finally: + admin_session.assert_icommand('irm -f ' + filename) + class TestStorageTieringPluginObjectLimit(ResourceBase, unittest.TestCase): def setUp(self): diff --git a/sample_configuration.json b/sample_configuration.json deleted file mode 100644 index 09c6dba..0000000 --- a/sample_configuration.json +++ /dev/null @@ -1,16 +0,0 @@ -"rule_engines": [ - { - "instance_name": "irods_rule_engine_plugin-storage_tiering-instance", - "plugin_name": "irods_rule_engine_plugin-storage_tiering", - "plugin_specific_configuration" : { - "access_time_attribute" : "irods::access_time", - "storage_tiering_group_attribute" : "irods::storage_tiering::group", - "storage_tiering_time_attribute" : "irods::storage_tiering::time", - "storage_tiering_query_attribute" : "irods::storage_tiering::query", - "storage_tiering_verification_attribute" : "irods::storage_tiering::verification", - "storage_tiering_restage_delay_attribute" : "irods::storage_tiering::restage_delay", - "default_restage_delay_parameters" : "1s1h DOUBLE UNTIL SUCCESS OR 6 TIMES", - "time_check_string" : "TIME_CHECK_STRING" - } - } -] diff --git a/storage_tiering.cpp b/storage_tiering.cpp index fae991e..e8e55fa 100644 --- a/storage_tiering.cpp +++ b/storage_tiering.cpp @@ -30,9 +30,12 @@ #include #include #include -#include #include + +#include +#include +#include #include extern irods::resource_manager resc_mgr; @@ -218,11 +221,7 @@ namespace irods { _comm, config_.preserve_replicas, _resource_name); - std::transform( - pres.begin(), - pres.end(), - pres.begin(), - ::tolower); + std::transform(pres.begin(), pres.end(), pres.begin(), [](unsigned char c) { return std::tolower(c); }); return ("true" == pres); } catch(const exception& _e) { @@ -292,6 +291,59 @@ namespace irods { } // get_restage_tier_resource_name + auto storage_tiering::get_group_tier_for_resource(RcComm* _comm, + const std::string& _resource_name, + const std::string& _group_name) -> int + { + const auto query_string = fmt::format("select META_RESC_ATTR_UNITS where META_RESC_ATTR_NAME = '{}' and " + "META_RESC_ATTR_VALUE = '{}' and RESC_NAME = '{}'", + config_.group_attribute, + _group_name, + _resource_name); + + const auto query = irods::query{_comm, query_string}; + + if (query.empty()) { + THROW(CAT_NO_ROWS_FOUND, + fmt::format("Resource [{}] has no tier for group [{}].", _resource_name, _group_name)); + } + + if (query.size() > 1) { + THROW(CONFIGURATION_ERROR, + fmt::format("Resource [{}] has multiple tiers for group [{}].", _resource_name, _group_name)); + } + + int tier_value{}; + const auto meta_resc_attr_units = query.front()[0]; + const auto [str_ptr, ec] = std::from_chars( + meta_resc_attr_units.c_str(), meta_resc_attr_units.c_str() + meta_resc_attr_units.size(), tier_value); + + if (ec == std::errc::invalid_argument) { + THROW(CONFIGURATION_ERROR, + fmt::format("Resource [{}] has invalid tier [{}] for group [{}]: not a number", + _resource_name, + meta_resc_attr_units, + _group_name)); + } + else if (ec == std::errc::result_out_of_range) { + THROW(CONFIGURATION_ERROR, + fmt::format("Resource [{}] has invalid tier [{}] for group [{}]: out of range", + _resource_name, + meta_resc_attr_units, + _group_name)); + } + + if (tier_value < 0) { + THROW(CONFIGURATION_ERROR, + fmt::format("Resource [{}] has invalid tier [{}] for group [{}]: negative value", + _resource_name, + meta_resc_attr_units, + _group_name)); + } + + return tier_value; + } // get_group_tier_for_resource + std::string storage_tiering::get_data_movement_parameters_for_resource( rcComm_t* _comm, const std::string& _resource_name) { @@ -794,12 +846,23 @@ namespace irods { comm_, config_.group_attribute, _object_path); - const auto low_tier_resource_name = get_restage_tier_resource_name( - comm_, - group_name); - // do not queue movement if data is on minimum tier + const auto low_tier_resource_name = get_restage_tier_resource_name(comm_, group_name); + const auto source_resource_tier = get_group_tier_for_resource(comm_, _source_resource, group_name); + const auto low_tier_resource_tier = get_group_tier_for_resource(comm_, low_tier_resource_name, group_name); + + // do not queue movement if data is on minimum tier or lower // TODO:: query for already queued movement? - if(low_tier_resource_name == _source_resource) { + if (source_resource_tier <= low_tier_resource_tier) { + rodsLog( + LOG_DEBUG, + fmt::format("Replica for object [{}] on resource [{}] (tier [{}]) already exists on the minimum " + "restage tier resource [{}] (tier [{}]) or an even lower tier. Skipping restage.", + _object_path, + _source_resource, + source_resource_tier, + low_tier_resource_name, + low_tier_resource_tier) + .c_str()); return; } diff --git a/storage_tiering.hpp b/storage_tiering.hpp index 91584e1..b9723b8 100644 --- a/storage_tiering.hpp +++ b/storage_tiering.hpp @@ -121,6 +121,10 @@ namespace irods { rcComm_t* _comm, const std::string& _resource_name); + auto get_group_tier_for_resource(RcComm* _comm, + const std::string& _resource_name, + const std::string& _group_name) -> int; + std::string get_data_movement_parameters_for_resource( rcComm_t* _comm, const std::string& _resource_name); diff --git a/storage_tiering_configuration.hpp b/storage_tiering_configuration.hpp index ac0a6c9..77458c4 100644 --- a/storage_tiering_configuration.hpp +++ b/storage_tiering_configuration.hpp @@ -29,7 +29,7 @@ namespace irods { int number_of_scheduling_threads{4}; int default_minimum_delay_time{1}; int default_maximum_delay_time{30}; - std::string default_data_movement_parameters{"60s DOUBLE UNTIL SUCCESS OR 5 TIMES"}; + std::string default_data_movement_parameters{"60s REPEAT UNTIL SUCCESS OR 5 TIMES"}; const std::string instance_name{}; explicit storage_tiering_configuration(const std::string& _instance_name);