Tier-out does not use tracked replica if preserved replicas exist in a higher tier

[alanking]

Discovered while investigating #272... Tip of 4-3-stable at time of writing (ce49e16)

I have a 3-resource tier group:

$ imeta ls -R ufs0
AVUs defined for resource ufs0:
attribute: irods::storage_tiering::group
value: example_group
units: 0
----
attribute: irods::storage_tiering::preserve_replicas
value: true
units: 
----
attribute: irods::storage_tiering::time
value: 10
units: 
$ imeta ls -R ufs1
AVUs defined for resource ufs1:
attribute: irods::storage_tiering::group
value: example_group
units: 1
----
attribute: irods::storage_tiering::preserve_replicas
value: true
units: 
----
attribute: irods::storage_tiering::time
value: 5
units: 
$ imeta ls -R ufs2
AVUs defined for resource ufs2:
attribute: irods::storage_tiering::group
value: example_group
units: 2

I tiered out an object all the way to the terminal tier:

$ ils -l foo
  rods              0 ufs0            5 2025-02-03.20:11 & foo
  rods              1 ufs1            5 2025-02-03.20:13 & foo
  rods              2 ufs2            5 2025-02-03.20:14 & foo
$ imeta ls -d foo
AVUs defined for dataObj /tempZone/home/rods/foo:
attribute: irods::access_time
value: 1738613647
units:
----
attribute: irods::storage_tiering::group
value: example_group
units: 2

I accessed the replica on ufs2 to trigger a restage. No problems there:

$ ils -l foo
  rods              0 ufs0            5 2025-02-03.20:11 & foo
  rods              1 ufs1            5 2025-02-03.20:13 & foo
$ imeta ls -d foo
AVUs defined for dataObj /tempZone/home/rods/foo:
attribute: irods::access_time
value: 1738613762
units:
----
attribute: irods::storage_tiering::group
value: example_group
units: 0

Then I ran the tiering rule again. A new replica appeared on ufs2 and this became the tracked replica:

$ ils -l foo
  rods              0 ufs0            5 2025-02-03.20:11 & foo
  rods              1 ufs1            5 2025-02-03.20:13 & foo
  rods              2 ufs2            5 2025-02-03.20:18 & foo
$ imeta ls -d foo
AVUs defined for dataObj /tempZone/home/rods/foo:
attribute: irods::access_time
value: 1738613889
units:
----
attribute: irods::storage_tiering::group
value: example_group
units: 2

The decision was built into the scheduled migration:

{
  "delay_conditions": "<INST_NAME>irods_rule_engine_plugin-unified_storage_tiering-instance</INST_NAME><EF>60s REPEAT UNTIL SUCCESS OR 5 TIMES</EF><PLUSET>19s</PLUSET>",
  "destination-resource": "ufs2",
  "group-name": "example_group",
  "md5": "9c804721d7c60731fd961fafc5e1329e",
  "object-path": "/tempZone/home/rods/foo",
  "preserve-replicas": true,
  "rule-engine-instance-name": "irods_rule_engine_plugin-unified_storage_tiering-instance",
  "rule-engine-operation": "irods_policy_data_movement",
  "source-replica-number": "1",
  "source-resource": "ufs1",
  "verification-type": "catalog"
}

So... is this the expected behavior?

I think not. The tracked replica should be the one that gets tiered out, not the replica on the highest tier. Even though no data migration would occur, the access_time continues to drive the policy and so it would get scheduled to tier out from ufs1 once the specified time had passed.

The only consequence of this case that I can think of right now is that the data arrives in the terminal tier sooner than expected. However, as we contemplate leaning into the "tracked" replica for tiering / restaging decision making, I think other effects could occur.

[alanking]

Perhaps a separate issue...

If you preserve replicas on the last tier, too, the replica will never tier out. It will sit there like this forever:

$ ils -l foo
  rods              0 ufs0            5 2025-02-03.20:11 & foo
  rods              1 ufs1            5 2025-02-03.20:13 & foo
  rods              2 ufs2            5 2025-02-03.20:18 & foo
$ imeta ls -d foo
AVUs defined for dataObj /tempZone/home/rods/foo:
attribute: irods::access_time
value: 1738615953
units: 
----
attribute: irods::storage_tiering::group
value: example_group
units: 0

Interestingly, if you set preserve_replicas to false on the tier where the tracked replica sits and run the tiering rule, the replica is trimmed and the tracked replica is updated as expected:

$ ils -l foo
  rods              1 ufs1            5 2025-02-03.20:13 & foo
  rods              2 ufs2            5 2025-02-03.20:18 & foo
$ imeta ls -d foo
AVUs defined for dataObj /tempZone/home/rods/foo:
attribute: irods::access_time
value: 1738615953
units: 
----
attribute: irods::storage_tiering::group
value: example_group
units: 1

Notice that the access_time did NOT update.

So, if no migration or trimming occurs, the tier-out rule will not do anything. If no data migration occurs, the access_time will not be updated. I'm not really sure which parts of this are expected or not - just writing down observations for now. :)