[BUG] Reserved quota locks up storage nodes

[benbierens]

Seen in testnet of v0.2.3.

After several days of successful running, the testnet has stopped filling new contracts. Examining the trace of a storage contract and logs from the nodes involved, we find this:

• The nodes successfully reserved a slot
• The nodes begin downloading
• The download failed: Storing the block would exceed the quota limit.

Typical quota values logged by this error message:

Used:        33387095674 (31GB)
reserved:    71839629312 (67GB)
limit:      105226698752 (98GB)

This shows that roughly 2/3 of the capacity of the node is used by reservations. This is true at a moment where the testnet is "quiet": There are no new/pending storage contracts.

This suggest that there is a possible flow of the sales module that does not properly clean up reservations.

[markspanbroek]

@benbierens and I dove a bit further in this, and found a likely explanation.

The testnet nodes run with a default block TTL of 365 days. The marketplace will update the expiry of blocks that it downloaded for slots by (indirectly) calling ensureExpiry with a new expiry date. However, ensureExpiry will call updateBlockMetadata, and that only updates the expiry if the new value is greater than the existing value. This means that data from slots that were filled by another node, or from requests that are finished, only get cleaned up after a year.

This data that sticks around for a year is no longer considered reserved by the marketplace (it told the repostore that it no longer needs it) and it's not cleaned up by the repostore either because the block expriy is not adjusted. So it fills up the used (non-reserved) part of the quota.

This leads to another problem; once the quota is full with 'used' and 'reserved' bytes, it is no longer possible to download anything into the reserved part. This is because the downloading state in sales can only release bytes from the 'reserved' portion of the quota after a block has already been downloaded. But we can't download a block, because there's no quota left.

Then there's a third problem that needs to be addressed if we want to fix the expiry of blocks. Currently we use the request expiry to update the block expiry, but that is a duration instead of a timestamp. This has been fixed as part of the vault PR, but that has not been merged yet (nor has it been reviewed yet).

[AuHau]

Well, it is a bit beyond my understanding why anybody would like to run nodes with default block TTL of 365 days 😳 That does not make sense, and it makes sense that it leads the nodes into trouble. IMHO that was the main problem.

This leads to another problem; once the quota is full with 'used' and 'reserved' bytes, it is no longer possible to download anything into the reserved part. This is because the downloading state in sales can only release bytes from the 'reserved' portion of the quota after a block has already been downloaded. But we can't download a block, because there's no quota left.

That is a very good catch though!

[benbierens]

This just in:
We have adjusted the blockTTL for our storage nodes from 365 days to 3 hours. It's making a difference, but it's not solving this entirely.

ERR 2025-06-05 06:32:22.810+00:00 failure returning bytes                    topics="sales cleanUp" tid=1 error="Quota usage would exceed the limit. Used: 17179759786'NByte, reserved: 88063721472'NByte, limit: 105226698752'NByte" bytes=67108864 requestId=0x2a1f..8e7d reservationId=0x58b8..b529 availabilityId=0xe82a..b94e slotIndex=0 count=4445710

It seems that:

• Slot cleanup is trying to remove a reservation
• This would increase the bytes locked up in the availability
• This results in trying to increase the reserved bytes at the repostore level
• Repostore doesn't have the bytes available

[markspanbroek]

Then there's a third problem that needs to be addressed if we want to fix the expiry of blocks. Currently we use the request expiry to update the block expiry, but that is a duration instead of a timestamp. This has been fixed as part of the vault PR, but that has not been merged yet (nor has it been reviewed yet).

I'm extracting the fix from the vault PR into #1258

[benbierens]

This one's not fixed yet.
We're seeing this in testnet on v0.2.5:
failure returning bytes is ocurring in the node logging while the nodes are busy attempting and sometimes failing to fill slots.
We end up in a situation like this:

The availability believes it is full, but the node has usually over 30GB of its 88GB free. (Checked the disk usages in the cluster: it lines up with what's reported by the space endpoint + 4 to 5 GB of metadata)

This explains the observation that our testnet should be able to store about 800 GB of active slots. But it's stuck at 520GB.

[emizzle]

My hope is that this will be fixed when we remove reservations from the repo store and availabilities. The current idea is for us to hold off removing reservations from availabilities until the RepoStore work was completed.

[benbierens]

Just adding some recent observations from the testnet:

As contracts finish and are restarted by the autoclient, more and more space is becoming locked up in the availability.
The quota says:

{
	"totalBlocks": 611799,
	"quotaMaxBytes": 94489280512,
	"quotaUsedBytes": 40094859264,
	"quotaReservedBytes": 86891520
}

=> 50.6 GB free (not reserved, and not used) of the 88 GB total.

But the availability says:

[
	{
		"id": "0x8e6643c76fe1bd757fe1528c82494421bab52c419a1d8c35c832f9b30e64becb",
		"totalSize": 86000000000,
		"freeSize": 515072,
		"duration": 604800,
		"minPricePerBytePerSecond": "5",
		"totalCollateral": "9999999999999999",
		"totalRemainingCollateral": "9999867435876351",
		"enabled": true,
		"until": 0
	}
]

=> 503 KB free.

[AuHau]

Can you get the availability's reservations? That should give us what Request and their state the availability hosts and then maybe we can find what the edge case is "draining" the availabilities?