[Guidance] Thanos store max-time and consistency-delay

[realRunlo]

Hi everyone!

This is the setup i'm working on:

flowchart LR

subgraph LandingZone["Landing Zone"]

    %% Azure Cloud
    subgraph Azure["Azure Cloud"]
        subgraph AzureScrape["Scrape"]
            AZA[Prometheus Agent]
        end
    end

    %% AWS Cloud
    subgraph AWS["AWS Cloud"]

        subgraph Scrape["Scrape"]
            A[Prometheus Agent / ADOT Collector]
        end

        subgraph Ingestion["Ingestion"]
            B[Receiver Router]
            C[Receiver Ingesting]
            A -- Remote write --> B --> C
        end

        subgraph Storage["Storage"]
            D[🪣 S3]
            E[Storeger]
            G[Compactor]
            D --> E
            G --> D
        end

        subgraph Query["Query"]
            F[Querier]
        end

        C --> D
        F -- Store API --> E
        F -- Store API --> C
    end

    %% Cross-cloud connection
    AZA -- Remote write --> B

end

Loading

All components run on ECS using Spot Instances. The receiver-ingester runs without persistent storage, and TSDB blocks are flushed to S3 every two hours or when a task terminates.

I am running without persistent storage for two reasons:

• Attaching an EBS volume to a service means that tasks are tied to a specific Availability Zone.
• Attaching an EFS volume would likely increase latency.

My question concerns the configuration of max-time and consistency-delay. In my setup, there is a possibility that, at a given time, only S3 contains samples to serve. This can happen because Spot Instance rotation causes all tasks that contain a given serie to terminate, or because I redeploy the receiver-ingesting to update resource allocation.

If I configure max-time to 24 hours, I will have a gap in metrics for the data that was flushed by the receiver-ingesting for up to 24 hours. The same logic applies to the consistency-delay configuration.

My only alternative is to not set max-time and to set consistency-delay to a very low value, for example, 15 seconds.

Is this setup recommended? What can go wrong, aside from increasing S3 operations and therefore costs?

Thanks in advance for your input.

[mg1986jp]

For --max-time, the idea is to prevent your store from serving blocks that are too recent — that way your receiver or sidecar handles the recent data and the store handles historical. A common pattern is --max-time=-2h (or whatever your --consistency-delay is) so there's no overlap between what the store serves and what the ingestor serves.

--consistency-delay on the store is how long to wait before making newly uploaded blocks visible. This avoids serving partially-uploaded blocks. The default is 30m which is fine for most setups. You'd mainly increase it if your compactor or shipper is slow.

For your setup with receiver router + ingestor, I'd probably set the store's --max-time to something like -2h or -3h, matching or slightly exceeding your receiver's --tsdb.retention so there's overlap rather than gaps. Better to have some overlap (deduplication handles it) than gaps where nobody serves the data.

[s3onghyun]

Leaving --max-time unset on the store gateway is the right call. Its default is effectively unlimited, so the store serves all historical blocks from S3 while your receivers serve the most recent, not-yet-uploaded samples. The overlap between them is fine because Thanos deduplicates at query time; a too-tight --max-time is what creates real gaps when a receiver rotates or a Spot task is terminated before its 2h block is uploaded.

On --consistency-delay, one correction to the premise. Its default is 0s, and Thanos's own flag help notes that "GCS and S3 are (roughly) strongly consistent", so a low value is not inherently dangerous on S3 (S3 has been strongly consistent since Dec 2020). The flag exists mainly for genuinely eventually-consistent stores, where it prevents loading a block whose upload or listing is not yet consistent. So 15s is not risky on S3 specifically. I would just avoid pushing it to zero as a habit and keep a modest value if your backend is anything other than S3/GCS.

Net: don't set --max-time, keep a modest --consistency-delay, and let dedup handle the receiver/store overlap. The only real cost of no max-time is extra S3 GET/LIST operations.