Skip to content

Commit 472ca92

Browse files
committed
CLOUDPREM: Update sizing recommendations
1 parent add1c3a commit 472ca92

File tree

1 file changed

+14
-17
lines changed

1 file changed

+14
-17
lines changed

content/en/cloudprem/configure/cluster_sizing.md

Lines changed: 14 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -26,33 +26,30 @@ These are starting recommendations. Monitor your cluster's performance and resou
2626

2727
## Indexers
2828

29-
- **Performance:** To index 5 MB/s of logs, CloudPrem needs approximately 1 vCPU and 2 GB of RAM.
30-
- **Recommended Pod Sizes:** Datadog recommends that you deploy indexer pods with either:
31-
- 2 vCPUs and 4 GB of RAM
32-
- 4 vCPUs and 8 GB of RAM
33-
- 8 vCPUs and 16 GB of RAM
34-
- **Storage:** Indexers require persistent storage (preferably SSDs, but local HDDs or remote EBS volumes can also be used) to store temporary data while constructing the index files.
35-
- Minimum: 100 GB per pod
36-
- Recommendation (for pods > 4 vCPUs): 200 GB per pod
37-
- **Example Calculation:** To index 1 TB per day (~11.6 MB/s):
38-
- Required vCPUs: `(11.6 MB/s / 5 MB/s/vCPU) ≈ 2.3 vCPUs`
39-
- Rounding up, you might start with one indexer pod configured with 3 vCPUs and 6 GB RAM, requiring a 100 GB EBS volume. (Adjust this configuration based on observed performance and redundancy needs.)
29+
- **Performance:** Indexing performance depends heavily on the characteristics of the ingest logs, such as their size, number of attributes, and level of nesting. However, we recommend using a baseline indexing throughput of **5 MB/s per vCPU** to determine your initial sizing.
30+
- **Memory:** We recommend 4 GB of RAM per vCPU.
31+
- **Recommended Pod Sizes:** Datadog recommends deploying indexer pods with at least 2 vCPUs and 8 GB of RAM.
32+
- **Storage:** Indexers require at least 200 GB of persistent storage (preferably local SSDs, but local HDDs or network-attached block storage volumes such as Amazon EBS, or Azure Managed Disks can also be used) to store temporary data while creating and merging index files. In addition, each indexer vCPU writes on disk at a rate of approximately 20 MB/s. For Amazon EBS volumes, this is equivalent to 320 IOPS per vCPU (assuming 64 KB IOPS).
33+
- **Example Calculation:** To index 1 TB of logs per day (~11.6 MB/s):
34+
- Required vCPUs: `11.6 MB/s / 5 MB/s per vCPU ≈ 2.3 vCPUs`
35+
- Required RAM: `2.3 vCPUs × 4 GB RAM ≈ 9 GB RAM`
36+
- Adding some headroom, you could start with one indexer pod configured with 3 vCPUs, 12 GB RAM, and a 200 GB disk. Adjust these values based on observed performance and redundancy needs.
4037

4138
## Searchers
4239

43-
- **Performance:** Search performance depends heavily on the workload (query complexity, concurrency, data scanned).
40+
- **Performance:** Search performance depends heavily on the workload (query complexity, concurrency, data scanned). For instance, term queries (`status:error AND message:exception`) are usually computationally less expensive than aggregations.
4441
- **Rule of Thumb:** A general starting point is to provision roughly double the total number of vCPUs allocated to Indexers.
4542
- **Memory:** We recommend 4 GB of RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests.
4643

4744
## Other services
4845

49-
The following components are typically lightweight:
46+
We recommend allocating the following resources for these lightweight components:
5047

51-
- **Control Plane:** 1 vCPU, 2 GB RAM
52-
- **Metastore:** 1 vCPU, 2 GB RAM
53-
- **Janitor:** 1 vCPU, 2 GB RAM
48+
- **Control Plane:** 2 vCPUs, 4 GB RAM, 1 replica
49+
- **Metastore:** 2 vCPUs, 4 GB RAM, 2 replicas
50+
- **Janitor:** 2 vCPUs, 4 GB RAM, 1 replica
5451

55-
## Postgres Metastore backend
52+
## PostgreSQL Database
5653

5754
- **Instance Size:** For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient
5855
- **AWS RDS Recommendation:** If using AWS RDS, the `t4g.medium` instance type is a suitable starting point

0 commit comments

Comments
 (0)