You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/en/cloudprem/configure/cluster_sizing.md
+14-17Lines changed: 14 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,33 +26,30 @@ These are starting recommendations. Monitor your cluster's performance and resou
26
26
27
27
## Indexers
28
28
29
-
-**Performance:** To index 5 MB/s of logs, CloudPrem needs approximately 1 vCPU and 2 GB of RAM.
30
-
-**Recommended Pod Sizes:** Datadog recommends that you deploy indexer pods with either:
31
-
- 2 vCPUs and 4 GB of RAM
32
-
- 4 vCPUs and 8 GB of RAM
33
-
- 8 vCPUs and 16 GB of RAM
34
-
-**Storage:** Indexers require persistent storage (preferably SSDs, but local HDDs or remote EBS volumes can also be used) to store temporary data while constructing the index files.
35
-
- Minimum: 100 GB per pod
36
-
- Recommendation (for pods > 4 vCPUs): 200 GB per pod
37
-
-**Example Calculation:** To index 1 TB per day (~11.6 MB/s):
- Rounding up, you might start with one indexer pod configured with 3 vCPUs and 6 GB RAM, requiring a 100 GB EBS volume. (Adjust this configuration based on observed performance and redundancy needs.)
29
+
-**Performance:** Indexing performance depends heavily on the characteristics of the ingest logs, such as their size, number of attributes, and level of nesting. However, we recommend using a baseline indexing throughput of **5 MB/s per vCPU** to determine your initial sizing.
30
+
-**Memory:** We recommend 4 GB of RAM per vCPU.
31
+
-**Recommended Pod Sizes:** Datadog recommends deploying indexer pods with at least 2 vCPUs and 8 GB of RAM.
32
+
-**Storage:** Indexers require at least 200 GB of persistent storage (preferably local SSDs, but local HDDs or network-attached block storage volumes such as Amazon EBS, or Azure Managed Disks can also be used) to store temporary data while creating and merging index files. In addition, each indexer vCPU writes on disk at a rate of approximately 20 MB/s. For Amazon EBS volumes, this is equivalent to 320 IOPS per vCPU (assuming 64 KB IOPS).
33
+
-**Example Calculation:** To index 1 TB of logs per day (~11.6 MB/s):
- Adding some headroom, you could start with one indexer pod configured with 3 vCPUs, 12 GB RAM, and a 200 GB disk. Adjust these values based on observed performance and redundancy needs.
40
37
41
38
## Searchers
42
39
43
-
-**Performance:** Search performance depends heavily on the workload (query complexity, concurrency, data scanned).
40
+
-**Performance:** Search performance depends heavily on the workload (query complexity, concurrency, data scanned). For instance, term queries (`status:error AND message:exception`) are usually computationally less expensive than aggregations.
44
41
-**Rule of Thumb:** A general starting point is to provision roughly double the total number of vCPUs allocated to Indexers.
45
42
-**Memory:** We recommend 4 GB of RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests.
46
43
47
44
## Other services
48
45
49
-
The following components are typically lightweight:
46
+
We recommend allocating the following resources for these lightweight components:
50
47
51
-
-**Control Plane:**1 vCPU, 2 GB RAM
52
-
-**Metastore:**1 vCPU, 2 GB RAM
53
-
-**Janitor:**1 vCPU, 2 GB RAM
48
+
-**Control Plane:**2 vCPUs, 4 GB RAM, 1 replica
49
+
-**Metastore:**2 vCPUs, 4 GB RAM, 2 replicas
50
+
-**Janitor:**2 vCPUs, 4 GB RAM, 1 replica
54
51
55
-
## Postgres Metastore backend
52
+
## PostgreSQL Database
56
53
57
54
-**Instance Size:** For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient
58
55
-**AWS RDS Recommendation:** If using AWS RDS, the `t4g.medium` instance type is a suitable starting point
0 commit comments