...
Overall
⭐ EPS = Event Per Second
Requirement | Recommendation | ||||
|---|---|---|---|---|---|
EPS | ES Configuration | Hardware per node (vCPU, RAM) | Elastic JVM RAM | Shards | Replica |
Up to 1K - without Replica | All-in-one | (8,16GB) | 8GB | 5 | 0 |
Up to 1K - with Replica | 3 node cluster | (8,16GB) | 8GB | 5 | 1 |
1K-5K - with Replica | 3 node cluster | (8,64GB) | 30GB | 5 | 1 |
5K-10K - with Replica | Coordinating and Master Node | (8,32GB) | 16GB | ||
3 Data Nodes | (8,64GB) | 30GB | 5 | 1 | |
10K-15K - with Replica | Coordinating Node | (16,32GB) | 16GB | ||
Master Node | (8,16GB) | 8GB | |||
3 Data Nodes | (16,64GB) | 30GB | 10 | 1 | |
... | |||||
35K-45K - with Replica | Coordinating Node | (16,64GB) | 30GB | ||
Master Node | (8,16GB) | 8GB | |||
9 Data Nodes | (16,64GB) | 30GB | 25 | 1 | |
Add 5K EPS - with Replica | Add 1 Data Node | (16,64GB) | 30GB | Add 3 Shards | 1 |
https://docs.fortinet.com/document/fortisiem/6.1.0/sizing-guide/307212/fortisiem-sizing-information
Storage per Day
Suppose
R: Number of Replica (at least 1 is recommended)
E: EPS
D: Retention (days) in Hot nodes
Data per day = E * #seconds in a day (86400) * 500 Bytes * (R +1)
Storage per day = 1.25 * Data per day
Recommended Elasticsearch Configuration
...
30K EPS configuuration on AWS EC2
| Type | AWS Instance Type | Hardware Spec | Num of Instances | Note |
|---|---|---|---|---|
| Collector | c4.xlarge | 4vCPU, 7 GB RAM | ||
| Worker | c4.2xlarge | 8vCPU, 15 GB RAM | 3 | logstash |
| Super | m4.4xlarge | 16vCPU, 64 GB RAM, CMDB Disk 10K IOPS | 1 | kibana |
| Elastic Search Master Node | c3.2xlarge | 8vCPU, 16 GB RAM 8 GB JVM | 1 | |
| Elastic Search Coordinating Node | m5.4xlarge | 16vCPU, 64 GB RAM 30 GB JVM | 1 | |
| Elastic Search Data Node | i3.4xlarge | 16vCPU, 122 GB RAM, 2 x 1900 NVMe SSD 30 GB JVM | 5 | hot, warm |
| EPS | Storage per Day | Retention (Days) | Hot Data Nodes (32vCPU, 64GB RAM, SSD) | |
|---|---|---|---|---|
| Node Count | Disk Size | |||
| 10K | 1TB | 7 | 4 | 2TB |
| 30 | 16 | 2TB | ||
| EPS | Storage per Day | Retention (Days) | Warm Data Nodes (32vCPU, 64GB RAM and ~100Gbps Disk I/O) | |
|---|---|---|---|---|
| Node Count | Disk Size | |||
| 10K | 1TB | 30 | 3 | 10TB |
| 60 | 6 | 10TB | ||
| 90 | 9 | 10TB | ||
...