Elasticsearch Sizing Guide

Overall

⭐ EPS = Event Per Second

Requirement	Recommendation
EPS	ES Configuration	Hardware per node (vCPU, RAM)	Elastic JVM RAM	Shards	Replica
Up to 1K - without Replica	All-in-one	(8,16GB)	8GB	5	0
Up to 1K - with Replica	3 node cluster	(8,16GB)	8GB	5	1
1K-5K - with Replica	3 node cluster	(8,64GB)	30GB	5	1
5K-10K - with Replica	Coordinating and Master Node	(8,32GB)	16GB
	3 Data Nodes	(8,64GB)	30GB	5	1
10K-15K - with Replica	Coordinating Node	(16,32GB)	16GB
	Master Node	(8,16GB)	8GB
	3 Data Nodes	(16,64GB)	30GB	10	1
...
35K-45K - with Replica	Coordinating Node	(16,64GB)	30GB
	Master Node	(8,16GB)	8GB
	9 Data Nodes	(16,64GB)	30GB	25	1
Add 5K EPS - with Replica	Add 1 Data Node	(16,64GB)	30GB	Add 3 Shards	1

Storage per Day

Suppose

Data per day = E * #seconds in a day (86400) * 500 Bytes * (R +1)

Storage per day = 1.25 * Data per day

Recommended Elasticsearch Configuration

Replica – at least 1
Master, Coordinator Only nodes and Data nodes on different machines
3 Master nodes – each with 8 vCPU, 16 GB RAM
At least 2 Coordinator Only nodes – each with 16 vCPU, 32 GB RAM. Two Coordinator Only nodes are chosen for failover. The exact number of Coordinator Only nodes depends on the EPS. See below for details.
At least 3 Hot Data nodes – each with 32 vCPU, 64GB RAM and SSD disks with at least 200 Gb/s I/O throughput. The exact number of Hot Data nodes depends on the EPS and retention policy (see below). (Memory to Disk Ratio = 1:30)
If you decide to deploy Warm nodes, deploy at least 3 Warm Data nodes – each with 32 vCPU, 64GB RAM and disks with at least 100 Gb/s I/O throughput. The exact number of Warm Data nodes depends on retention policy (see below). (memory to disk ratio = 1:160)
If you decide to deploy Frozen nodes, deploy at least 3 Frozen Data nodes – each with 16 vCPU, 64GB RAM and around 100 Gb/s I/O throughput. The exact number of Frozen Data nodes depends on retention policy

Keep num of shards under 15K, 샤드 개수는 15K 이하로 유지할 것

In Elasticsearch 6.x, Fortinet has observed that Elasticsearch CLI performance degrades when the total number of shards in the cluster (including Hot and Warm nodes) is more than 15K. Newer versions may have a higher upper limit.

Instance count	Master node RAM size	Maximum supported shard count	Recommended minimum dedicated master instance type
1–10	8 GiB	10K	m5.large.search or m6g.large.search
11–30	16 GiB	30K	c5.2xlarge.search or c6g.2xlarge.search
31–75	32 GiB	40K	r5.xlarge.search or r6g.xlarge.search
76 – 125	64 GiB	75K	r5.2xlarge.search or r6g.2xlarge.search
126 – 200	128 GiB	75K	r5.4xlarge.search or r6g.4xlarge.search

Type	AWS Instance Type	Hardware Spec	Num of Instances	Note
Collector	c4.xlarge	4vCPU, 7 GB RAM
Worker	c4.2xlarge	8vCPU, 15 GB RAM	3	logstash
Super	m4.4xlarge	16vCPU, 64 GB RAM, CMDB Disk 10K IOPS	1	kibana
Elastic Search Master Node	c3.2xlarge (구세대 인스턴스)	8vCPU, 16 GB RAM 8 GB JVM 2 x 80GB Instance Store	1
Elastic Search Coordinating Node	m5.4xlarge	16vCPU, 64 GB RAM 30 GB JVM	1
Elastic Search Data Node	i3.4xlarge	16vCPU, 122 GB RAM, 2 x 1900 NVMe SSD 30 GB JVM	5	hot, warm

EPS	Storage per Day	Retention (Days)	Hot Data Nodes (32vCPU, 64GB RAM, SSD)
			Node Count	Disk Size
10K	1TB	7	4	2TB
		30	16	2TB

EPS	Storage per Day	Retention (Days)	Warm Data Nodes (32vCPU, 64GB RAM and ~100Gbps Disk I/O)
			Node Count	Disk Size
10K	1TB	30	3	10TB
		60	6	10TB
		90	9	10TB