소스 보기

Overall

⭐ EPS = Event Per Second

Requirement	Recommendation
EPS	ES Configuration	Hardware per node (vCPU, RAM)	Elastic JVM RAM	Shards	Replica
Up to 1K - without Replica	All-in-one	(8,16GB)	8GB	5	0
Up to 1K - with Replica	3 node cluster	(8,16GB)	8GB	5	1
1K-5K - with Replica	3 node cluster	(8,64GB)	30GB	5	1
5K-10K - with Replica	Coordinating and Master Node	(8,32GB)	16GB
	3 Data Nodes	(8,64GB)	30GB	5	1
10K-15K - with Replica	Coordinating Node	(16,32GB)	16GB
	Master Node	(8,16GB)	8GB
	3 Data Nodes	(16,64GB)	30GB	10	1
...
35K-45K - with Replica	Coordinating Node	(16,64GB)	30GB
	Master Node	(8,16GB)	8GB
	9 Data Nodes	(16,64GB)	30GB	25	1
Add 5K EPS - with Replica	Add 1 Data Node	(16,64GB)	30GB	Add 3 Shards	1

Recommended Elasticsearch Configuration

Replica – at least 1
Master, Coordinator Only nodes and Data nodes on different machines
3 Master nodes – each with 8 vCPU, 16 GB RAM
At least 2 Coordinator Only nodes – each with 16 vCPU, 32 GB RAM. Two Coordinator Only nodes are chosen for failover. The exact number of Coordinator Only nodes depends on the EPS. See below for details.
At least 3 Hot Data nodes – each with 32 vCPU, 64GB RAM and SSD disks with at least 200 Gb/s I/O throughput. The exact number of Hot Data nodes depends on the EPS and retention policy (see below). (Memory to Disk Ratio = 1:30)
If you decide to deploy Warm nodes, deploy at least 3 Warm Data nodes – each with 32 vCPU, 64GB RAM and disks with at least 100 Gb/s I/O throughput. The exact number of Warm Data nodes depends on retention policy (see below). (memory to disk ratio = 1:160)
If you decide to deploy Frozen nodes, deploy at least 3 Frozen Data nodes – each with 16 vCPU, 64GB RAM and around 100 Gb/s I/O throughput. The exact number of Frozen Data nodes depends on retention policy

Keep num of shards under 15K, 샤드 개수는 15K 이하로 유지할 것

In Elasticsearch 6.x, Fortinet has observed that Elasticsearch CLI performance degrades when the total number of shards in the cluster (including Hot and Warm nodes) is more than 15K. Newer versions may have a higher upper limit.

Type	AWS Instance Type	Hardware Spec	Num of Instances	Note
Collector	c4.xlarge	4vCPU, 7 GB RAM
Worker	c4.2xlarge	8vCPU, 15 GB RAM	3	logstash
Super	m4.4xlarge	16vCPU, 64 GB RAM, CMDB Disk 10K IOPS	1	kibana
Elastic Search Master Node	c3.2xlarge	8vCPU, 16 GB RAM 8 GB JVM	1
Elastic Search Coordinating Node	m5.4xlarge	16vCPU, 64 GB RAM 30 GB JVM	1
Elastic Search Data Node	i3.4xlarge	16vCPU, 122 GB RAM, 2 x 1900 NVMe SSD 30 GB JVM	5	hot, warm

EPS	Storage per Day	Retention (Days)	Hot Data Nodes (32vCPU, 64GB RAM, SSD)
			Node Count	Disk Size
10K	1TB	7	4	2TB
		30	16	2TB

EPS	Storage per Day	Retention (Days)	Warm Data Nodes (32vCPU, 64GB RAM and ~100Gbps Disk I/O)
			Node Count	Disk Size
10K	1TB	30	3	10TB
		60	6	10TB
		90	9	10TB