Hello guys i want to build my SIEM to handle at least 10000EPS what configuration should i do and what requirement need to use with what number of nodes and info i need
Any one knew how i can do it
There is no quick answer to this, you need to do some tests on a proof of concept environment.
The number of nodes depends on a lot of other factos, like how many data you have per day, for how long you want to keep this data, if you are going to have snapshtos or not etc.
Try to spin up a small cluster with 3 nodes where all 3 are master eligible and data nodes, this will make it easier to estimate the resources you will need in production once you have your ingestion working.
I would say that one important requirement is to use fast disks, ssd or nvme disks for example.
^^^^^ Exactly This.
If you want to see what the HW profiles we run in Elastic Cloud see here.
These are a good place to start with specs
This a good answer
Is there any way to simulate the trafic of more than 8k EPS to take a look of stability of it
There's lots of logs generator simulators out there...
Or just go to chat gpt and ask it to write it for you in Python. It'll get 90% there the first time
And of course there's our formal rally benchmark tool if you want to use that. That takes a little bit of work and understanding
Oh and just to set your mind at ease 10K EPS ingest on a well set up 3 now de cluster should not be a problem... Then it comes down to how long you want to keep the data and how you are going to search / use it.
Can u give me some documentation can help me in this
The quick start take just a couple commands
To set target throughput us target-throughput configuration