We‘ve set up a machine for Self-Managed Deployments (specifications below) with Docker as it’s said in the documentation.
Machine specifications:
- Operating system: Linux (ubuntu 22.04)
- Disk size: 30GiB
- RAM: 4GiB
- Performance tier: P10 - 500 IOPS, 100MMB/s
We succesfully created some Data Collector and Transfomer engines. The problem occurs when we try to restart/run all of the engines at the same time. Server just freezes and eventually stops responding.
If you look at the metrics of the machine, you can see that it stops usually when “Disk Bytes” peek.
- My first question is, what could cause this huge usage of resources to crash the server eventually?
The next things are Transformer engines. In documentation it says:
“Important: When using a Transformer engine that works with a Spark cluster, edit the transformer.base.http.url property on the Transformer Configuration tab. Uncomment the property and set it to the Transformer URL. For more information, see Granting the Spark Cluster Access to Transformer in the Transformer engine documentation.”
- So my second question is this needed to be done for every/any kind of transformer engine or only for Spark ones, and how do you know it is Spark’s?
Thank you for all the answers and suggesstions provided!