Issue:
When using the StreamSets Test Framework and trying to run a test it fails with the following error:
$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py
2019-01-28 09:03:16 AM INFO] streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ...
INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0
benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /root/tests, inifile:
plugins: benchmark-3.1.1
collected 7 items
stage/test_dev_stages.py 2019-01-28 09:03:27 AM 3INFO] streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ...
2019-01-28 09:05:41 AM 0ERROR] streamsets.testframework.sdc] Error reaching Data Collector server. Data Collector server log follows ...
------------------------- SDC log - Begins -----------------------
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Configuration of maximum open file limit is too low: 1024 (expected at least 32768). Please consult https://goo.gl/LgvGFlSolution:
We can set the ulimit for Docker as:
# vi /etc/sysconfig/docker
OPTIONS='--default-ulimit nproc=32768:32768'This will set a soft limit of 32768 and a hard limit of 32768 child processes for all containers.