STF - Configuration of maximum open file limit is too low: 1024 (expected at least 32768).

  • 13 June 2022
  • 2 replies
  • 362 views

Userlevel 4
Badge

Issue:

When using the StreamSets Test Framework and trying to run a test it fails with the following error:

$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 
2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... 
INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... 
============================= test session starts ============================== 
platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 
benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) 
rootdir: /root/tests, inifile: 
plugins: benchmark-3.1.1 
collected 7 items
stage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 
2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Collector server. Data Collector server log follows ... 

------------------------- SDC log - Begins ----------------------- 
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS 
Configuration of maximum open file limit is too low: 1024 (expected at least 32768). Please consult https://goo.gl/LgvGFl

Solution:

We can set the ulimit for Docker as:

# vi /etc/sysconfig/docker
OPTIONS='--default-ulimit nproc=32768:32768'

This will set a soft limit of 32768 and a hard limit of 32768 child processes for all containers.


2 replies

Hello Akshay,

Trying to install a Data collector on a VM in Azure. It always gives me the below error

 

Nov 6, 2022, 10:54:34 AM ERROR 2c50213d91b5

Configuration of maximum open files limit is too low: 1024 (expected at least 32768). Please consult https://bit.ly/3ihM9ok

 

 

I’ve set all users ulimit to 1 million

using ulimit -n 100000

 

Are there any other things to do?

Userlevel 4
Badge

Hello @Varun PVS  - Thank you for reaching out to Streamsets Community. 

You can try setting the ulimit in the ~/. bashrc file so that as soon as you login to the server the bashrc script executed and it will set the limit.

prlimit --pid --nofile=33000:33000 

Let me know how it goes.

Thank you - AkshayJ

Reply