Issue:
After configuring properly SDC to use JKS credential store and adding a credential into it when we try to use the function on the pipeline to retrieve the password we can see the following exception:
com.streamsets.pipeline.api.StageException: JKS_CRED_STORE_003 - Credential 'my_cred' not found
Also, when checking on the path the credential store should be created (usually /etc/sdc) we cannot see the file, but if we try to list the credential store contents through the command line we can see them:
bin/streamsets stagelib-cli jks-credentialstore list -i jks
The Java KeyStore 'jks' has the following credentials
---------------------------------------------------------------
my_cred
Solution:
Usually, this issue is caused by a missing configuration. The script copied at the systemd folder (sdc.service) defines the service and contains some environment variable overrides, including:
Environment=SDC_CONF=/etc/sdc
This ensures that when SDC is run via systemctl start sdc
, it uses /etc/sdc/ as the conf dir.
The problem is then when setting up the keystore file. The bin/streamsets stagelib-cli jks-credentialstore <add|list>
commands go through the same scripts to load the environment variables. However, they don't get the injected environment variables from sdc.service because they're not a service. This means that SDC_CONF is set to the default as determined by /opt/streamsets-datacollector/libexec/_sdc, which is:
BASEDIR=`dirname ${PRG}` # this is the /opt/streamsets-datacollector/libexec/ directory
BASEDIR=`cd ${BASEDIR}/..;pwd`
SDC_DIST=${BASEDIR}
SDC_HOME=${SDC_HOME:=${SDC_DIST}}
SDC_CONF=${SDC_CONF:=${SDC_HOME}/etc}
In order to fix this you would need to execute a similar command as for the one used to configure the SDC_RESOURCES directory:
sed -i 's|#export SDC_CONF=/etc/sdc|export SDC_CONF=/etc/sdc|' /opt/streamsets-datacollector/libexec/sdc-env.sh
It is also recommended to uncomment those other lines with the following commands:
sed -i 's|#export SDC_LOG=/var/log/sdc|export SDC_LOG=/var/log/sdc|' /opt/streamsets-datacollector/libexec/sdc-env.sh
sed -i 's|#export SDC_DATA=/var/lib/sdc|export SDC_DATA=/var/lib/sdc|' /opt/streamsets-datacollector/libexec/sdc-env.sh