"Failed at step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux
Problem
When starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup. The following error is shown when the status of the service is checked (using the systemctl status sdc
command).
systemdt13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied
Cause
This error indicates that the script SDC_HOME/bin/streamsets
(which is used to start SDC) could not be launched by systemd. This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled).
Solution
Step 1: Examine the Service Configuration
Before looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service
. Examine this file and look for the following lines to find the user and group. The below example shows the default (and recommended) values:
tService]
User=sdc
Group=sdc
The SDC installation directory can be found a few lines further down in this file. Again, this example shows the default setting:
Environment=SDC_HOME=/opt/streamsets-datacollector
Ensure that the values configured in the unit file are correct.
On systems where SELinux is enabled, the path for the SDC installation directory must not begin with /home
or /root
, as SELinux forbids system services from running out of these directories (see Step 3: Validate SELinux Context for more information).
Step 2: Validate Permissions
The user which runs the SDC service must have access to read and execute the SDC_HOME/bin/streamsets
file (where SDC_HOME
represents the SDC installation directory, as validated in Step 1 above). Due to the nature of directory objects and permissions, this also requires that the user have read and execute permissions on the SDC_HOME
directory itself, and the SDC_HOME/bin
directory as well.
By default, both directories and the script file itself have the following permissions:
- owner: read, write, execute
- group: read, write, execute
- others: read, execute
This permission set is frequently described in Linux by the text rwxrwxr-x
or the number 775
.
Permissions on a directory or file object can be examined with the stat path
command (where path
represents the file or directory path of the object to be examined)
For example, in a default installation (where SDC was installed in the directory /opt/streamsets-datacollector
) the command to examine the installation directory is:
stat /opt/streamsets-datacollector
and it returns output similar to the following:
# stat /opt/streamsets-datacollector
File: /opt/streamsets-datacollector
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: ca04h/51716d Inode: 310384541 Links: 18
Access: (0775/drwxrwxr-x) Uid: ( 1000/ec2-user) Gid: ( 1000/ec2-user)
The file permissions are shown in the Access
field from these results, which is highlighted in the example above. The permissions shown in this example “(0775/drwxrwxr-x)
” are the correct permissions for these objects. Note that the “d
” after the slash in this field indicates this is a directory, and when running stat on the SDC_HOME/bin/streamsets
file it will show a “-” there (indicating a regular file) instead.
There is an additional special case where the installation directory may show an “l” after the slash in the Access field instead of a “d
” (for example, “(0775/lrwxrwxr-x)
”. This is also valid, but it indicates that this path is a “symbolic link” and that the actual installation files are located elsewhere in the filesystem. More information on symbolic links can be found in the section, Installation Directory as a Symbolic Link later in this document.
If any of these objects does not have the correct permissions, the permissions can be set with the “sudo chmod 775 path
” command (again, where path
represents the file or directory path of the object to be modified).
For example, the commands for the two directories and the script file in a default environment are:
sudo chmod 775 /opt/streamsets-datacollector
sudo chmod 775 /opt/streamsets-datacollector/bin
sudo chmod 775 /opt/streamsets-datacollector/bin/streamsets
Step 3: Validate SELinux Context
On systems where Security Enhanced Linux (SELinux) is enabled, the output of the stat
command will include an additional field of importance (highlighted in the example below) named “Context
”:
# stat /opt/streamsets-datacollector
File: /opt/streamsets-datacollector
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: ca04h/51716d Inode: 310384541 Links: 18
Access: (0775/drwxrwxr-x) Uid: ( 1000/ec2-user) Gid: ( 1000/ec2-user)
Context: unconfined_u:object_r:usr_t:s0
The “Context” field describes the “SELinux context”, which is a set of labels used by SELinux to restrict access to system resources.
The SELinux context includes a user label (“unconfined_u
” in the example above), a role label (“object_r
” in the example above), a type label (“usr_t”
in the example above), and a level (“s0
” in the example above).
The type label is noteworthy because it is initially set by SELinux based on the location of the object, and this is the reason why the SDC installation directory must not reside in the /home
or /root
directory trees on systems where SELinux is enabled. Objects which reside in the /home directory tree are assigned the type of “user_home_t
”, and objects which reside in the /root directory tree are assigned the type of “admin_home_t
”. SELinux does not permit files of these types to be executed as system services.
As such, if the SDC installation directory is in the /root
or /home
directory trees on a system where SELinux is enabled, attempting to start the sdc
service will result in a Permission Denied error, as in this example:
systemde13801]: sdc.service: Failed at step EXEC spawning /root/streamsets-datacollector-5.5.0/bin/streamsets: Permission denied
Examining /root/streamsets-datacollector-5.5.0/bin/streamsets
in this case will show the following:
# stat /root/streamsets-datacollector-5.5.0/bin/streamsets
File: /root/streamsets-datacollector-5.5.0/bin/streamsets
Size: 3493 Blocks: 8 IO Block: 4096 regular file
Device: ca04h/51716d Inode: 117440927 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 990/ sdc) Gid: ( 990/ sdc)
Context: unconfined_u:object_r:admin_home_t:s0
The type label is admin_home_t
, which disallows the script from being executed as a systemd service.
In most cases where SDC has been installed in the /root
or /home
directory tree, the simplest solution is to follow these steps:
- Re-extract the SDC files from the tarball in a new location outside of the
/home
or/root
directory tree. - Modify the unit file (
/etc/systemd/system/sdc.service
) to use the new location forSDC_HOME
.
Alternately, this method can also be used:
- Move the existing SDC installation directory (using the
mv
command) to a new location outside the/home
or/root
directory trees - Correct the SELinux context for the SDC files with the command:
restorecon -R -v NEW_PATH
(whereNEW_PATH
represents the new installation directory path).- This is necessary because SELinux does not update the context automatically when the files are moved.
- Modify the unit file (
/etc/systemd/system/sdc.service
) to use the new installation directory forSDC_HOME
.
Installation Directory as a Symbolic Link
In the case described in the section "Step 3: Validate SELinux Context" above, the fact that the SDC installation directory resides in the /root
directory tree is evident. However, the use of symbolic links can mask the location of the installation directory.
A symbolic link is a special kind of file which references another location in the filesystem, known as the "target" of the link. A symbolic link's target can be a single file, but it can also be a directory, and when a symbolic link is targeted on a directory, the path of the link can be used to reference files in that directory (and its subdirectories).
As an example, consider a symbolic link /opt/streamsets-datacollector
, whose target is the directory /root/streamsets-datacollector-5.5.0
, where the actual SDC files are extracted.
Examining /opt/streamsets-datacollector
in this case demonstrates that it is a symbolic link:
# stat /opt/streamsets-datacollector
File: /opt/streamsets-datacollector -> /root/streamsets-datacollector-5.5.0
Size: 36 Blocks: 0 IO Block: 4096 symbolic link
Device: ca04h/51716d Inode: 12887 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:usr_t:s0
Note the highlighted sections in the example output above:
- The
File:
field on the first line of the output shows an arrow ("->
"), followed by the destination of the symbolic link (in this case,/root/streamsets-datacollector-5.5.0
) - The file type is shown as "
symbolic link
". - The
Access:
field shows that the first character of the permission string is "l
" (indicating a symbolic link) rather thand
(indicating a directory).
In this case, the system can find the streamsets
script at /opt/streamsets-datacollector/bin/streamsets
# ls -l /opt/streamsets-datacollector/bin/streamsets
-rwxr-xr-x. 1 sdc sdc 3493 Apr 18 18:01 /opt/streamsets-datacollector/bin/streamsets
However, configuring the path /opt/streamsets-datacollector/
for SDC_HOME
in the service unit file /etc/systemd/system/sdc.service
and attempting to start the service results in the following error:
systemdl13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied
Examining /opt/streamsets-datacollector/bin/streamsets
shows the reason that permission is denied: the SELinux context type label is "admin_home_t
":
# stat /opt/streamsets-datacollector/bin/streamsets
File: /opt/streamsets-datacollector/bin/streamsets
Size: 3493 Blocks: 8 IO Block: 4096 regular file
Device: ca04h/51716d Inode: 117440927 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 990/ sdc) Gid: ( 990/ sdc)
Context: unconfined_u:object_r:
admin_home_t:s0
This is because the SELinux context type label is set based on the actual location of the file, which is /root/streamsets-datacollector-5.5.0/bin/streamsets
In this case, the solution is similar to that in Step 3: Validate SELinux Context above; the SDC files must be located in a directory outside of /home
or /root
.
In most cases, the simplest solution is to follow these steps:
- Re-extract the SDC files in a new location outside of the
/home
or/root
directory tree - Re-create the symbolic link to target the new installation directory.
Alternately, this method can also be used:
- Move the existing SDC installation directory (using the mv command) to a new location outside the
/home
or/root
directory trees - Update the SELinux context for the SDC files by running the following command:
restorecon -R -v NEW_PATH
(whereNEW_PATH
represents the new installation directory path).- This is necessary because SELinux does not update the context automatically when the files are moved.
- Re-create the symbolic link to target the new installation directory.
Summary
The Failed at step EXEC / Permission denied
error occurs when systemd
is unable to read and/or execute the SDC_HOME/bin/streamsets
script as the user defined in the unit file. This is usually caused by one of two problems:
- The file or directory permissions are incorrect, preventing the user from reading or executing the file.
- On systems where SELinux is enabled, the SELinux context type label is one that disallows execution as a
systemd
service (admin_home_t
oruser_home_t
)
The error can generally be corrected by ensuring that the SDC_HOME
and SDC_HOME/bin
directories, as well as the SDC_HOME/bin/streamsets
file, have the correct permissions, and by ensuring that (on systems where SELinux is enabled) the SDC files do not reside in the /root
or /home
directory trees, per the methods provided in this document.