"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux

Problem

When starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).

systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied

Cause

This error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled).

Solution

Step 1:  Examine the Service Configuration

Before looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Examine this file and look for the following lines to find the user and group.  The below example shows the default (and recommended) values:

[Service]

User=sdc

Group=sdc

The SDC installation directory can be found a few lines further down in this file.  Again, this example shows the default setting:

Environment=SDC_HOME=/opt/streamsets-datacollector

Ensure that the values configured in the unit file are correct.

On systems where SELinux is enabled, the path for the SDC installation directory must not begin with /home or /root, as SELinux forbids system services from running out of these directories (see Step 3: Validate SELinux Context for more information).

Step 2: Validate Permissions

The user which runs the SDC service must have access to read and execute the SDC_HOME/bin/streamsets file (where SDC_HOME represents the SDC installation directory, as validated in Step 1 above).  Due to the nature of directory objects and permissions, this also requires that the user have read and execute permissions on the SDC_HOME directory itself, and the SDC_HOME/bin directory as well.

By default, both directories and the script file itself have the following permissions:

• owner: read, write, execute
• group: read, write, execute
• others: read, execute

This permission set is frequently described in Linux by the text rwxrwxr-x or the number 775.

Permissions on a directory or file object can be examined with the stat path command (where path represents the file or directory path of the object to be examined)

For example, in a default installation (where SDC was installed in the directory /opt/streamsets-datacollector) the command to examine the installation directory is:

stat /opt/streamsets-datacollector

and it returns output similar to the following:

# stat /opt/streamsets-datacollector

File: /opt/streamsets-datacollector

Size: 4096    Blocks: 8    IO Block: 4096   directory

Device: ca04h/51716d   Inode: 310384541   Links: 18

Access: (0775/drwxrwxr-x)  Uid: ( 1000/ec2-user)   Gid: ( 1000/ec2-user)

The file permissions are shown in the Access field from these results, which is highlighted in the example above.  The permissions shown in this example “(0775/drwxrwxr-x)” are the correct permissions for these objects.  Note that the “d” after the slash in this field indicates this is a directory, and when running stat on the SDC_HOME/bin/streamsets file it will show a “-” there (indicating a regular file) instead.

There is an additional special case where the installation directory may show an “l” after the slash in the Access field instead of a “d” (for example, “(0775/lrwxrwxr-x)”.  This is also valid, but it indicates that this path is a “symbolic link” and that the actual installation files are located elsewhere in the filesystem.  More information on symbolic links can be found in the section, Installation Directory as a Symbolic Link later in this document.

If any of these objects does not have the correct permissions, the permissions can be set with the “sudo chmod 775 path” command (again, where path represents the file or directory path of the object to be modified).

For example, the commands for the two directories and the script file in a default environment are:

sudo chmod 775 /opt/streamsets-datacollector

sudo chmod 775 /opt/streamsets-datacollector/bin

sudo chmod 775 /opt/streamsets-datacollector/bin/streamsets

Step 3: Validate SELinux Context

On systems where Security Enhanced Linux (SELinux) is enabled, the output of the stat command will include an additional field of importance (highlighted in the example below) named “Context”:

# stat /opt/streamsets-datacollector

File: /opt/streamsets-datacollector

Size: 4096    Blocks: 8    IO Block: 4096   directory

Device: ca04h/51716d   Inode: 310384541   Links: 18

Access: (0775/drwxrwxr-x)  Uid: ( 1000/ec2-user)   Gid: ( 1000/ec2-user)

Context: unconfined_u:object_r:usr_t:s0

The “Context” field describes the “SELinux context”, which is a set of labels used by SELinux to restrict access to system resources.

The SELinux context includes a user label (“unconfined_u” in the example above), a role label (“object_r” in the example above), a type label (“usr_t” in the example above), and a level (“s0” in the example above).

The type label is noteworthy because it is initially set by SELinux based on the location of the object, and this is the reason why the SDC installation directory must not reside in the /home or /root directory trees on systems where SELinux is enabled.  Objects which reside in the /home directory tree are assigned the type of “user_home_t”, and objects which reside in the /root directory tree are assigned the type of “admin_home_t”.  SELinux does not permit files of these types to be executed as system services.

As such, if the SDC installation directory is in the /root or /home directory trees on a system where SELinux is enabled, attempting to start the sdc service will result in a Permission Denied error, as in this example:

systemd[13801]: sdc.service: Failed at step EXEC spawning /root/streamsets-datacollector-5.5.0/bin/streamsets: Permission denied

Examining /root/streamsets-datacollector-5.5.0/bin/streamsets in this case will show the following:

# stat /root/streamsets-datacollector-5.5.0/bin/streamsets

File: /root/streamsets-datacollector-5.5.0/bin/streamsets

Size: 3493   Blocks: 8   IO Block: 4096   regular file

Device: ca04h/51716d   Inode: 117440927   Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 990/ sdc)  Gid: ( 990/ sdc)

Context: unconfined_u:object_r:admin_home_t:s0

The type label is admin_home_t, which disallows the script from being executed as a systemd service.

In most cases where SDC has been installed in the /root or /home directory tree, the simplest solution is to follow these steps:

1. Re-extract the SDC files from the tarball in a new location outside of the /home or /root directory tree.
2. Modify the unit file (/etc/systemd/system/sdc.service) to use the new location for SDC_HOME.

Alternately, this method can also be used:

1. Move the existing SDC installation directory (using the mv command) to a new location outside the /home or /root directory trees
2. Correct the SELinux context for the SDC files with the command:  restorecon -R -v NEW_PATH (where NEW_PATH  represents the new installation directory path).
   1. This is necessary because SELinux does not update the context automatically when the files are moved.
3. Modify the unit file (/etc/systemd/system/sdc.service) to use the new installation directory for SDC_HOME.

Installation Directory as a Symbolic Link

In the case described in the section "Step 3: Validate SELinux Context" above, the fact that the SDC installation directory resides in the /root directory tree is evident.  However, the use of symbolic links can mask the location of the installation directory.

A symbolic link is a special kind of file which references another location in the filesystem, known as the "target" of the link.  A symbolic link's target can be a single file, but it can also be a directory, and when a symbolic link is targeted on a directory, the path of the link can be used to reference files in that directory (and its subdirectories).

As an example, consider a symbolic link /opt/streamsets-datacollector, whose target is the directory /root/streamsets-datacollector-5.5.0, where the actual SDC files are extracted.  

Examining /opt/streamsets-datacollector in this case demonstrates that it is a symbolic link:

# stat /opt/streamsets-datacollector

File: /opt/streamsets-datacollector -> /root/streamsets-datacollector-5.5.0

Size: 36    Blocks: 0    IO Block: 4096   symbolic link

Device: ca04h/51716d   Inode: 12887   Links: 1

Access: (0777/lrwxrwxrwx)  Uid: ( 0/ root)   Gid: ( 0/ root)

Context: unconfined_u:object_r:usr_t:s0

Note the highlighted sections in the example output above:

• The File: field on the first line of the output shows an arrow ("->"), followed by the destination of the symbolic link (in this case, /root/streamsets-datacollector-5.5.0)
• The file type is shown as "symbolic link".
• The Access: field shows that the first character of the permission string is "l" (indicating a symbolic link) rather than d (indicating a directory).

In this case, the system can find the streamsets script at /opt/streamsets-datacollector/bin/streamsets

# ls -l /opt/streamsets-datacollector/bin/streamsets

-rwxr-xr-x. 1 sdc sdc 3493 Apr 18 18:01 /opt/streamsets-datacollector/bin/streamsets

However, configuring the path /opt/streamsets-datacollector/ for SDC_HOME in the service unit file /etc/systemd/system/sdc.service and attempting to start the service results in the following error:

systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied

Examining /opt/streamsets-datacollector/bin/streamsets shows the reason that permission is denied:  the SELinux context type label is "admin_home_t":

  # stat /opt/streamsets-datacollector/bin/streamsets

File: /opt/streamsets-datacollector/bin/streamsets

Size: 3493   Blocks: 8   IO Block: 4096   regular file

Device: ca04h/51716d   Inode: 117440927   Links: 1

Access: (0755/-rwxr-xr-x)  Uid: (  990/ sdc)   Gid: (  990/ sdc)

Context: unconfined_u:object_r:admin_home_t:s0

This is because the SELinux context type label is set based on the actual location of the file, which is /root/streamsets-datacollector-5.5.0/bin/streamsets

In this case, the solution is similar to that in Step 3: Validate SELinux Context above; the SDC files must be located in a directory outside of /home or /root.

In most cases, the simplest solution is to follow these steps:

1. Re-extract the SDC files in a new location outside of the /home or /root directory tree
2. Re-create the symbolic link to target the new installation directory.

Alternately, this method can also be used:

1. Move the existing SDC installation directory (using the mv command) to a new location outside the /home or /root directory trees
2. Update the SELinux context for the SDC files by running the following command:  restorecon -R -v NEW_PATH (where NEW_PATH  represents the new installation directory path).
   1. This is necessary because SELinux does not update the context automatically when the files are moved.
3. Re-create the symbolic link to target the new installation directory.

Summary

The Failed at step EXEC / Permission denied error occurs when systemd is unable to read and/or execute the SDC_HOME/bin/streamsets script as the user defined in the unit file.  This is usually caused by one of two problems:

1. The file or directory permissions are incorrect, preventing the user from reading or executing the file.
2. On systems where SELinux is enabled, the SELinux context type label is one that disallows execution as a systemd service (admin_home_t or user_home_t)

The error can generally be corrected by ensuring that the SDC_HOME and SDC_HOME/bin directories, as well as the SDC_HOME/bin/streamsets file, have the correct permissions, and by ensuring that (on systems where SELinux is enabled) the SDC files do not reside in the /root or /home directory trees, per the methods provided in this document.