Python SDK Error 503 Server Error ?

Issue:

The user has a proxy configured in their environment. Within their local network they have an SDC and it is registered with SCH on-cloud. The client tries to retrieve the SDC id with the following code:

import os
from streamsets.sdk import ControlHub, DataCollector

# setting proxy proxy = “<proxy ip>:<port>”
os.environ['HTTP_PROXY'] = proxy
os.environ['HTTPS_PROXY'] = proxy

control_hub = ControlHub('https://cloud.streamsets.com', username=<your username>, password=<your password>)
data_collector = DataCollector(<your datacollector url>, control_hub=control_hub)
data_collector.id

But they get the following error:

data_collector = DataCollector(server_url='*******', username='*******', password='*******')
data_collector.id
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/software/python/python-3.8.0/lib/python3.8/site-packages/streamsets/sdk/sdc.py", line 136, in id
sdc_id_command = self.api_client.get_sdc_id()
File "streamsets/sdk/sdc_api.pyx", line 776, in sdk.sdc_api.ApiClient.get_sdc_id
File "streamsets/sdk/sdc_api.pyx", line 1198, in sdk.sdc_api.ApiClient._get
File "streamsets/sdk/sdc_api.pyx", line 1228, in sdk.sdc_api.ApiClient._handle_http_error
File "/software/python/python-3.8.0/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http://*********/rest/v1/system/info/id

Solution:

Setting the environment variable NO_PROXY and adding the SDC IP to it solved the problem. The user must add a proxy exception so that the Python SDK code when connecting to the Internet through the proxy, can reach the SCH in the cloud and connect to the customer's local network where the SDC is without issues.

The code should look like this:

import os
from streamsets.sdk import ControlHub, DataCollector

# setting proxy
proxy = “<proxy ip>:<port>”
os.environ['HTTP_PROXY'] = proxy
os.environ['HTTPS_PROXY'] = proxy

# Proxy exception: adding SDC IP address to the exclusion list with NO_PROXY
os.environ['NO_PROXY'] = '<SDC IP ADDRESS>'

control_hub = ControlHub('https://cloud.streamsets.com', username=<your username>, password=<your password>)
data_collector = DataCollector(<your datacollector url>, control_hub=control_hub)
data_collector.id

Link to documentation about how to set up NO_PROXY variable: http://www.gnu.org/software/wget/manual/html_node/Proxies.html

Note that in case the user has more than one SDC, they should also add their IP addresses to the NO_PROXY list or add the entire subnet like this:

NO_PROXY=master.hostname.example.com,10.1.0.0/16,172.30.0.0/1