The Ideal approach via provisioning agent:
A Provisioning Agent is a containerized application that runs in a Kubernetes container orchestration framework. The agent communicates with Control Hub to automatically provision Data Collector containers in the Kubernetes cluster in which it runs. Provisioning includes deploying, registering, starting, scaling, and stopping the Data Collector containers. You can configure the Provisioning Agent to provision Data Collector containers enabled for Kerberos authentication. You can find steps here
However, if you want to avoid provisionig agent installation due to any reason. you can follow the below steps for reference.
This is the bare minimum setup required to deploy SDC in any k8s without control-agent.
Step 1: Create a lower-cased UUID and store it in a secret
$SDC_ID=`uuidgen | tr "`:upper:]" "r:lower:]"`
$echo "Generated sdc.id "${SDC_ID}
$kubectl create secret generic sdc-id --from-literal=sdc.id=${SDC_ID}
Step 2: Get an SDC auth token from Control Hub and store in k8s secret
SCH_AUTH_TOKEN= ${TOKEN}
You can get this from UI --> Engine --> Three vertical dots --> Data Collector component --> generate Token
$ kubectl create secret generic sdc-auth-token --from-literal=application-token.txt=${SDC_AUTH_TOKEN}
Step 3: Deploy ConfigMap for dpm.properties
$kubectl apply -f dpm-configmap.yaml
Below is my YAML, Please verify `dpm.base.url` that it point to correct platform
kind: ConfigMap
apiVersion: v1
metadata:
name: dpm-config
data:
dpm.properties: |-
dpm.enabled=true
dpm.base.url=https://eu02.hub.streamsets.com
dpm.registration.retry.attempts=5
dpm.security.validationTokenFrequency.secs=60
dpm.appAuthToken=@application-token.txt@
dpm.remote.control.job.labels=
dpm.remote.control.ping.frequency=5000
dpm.remote.control.events.recipient=jobrunner-app
dpm.remote.control.process.events.recipients=jobrunner-app,timeseries-app
dpm.remote.control.status.events.interval=60000
http.meta.redirect.to.sso=false
Step 4: Final SDC deployment
$kubectl apply -f sdc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sdc
labels:
app : sdc
spec:
replicas: 1
selector:
matchLabels:
app: sdc
template:
metadata:
labels:
app: sdc
spec:
containers:
- name : sdc
image: streamsets/datacollector:4.3.0
ports:
- containerPort: 18630
env:
- name: SDC_JAVA_OPTS
value: "-Xmx2048m -Xms2048m"
volumeMounts:
- name: sdc-id
mountPath: /data/sdc.id
subPath: sdc.id
- name: sdc-auth-token
mountPath: /etc/sdc/application-token.txt
subPath: application-token.txt
- name: dpm-config
mountPath: /etc/sdc/dpm.properties
subPath: dpm.properties
volumes:
- name: sdc-id
secret:
secretName: sdc-id
- name: sdc-auth-token
secret:
secretName: sdc-auth-token
- name: dpm-config
configMap:
name: dpm-config
Step 5: Verify SDC engine is available in UI