Skip to main content

The Ideal approach via provisioning agent:

Provisioning Agent is a containerized application that runs in a Kubernetes container orchestration framework. The agent communicates with Control Hub to automatically provision Data Collector containers in the Kubernetes cluster in which it runs. Provisioning includes deploying, registering, starting, scaling, and stopping the Data Collector containers. You can configure the Provisioning Agent to provision Data Collector containers enabled for Kerberos authentication. You can find steps here

 

However, if you want to avoid provisionig agent installation due to any reason. you can follow the below steps for reference.

This is the bare minimum setup required to deploy SDC in any k8s without control-agent.

 

Step 1: Create a lower-cased UUID and store it in a secret
 

$SDC_ID=`uuidgen | tr "`:upper:]" "r:lower:]"`
$echo "Generated sdc.id "${SDC_ID}
$kubectl create secret generic sdc-id --from-literal=sdc.id=${SDC_ID}



Step 2: Get an SDC auth token from Control Hub and store in k8s secret
 

SCH_AUTH_TOKEN= ${TOKEN}


You can get this from UI --> Engine --> Three vertical dots --> Data Collector component --> generate Token
 

$ kubectl create secret generic sdc-auth-token --from-literal=application-token.txt=${SDC_AUTH_TOKEN}


Step 3: Deploy ConfigMap for dpm.properties
 

$kubectl apply -f dpm-configmap.yaml



Below is my YAML, Please verify `dpm.base.url`  that it point to correct platform

kind: ConfigMap
apiVersion: v1
metadata:
  name: dpm-config
data:
    dpm.properties: |-
      dpm.enabled=true
      dpm.base.url=https://eu02.hub.streamsets.com
      dpm.registration.retry.attempts=5
      dpm.security.validationTokenFrequency.secs=60
      dpm.appAuthToken=@application-token.txt@
      dpm.remote.control.job.labels=
      dpm.remote.control.ping.frequency=5000
      dpm.remote.control.events.recipient=jobrunner-app
      dpm.remote.control.process.events.recipients=jobrunner-app,timeseries-app
      dpm.remote.control.status.events.interval=60000
      http.meta.redirect.to.sso=false


Step 4: Final SDC deployment
 

 $kubectl apply -f sdc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sdc
  labels:
    app : sdc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sdc
  template:
    metadata:
      labels:
        app: sdc
    spec:
      containers:
      - name : sdc
        image: streamsets/datacollector:4.3.0
        ports:
        - containerPort: 18630
        env:
        - name: SDC_JAVA_OPTS
          value: "-Xmx2048m -Xms2048m"
        volumeMounts:
        - name: sdc-id
          mountPath: /data/sdc.id
          subPath: sdc.id
        - name: sdc-auth-token
          mountPath: /etc/sdc/application-token.txt
          subPath: application-token.txt
        - name: dpm-config
          mountPath: /etc/sdc/dpm.properties
          subPath: dpm.properties
      volumes:
      - name: sdc-id
        secret:
          secretName: sdc-id
      - name: sdc-auth-token
        secret:
          secretName: sdc-auth-token
      - name: dpm-config
        configMap:
          name: dpm-config



Step 5: Verify SDC engine is available in UI

Be the first to reply!

Reply