In this article we will walkthroughs the steps involved to deploy engines in kubernetes using “Legacy Kubernetes” support in DataOps Platform. 

Note: Legacy Kubernetes integration is enabled only for some paid accounts. For more information, contact your StreamSets account team.

Once you have Legacy Kubernetes enabled for your org, you should see the following additional option.

Prerequisite:

• jq installed and of course Kubernates environment of your choice
• Create a namespace where you are expected to have the control-agent, StreamSets engines(SDC, Transformer for Spark etc.)

         Eg command: 

kubectl create namespace dataops

Step-1: Clone the git repo https://github.com/onefoursix/control-agent-k8s-deployment

Step-2: 

Set these variables at the top of the file deploy-control-agent-on-dataops-platform.sh:

ORG_ID=<Your DataOps Platform Org ID>
SCH_URL=<Your DataOps Platform URL> # for example: https://na01.hub.streamsets.com
CRED_ID=<Your API Credential CRED_ID>
CRED_TOKEN=<Your API Credential CRED_TOKEN>
KUBE_NAMESPACE=<YOUR-K8S-NAMESPACE> # The Namespace will be created if it does not exist

Step-3: Execute the script deploy-control-agent-on-dataops-platform.sh

Step-4: Verify if step-3 fails with below and if it is not, move to step-6.

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kuber
kind: ServiceAccount
netes.default.svc.cluster.local/api/v1/namespaces/dataops/secrets. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets is forbidden: User "system:serviceaccount:dataops:streamsets-agent" cannot create resource "secrets" in API group "" in the namespace "dataops".

Step-5:

Update secrets resource group in relevant control-agent-rbac*.yaml as below.

- apiGroups: b""]
  resources: <"secrets"]
  verbs: t"create", "get", "update"]

Step-6: If script in step-3 is successful, you would be able to see. control-agent in dataops > Legacy Kubernates > Provision Agents.

Step-7: Create deployment and mention Number of instance , data collector label and  Don’t forget to edit deployment and mention correct namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: datacollector-deployment
  namespace: dataops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: datacollector-deployment
  template:
    metadata:
      labels:
        app: datacollector-deployment
    spec:
      containers:
      - name: datacollector
        image: streamsets/datacollector:latest
        ports:
        - containerPort: 18630
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: PORT0
          value: "18630"

Step 8 : Once you activate this deployment you should see new SDC(s) under Engines