How to deploy Legacy kubernestes deployment in DataOps Platform ?

In this article we will walkthroughs the steps involved to deploy engines in kubernetes using “Legacy Kubernetes” support in DataOps Platform. 

Note: Legacy Kubernetes integration is enabled only for some paid accounts. For more information, contact your StreamSets account team.

Once you have Legacy Kubernetes enabled for your org, you should see the following additional option.

Some Prerequisites: 

• You have kubernetes cluster ready on any k8s provide and create new namespace.
• kubectl installed on your workstation to interact with k8s cluster

kubectl create namespace dataops

• Helm installed : You can follow instruction here 

Now let’s get started 

Step 1:  Clone streamsets helm chart repository

https://github.com/streamsets/helm-charts.git

Step 2: Go to control-agent directory

$cd helm-charts/stable/control-agent/

Step 3:  Edit the charts values.yaml 

•      OrgId
•      Control Hub Url
•      Control Agent Token ( Generate this token as mentioned in step 4) 
•      Tag 

i.e

# Default values for Streamsets Control Agent
image:
  repository: streamsets/control-agent
  tag: 4.0.0
  pullPolicy: Always
streamsets:
  orgId: 8c2c652f-e3d9-11eb-9fb3-b974ac4c3f67
  crdEnabled: false
  api:
    url: https://na01.hub.streamsets.com/
    token: ${TOKEN}
rbac:
  enabled: true
krb:
  enabled: false
  encryptionTypes: <encryption types to use when creating a keytab for service principal>
  containerDn: <distinguished name of the container under which new principals will be created>
  ldapUrl: <URL of the LDAP service provider>
  adminPrincipal: <user account which has privileges to create, search and destroy service principals>
  adminKey: <secret key for the above account>
  realm: <the realm of the organization>
  kdcType: < AD | MIT >
##
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}
nodeSelector: {}

Step 4: Generate provisioning agent token from UI and copy 

Step 5: Deploy the control agent in desired namespace

❯ helm  install --namespace dataops streamsets/control-agent --values values.yaml --generate-name


NAME: control-agent-1644298742
LAST DEPLOYED: Tue Feb  8 11:09:07 2022
NAMESPACE: dataops
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The agent has been successfully installed.
Visit https://na01.hub.streamsets.com//sch/provisioning/deployments to continue setup.

Step 6: Verify the control agent is visible

Step 7 : Create deployment and mention Number of instance , data collector label and  Don’t forget to edit deployment and mention correct namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: datacollector-deployment
  namespace: dataops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: datacollector-deployment
  template:
    metadata:
      labels:
        app: datacollector-deployment
    spec:
      containers:
      - name: datacollector
        image: streamsets/datacollector:latest
        ports:
        - containerPort: 18630
        env:
        - name: HOST
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: PORT0
          value: "18630"

Step 8 : Once you activate this deployment you should see two new SDC under Engines 

 

Step 9: You are all set, you can design new pipeline and run job using this SDC’s

ref:  here