Configuring Google Secret Manager Credential Store for data collector.

  • 1 October 2023
  • 1 reply
  • 76 views

Userlevel 4
Badge

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.

Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.

These are the secret created in GCP:-

 

The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.

  1. credentialStores=gcp
  2. credentialStore.gcp.config.project.id= <gcp-projectID>
  3. credentialStore.gcp.config.credentialsMode=jsonPath
  4. credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.json

Once done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:

Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}

Password => ${credential:get("gcp", "all@all", "akshay_sqlpassword?latest")}

Where, 

  • gcp → credentialStores value.
  • akshay_sqlusername?latest → secret name ? version of secret i.e latest.

https://docs.streamsets.com/portal/datacollector/5.0.x/help/datacollector/UserGuide/Configuration/CredentialStores.html#concept_xzg_3h1_hqb

 

Pipeline validation success.
Pipeline is retrieving the secret and fetching the data from mysql server.

 


1 reply

Hello Akshay,

If a GCP secret is a json, how do we read it in StreamSets pipeline? 

One of the customers I work with want to do that. 

They have successfully used secrets for username and password as you have shown in the example already. So, SDC setup is correct.

Reply