Question

GCP: credential:get function - Are there limitations where this function can be used?

  • 10 November 2023
  • 6 replies
  • 49 views
D
Rank

Hello, 

One of my customer is using StresmSets to access secrets saved in GCP. 

The SDC setup and GCP secrets setup is correct. For example,  I have used secrets like below successfully in JDBC producer stage’s credential tab and pipeline works fine, picks up latest version of the secret as expected. 

${credential:get("gcp", "group@org", "dbuser?latest")}

${credential:get("gcp", "group@org", "dbpass?latest")}

Now, they want to read a secret for which a json file was uploaded as value, and they want to extract credential information from this json. 

 

I tried to use credential:get in an Expression evaluator but it gave an error. (was planning to add more processors after this to extract userid and password from the json returned by credential:get) 

How can i use secret like this in my StreamSets pipeline?

 

Thanks!

 

~Dhanashri

6 replies

Bikram
Rank
Userlevel 5
Badge +1

@Dhanashri_Bhate 

If json is a file which contains the secrets in it then you can keep the file in s3 or local FS , by using expression evaluator , fetch the data from json file contains secret in it. 

 

The pipeline should look like this.

 

Directory -> JSON Parser -> Field Expression Evaluator

 

Please let me know if it helps.

D
Rank

Hi @Bikram , unfortunately this does not help. I have used json files often and parsed them using the stages you mentioned.

The question is not about how to parse a json (the secret could be any other type of file too!); the question is : How do I access the json that is saved as a secret in GCP?

To read GCP secrets we have credential:get function available, but that only works when I use it to specify credentials for a connection, in origin/destination stage configuration. This place does not help since there is a limitation on how much processing I can to in that expression box. Do we have any pocessor stages where “credential:get” can be used?

The customer uses GCP cloud, so saving the file somewhere else is not an option I can propose.

Also, they are planning to use a json with multiple user creds or to save the creds of different environments (dev/qa/prod) in one json that they upload as a secret (instead of creating a separate secret for each). 

 

~Dhanashri
D
Rank

Tried a few other options to try and extract the information in the credentials field itself but that did not work. Parsing the information in the secret needs some string function or regex etc. But, StreamSets does not allow any modification in the string if we are using ‘credential:get()’ in property.

Is there any other way? Is there a feature pending for this? or do we need to tell users to just use simple single values in a secret like only the userid and password?

~Dhanashri
Bikram
Rank
Userlevel 5
Badge +1

@Dhanashri_Bhate 

There might be some limitations in it .Is it possible to create custom function for it and it can be used in expression evaluator.

Just create functions and return the user and password from there.

you can give a try .

 

D
Rank

@Bikram I have not used custom functions before. Can you please share more information on this?

Did not find it in the documentation.

~Dhanashri
Bikram
Rank
Userlevel 5
Badge +1

@Dhanashri_Bhate 

Please refer the below article  and hope it will help in your case study.

https://streamsets.com/blog/create-custom-expression-language-function-streamsets-data-collector/

Reply


Terms & Conditions