Question

How to initiate the emr cluster through streamset

  • 2 November 2022
  • 6 replies
  • 54 views
P
Rank

How to initiate the emr cluster through streamset

6 replies

Rishi
Rank
Userlevel 4
Badge

Hi @Priya151997 I am assuming you are talking in-regards to transformer engine for EMR cluster type.

Please check out the following doc : https://docs.streamsets.com/portal/transformer/latest/help/transformer/Clusters/EMR.html#concept_vlq_ht1_wkb 

P
Rank

Yes but there is nothing mention how i can trigger my emr cluster if any event happend in aws s3

Rishi
Rank
Userlevel 4
Badge

@Priya151997 With above mentioned document, it has reference of how to provision the new emr  cluster. So each time you trigger this transformer job. It will create new cluster and run job on that cluster.

Are you looking for this or something else. Please share your complete use-case details. 

 

P
Rank

Hi,thanks for the help actually I am new in streamsets.

actually in my usecase i want to trigger (launch)emr cluster using python script.can this possible in data collector?

saleempothiwala
Rank
Userlevel 4
Badge

@Priya151997

You can use HTTP Client processor to call any APIs so if you can start a new cluster using APIs then you can definitely use the HTTP Client processor to do it from within a StreamSets pipeline. This is from SDC pipeline.

 

For Transformer, when you start your pipeline, the underlying EMR cluster that you have configured should be initiated automatically.

P
Rank

Hi All,

We are not going to use any EMR cluster within Streamsets,I need to know how to initiate EMR cluster which will launch in my AWS account for further job executions.

this will be only triggering events just to start the cluster.

and want to know for this purpose only data collector will be enough.

again want to mention that EMR cluster will hosted as an individual service on AWS account and not with Streamsets for using in data transformer.

Reply


Terms & Conditions