Skip to main content

Hi guys, can anyone help me?
I have this pipeline that gets some ids from bigquery and injects them in a api but I keep receiving this error "Too Many Requests" even in the first record.
I've configured the http to wait 1000 ms between the requests and it supposed to be more than enough to respect the API limits.

 

@miguelrezende7rox 

Hi 

Can you please set the rate limit as ‘0’and batch wait time ‘100000 ms’ and  rerun the pipeline after reset to origin and let me know if it helps.

 

Thanks & Regards

Bikram_

@miguelrezende7rox

Hi 

Can you please set the rate limit as ‘0’and batch wait time ‘100000 ms’ and  rerun the pipeline after reset to origin and let me know if it helps.

 

Thanks & Regards

Bikram_

Thanks for answering but it didnt help. I believe when I set rate limit to 0 I dont respect the API limit that supposed to be 600ms.

I also tried rate limit as 800 ms (with some margin) and the batch wait time ‘100000 ms’ but still got 429 in some records.

I have the impression that Streamsets requests the API more than once for the same record.

 

 

 

 

 

@miguelrezende7rox have you tinkered with the number of records in the batch. Try reducing that to a smaller value and then increase it until you get an optimum one.

@miguelrezende7rox have you tinkered with the number of records in the batch. Try reducing that to a smaller value and then increase it until you get an optimum one.

Even in a optimum level I still operate way below the api request limit. But It helped. Thank your for answering.

@miguelrezende7rox glad it helped.

Reply


Terms & Conditions