Hi guys, can anyone help me?
I have this pipeline that gets some ids from bigquery and injects them in a api but I keep receiving this error "Too Many Requests" even in the first record.
I've configured the http to wait 1000 ms between the requests and it supposed to be more than enough to respect the API limits.
Best answer by saleempothiwala
+1Hi
Can you please set the rate limit as ‘0’and batch wait time ‘100000 ms’ and rerun the pipeline after reset to origin and let me know if it helps.
Thanks & Regards
Bikram_
Hi
Can you please set the rate limit as ‘0’and batch wait time ‘100000 ms’ and rerun the pipeline after reset to origin and let me know if it helps.
Thanks & Regards
Bikram_
Thanks for answering but it didnt help. I believe when I set rate limit to 0 I dont respect the API limit that supposed to be 600ms.
I also tried rate limit as 800 ms (with some margin) and the batch wait time ‘100000 ms’ but still got 429 in some records.
I have the impression that Streamsets requests the API more than once for the same record.
Even in a optimum level I still operate way below the api request limit. But It helped. Thank your for answering.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
