The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.
“ERROR A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”
Hi
The error basically means that the process runs out of memory, it could be due to a variety of things, it could be just that you are trying to use more memory that what you have allocated for it.
This is the kind of errors that usually require high troubleshooting, so if you are a customer, the best option would be to reach out to support.
Considering you are not a StreamSets customer, the following questions might help…
Thanks
Sorry for the late response. Hopefully answers below would help.
My first recommendation will be to increase the memory allocated, 1 GB might be too low, some stages are very memory intensive (depending on the data since it will be basically stored in memory).
Apart from that, 3.16 is pretty old version, I’ve myself fixed some memory issues since then, and I’m pretty sure there are a ton of improvements and features you can benefit from basically upgrading to a newer version, my recommendation will be to 5.x line if possible.
Thank you Alex. We will look into the recommendations.
We saw new error today. Any thoughts on this?
streamsets_server:pc1udauksfr01.abacus-us.com:18631 unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
That seems totally unrelated, please ask a separate question, you might be able to get better help.
The issue seems to be a validation error due to a null check, is that possible?
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
