Skip to main content

Hi!

I have problem with size 'Request Data' in HTTP Client.

This step have ‘Required Fields’ in which comes xml different size.
And when the size for the 'Request Data' is exceeded, the pipeline doesn’t work.

 

Parameters:

HTTP Method: POST
Request Data: <?xml version="1.0" encoding="utf-8"?>
<request>
    <internal_id>${record:value("/xml_value")}</internal_id>
</request>
Default Request Content Type: application/xml
Data format: Text with Max Line Length → ${512 * 1024}

ERROR	Failed executing stage 'HTTPClient_02': com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request null	PipeRunner	*admin	0	ProductionPipelineRunnable-TEST4ed56edcc-3dbb-47c1-8295-bbec1ab0ebb6-TEST4
com.streamsets.pipeline.api.base.OnRecordErrorException: HTTP_01 - Error fetching resource. HTTP-Status: 400 Reason: Bad Request null
at com.streamsets.pipeline.stage.processor.http.HttpProcessor.processResponse(HttpProcessor.java:793)
at com.streamsets.pipeline.stage.processor.http.HttpProcessor.process(HttpProcessor.java:389)
at com.streamsets.pipeline.api.base.SingleLaneProcessor.process(SingleLaneProcessor.java:95)
at com.streamsets.pipeline.api.base.configurablestage.DProcessor.process(DProcessor.java:35)
at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:299)
at com.streamsets.pipeline.api.impl.CreateByRef.call(CreateByRef.java:40)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:244)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:311)
at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:221)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:864)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:908)
at com.streamsets.datacollector.runner.PipeRunner.acceptConsumer(PipeRunner.java:207)
at com.streamsets.datacollector.runner.PipeRunner.forEachInternal(PipeRunner.java:152)
at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:132)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:907)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:885)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runPollSource(ProductionPipelineRunner.java:609)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.run(ProductionPipelineRunner.java:400)
at com.streamsets.datacollector.runner.Pipeline.run(Pipeline.java:535)
at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:112)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.startInternal(StandaloneRunner.java:762)
at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:755)
at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:157)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

Please tell me what setting I need to change to increase the size for the field 'Request Data'.

I use StreamSets Data Collector 3.19.1.

@Eugene if you are referring to raising the limit at datacollector level then the name of the config is parser.limit and it is defined under sdc.properties file


@Sanjeev 

parser.limit 67108864

 

In this case I have not so big xml.

But sometimes it can be more than 5Mb, but in any case less than 67108864


@Eugene 

 

Under Data Format Tab, if you look at this field: “Max Record Length (chars) “

Change it to more than the max length of data per XML record and you should be good.


@saleempothiwala I have tried this option before. And also for type Text.

But result is the same

 


@Eugene please share your pipeline without any sensitive information.


Reply