Skip to main content

Hello, 

I comming with question about SDC 3.16 compatibility.

We can see in support matrix for MapR environment the latest supported version for MapR core 6.1.0. Since support matrix is from 2019, when MapR 6.1.1 wasn’t even released, I’d like to ask whether version of SDC 3.16.is compatibile with  MapR 6.1.1  (and MEP 6.3.6)

As we can see in documentation for MapR setup ( https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Installation/MapR-Prerequisites.html,) we can run Set Up command to setup configuration files and create requeired symbolic links. 

Is it possible to perform these steps even if version of MapR is 6.1.1? 

 

Thanks

Michal K

Hi @michal.kertes  Assuming there are no major change between 6.1.1 and 6.1.0 and it’s just the bugfix version then it should be fine and it should work with SDC( You can try follow the same instruction) . However it’s not tested against these new versions.

 

Hello, 

thanks for your answer. We performed mapr-setup as per documentation, but unfortunatelly we are facing with following entries in sdc.log: 


#####
4-9963-6352a0fc023c-tsystems_maprfs_test] ]stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation
com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0702 - Pipeline initialization error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation
        at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:106)
        at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:75)
        at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:726)
        at com.streamsets.datacollector.execution.AbstractRunner.lambda$scheduleForRetries$0(AbstractRunner.java:363)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.lambda$call$0(SafeScheduledExecutorService.java:226)
        at com.streamsets.datacollector.security.GroupsInScope.execute(GroupsInScope.java:34)
        at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:222)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.streamsets.datacollector.metrics.MetricSafeScheduledExecutorService$MetricsTask.run(MetricSafeScheduledExecutorService.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.UserGroupInformation
        at com.streamsets.datacollector.security.MapRLoginUgiProvider.getLoginUgi(MapRLoginUgiProvider.java:48)
        at com.streamsets.datacollector.security.HadoopSecurityUtil.getLoginUser(HadoopSecurityUtil.java:35)
        at com.streamsets.pipeline.lib.hdfs.common.HdfsBaseConfigBean.validateHadoopFS(HdfsBaseConfigBean.java:286)
        at com.streamsets.pipeline.stage.origin.hdfs.HdfsSourceConfigBean.init(HdfsSourceConfigBean.java:40)
        at com.streamsets.pipeline.stage.origin.hdfs.HdfsSource.init(HdfsSource.java:59)
        at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:48)
        at com.streamsets.pipeline.api.base.configurablestage.DStage.init(DStage.java:36)
        at com.streamsets.datacollector.runner.StageRuntime.lambda$init$0(StageRuntime.java:220)
        at com.streamsets.datacollector.util.LambdaUtil.withClassLoaderInternal(LambdaUtil.java:148)
        at com.streamsets.datacollector.util.LambdaUtil.withClassLoader(LambdaUtil.java:44)
        at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:218)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:109)
        at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:44)
        at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:396)
        at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:301)
        at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:98)
        ... 13 more
#####

Do you have any idea how to solve it? We checked the libraries and symbolic link were created successfully for new version of MapR beside old ones. We set also correct ownership and right, but still the same.

 

Thank you!

 

michal.kertes​​​​​​

Reply


Terms & Conditions