Streamsets solrcloud on CDH 5.7 unable to connect to Solr

Question

I am using streamsets on CDH version 5.7.0

A sample workflow to load a file from HDFS(origin) and create records on Solr (destination). It is failing on validation - SOLR_03 - Could not connect to the Solr instance: org.apache.solr.common.cloud.ZooKeeperException: : ZooKeeper Connection String.

However, from the logs, I see that it is able to connect to zookeeper, but, somehow closes the session after calling the /live_nodes zookeeper api.

2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.ConnectionManager: Client is connected to ZooKeeper

2016-06-03 16:30:21,336 INFO org.apache.solr.common.cloud.SolrZkClient: Using default ZkACLProvider

2016-06-03 16:30:21,338 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,42949673777,0  request:: '/clusterstate.json,F  response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788} 

2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,42949673777,0  request:: '/aliases.json,F  response:: s{38654846789,38654846789,1464989414814,1464989414814,0,0,0,0,0,0,38654846789} 

2016-06-03 16:30:21,339 INFO org.apache.solr.common.cloud.ZkStateReader: Updating cluster state from ZooKeeper... 

2016-06-03 16:30:21,339 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 3,3  replyHeader:: 3,42949673777,0  request:: '/clusterstate.json,T  response:: s{38654846788,38654846788,1464989414801,1464989414801,0,0,0,0,0,0,38654846788} 

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 4,8  replyHeader:: 4,42949673777,-101  request:: '/live_nodes,T  response:: v{} 

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ZooKeeper: Closing session: 0x355187691370039

2016-06-03 16:30:21,340 DEBUG org.apache.zookeeper.ClientCnxn: Closing client for session: 0x355187691370039

2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Reading reply sessionid:0x355187691370039, packet:: clientPath:null serverPath:null finished:false header:: 5,-11  replyHeader:: 5,42949673778,0  request:: null response:: null

2016-06-03 16:30:21,352 DEBUG org.apache.zookeeper.ClientCnxn: Disconnecting client for session: 0x355187691370039

2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ZooKeeper: Session: 0x355187691370039 closed

2016-06-03 16:30:21,352 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down

2016-06-03 16:30:21,352 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete

2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'

2016-06-03 16:30:21,352 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returned a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down

2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.execution.manager.standalone.StandaloneAndClusterPipelineManager: Status of previewer with id: '0bec83af-c152-4519-98f4-201394bebc36' changed to status: 'INVALID'

2016-06-03 16:30:21,353 INFO com.streamsets.pipeline.cluster.ControlChannel: Producer complete

2016-06-03 16:30:21,353 DEBUG com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Returning private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib'

2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask: Could not return a private ClassLoader for 'streamsets-datacollector-cdh_5_7-lib', active private ClassLoaders='0'

2016-06-03 16:30:21,353 WARN com.streamsets.datacollector.runner.Pipeline: Exception thrown during pipe 'StagePipe[instance='com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01' input='[]' output='[com_streamsets_pipeline_stage_origin_hdfs_cluster_ClusterHdfsDSource_01OutputLane14649766295360::s]']' destroy: java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

java.lang.RuntimeException: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:448)

at com.streamsets.datacollector.runner.preview.PreviewStageLibraryTask.releaseStageClassLoader(PreviewStageLibraryTask.java:92)

at com.streamsets.datacollector.creation.StageBean.releaseClassLoader(StageBean.java:60)

at com.streamsets.datacollector.runner.StageRuntime.destroy(StageRuntime.java:203)

at com.streamsets.datacollector.runner.StagePipe.destroy(StagePipe.java:275)

at com.streamsets.datacollector.runner.Pipeline.destroy(Pipeline.java:187)

at com.streamsets.datacollector.runner.preview.PreviewPipeline.destroy(PreviewPipeline.java:71)

at com.streamsets.datacollector.execution.preview.sync.SyncPreviewer.start(SyncPreviewer.java:205)

at com.streamsets.datacollector.execution.preview.async.AsyncPreviewer$2.call(AsyncPreviewer.java:96)

at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:233)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalStateException: Object has already been returned to this pool or is invalid

at org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:484)

at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.releaseStageClassLoader(ClassLoaderStageLibraryTask.java:442)

... 15 more

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager is shutting down

2016-06-03 16:30:21,353 DEBUG org.apache.http.impl.conn.PoolingClientConnectionManager: Connection manager shut down

So, to verify, I connected to zookeeper and queried the live_nodes:

[root@hadoop3 ~]# zookeeper-client

Connecting to localhost:2181

Welcome to ZooKeeper!

[zk: localhost:2181(CONNECTED) 5] ls /solr

[configs, overseer, aliases.json, live_nodes, collections, overseer_elect, clusterstate.json, solr.xml, zkdtsm, clusterprops.json, token]

[zk: localhost:2181(CONNECTED) 6] ls /solr/live_nodes

[hadoop3:8983_solr, hadoop4:8983_solr, hadoop2:8983_solr, hadoop5:8983_solr]

Not sure, why the response is empty when streamsets calls the API and why it complains about "Zookeeper Connection String" when it is able to connect... Any help?

Answer 1

Add /solr to your ZooKeeper URL, so it looks like : host:port/solr.