solr - SolrCloud with SSL and Basic Authentication -
is possible configure solrcloud ssl , basic authentication?
i have configured 3 nodes of solr in solrcloud ssl using this: https://cwiki.apache.org/confluence/display/solr/enabling+ssl
and have added authentication , authorization following this: https://cwiki.apache.org/confluence/display/solr/basic+authentication+plugin, https://cwiki.apache.org/confluence/display/solr/rule-based+authorization+plugin
when ssl enabled works.
when authentication + authorization enabled works
when both enabled following stacktrace during startup:
2016-06-01 17:19:41.933 info (overseerstateupdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.o.zkstatewriter going update_collection /collections/testowa/state.json version: 1350 2016-06-01 17:19:41.935 info (zkcallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.zkstatereader cluster state change: [watchedevent state:syncconnected type:nodedatachanged path:/collections/testowa/state.json] collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-01 17:19:41.937 info (zkcallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.zkstatereader updating data [testowa] [1350] [1351] 2016-06-01 17:19:43.557 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.shardleaderelectioncontext enough replicas found continue. 2016-06-01 17:19:43.557 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.shardleaderelectioncontext may new leader - try , sync 2016-06-01 17:19:43.557 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.syncstrategy sync replicas https://172.30.92.66:8983/solr/testowa_shard1_replica3/ 2016-06-01 17:19:43.561 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.peersync peersync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr start replicas=[https://172.30.182.43:8983/solr/testowa_shard1_replica1/, https://172.30.182.44:8983/solr/testowa_shard1_replica2/] nupdates=100 2016-06-01 17:19:44.580 warn (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.peersync peersync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr exception talking https://172.30.182.44:8983/solr/testowa_shard1_replica2/, failed org.apache.solr.client.solrj.impl.httpsolrclient$remotesolrexception: error server @ https://172.30.182.44:8983/solr/testowa_shard1_replica2: expected mime type application/octet-stream got text/html. <html> <head> <meta http-equiv="content-type" content="text/html;charset=utf-8"/> <title>error 401 unauthorized request, response code: 401</title> </head> <body><h2>http error 401</h2> <p>problem accessing /solr/testowa_shard1_replica2/get. reason: <pre> unauthorized request, response code: 401</pre></p> </body> </html> @ org.apache.solr.client.solrj.impl.httpsolrclient.executemethod(httpsolrclient.java:545) @ org.apache.solr.client.solrj.impl.httpsolrclient.request(httpsolrclient.java:241) @ org.apache.solr.client.solrj.impl.httpsolrclient.request(httpsolrclient.java:230) @ org.apache.solr.client.solrj.solrclient.request(solrclient.java:1219) @ org.apache.solr.handler.component.httpshardhandler$1.call(httpshardhandler.java:198) @ org.apache.solr.handler.component.httpshardhandler$1.call(httpshardhandler.java:163) @ java.util.concurrent.futuretask.run(futuretask.java:277) @ java.util.concurrent.executors$runnableadapter.call(executors.java:522) @ java.util.concurrent.futuretask.run(futuretask.java:277) @ org.apache.solr.common.util.executorutil$mdcawarethreadpoolexecutor.lambda$execute$0(executorutil.java:229) @ org.apache.solr.common.util.executorutil$mdcawarethreadpoolexecutor$$lambda$3.000000003c022970.run(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1153) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:628) @ java.lang.thread.run(thread.java:785) 2016-06-01 17:19:44.582 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.peersync peersync: core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr done. sync failed 2016-06-01 17:19:44.583 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.syncstrategy leader's attempt sync shard failed, moving next candidate 2016-06-01 17:19:44.585 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.shardleaderelectioncontext there may better leader candidate - going recovery 2016-06-01 17:19:44.585 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.electioncontext canceling election /collections/testowa/leader_elect/shard1/election/168013962670440512-core_node1-n_0000000882 2016-06-01 17:19:44.588 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.shardleaderelectioncontextbase no version found ephemeral leader parent node, won't remove previous leader registration. 2016-06-01 17:19:44.590 info (updateexecutor-2-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.defaultsolrcorestate running recovery 2016-06-01 17:19:44.592 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.leaderelector joined leadership election path: /collections/testowa/leader_elect/shard1/election/168013962670440512-core_node1-n_0000000885 2016-06-01 17:19:44.594 info (recoveryexecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.recoverystrategy starting recovery process. recoveringafterstartup=true 2016-06-01 17:19:44.597 info (recoveryexecutor-3-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.recoverystrategy ###### startupversions=[[1535485004938739712, 1535485004934545409, 1535485004934545408, 1535485004930351104, 1535485004926156801, 1535485004926156800, 1535485004919865346, 1535485004919865345, 1535485004919865344, 1535485004914622464, 1535485004908331010, 1535485004908331009, 1535485004908331008, 1535485004902039552, 1535485004898893824, 1535485004894699521, 1535485004894699520, 1535485004891553792, 1535485004887359488, 1535485004883165185, 1535485004883165184, 1535485004878970880, 1535485004875825152, 1535485004871630849, 1535485004871630848, 1535485004867436544, 1535485004864290816, 1535485004860096513, 1535485004860096512, 1535485004855902208, 1535485004851707905, 1535485004851707904, 1535485004847513600, 1535485004843319297, 1535485004843319296, 1535485004837027841, 1535485004837027840, 1535485004832833538, 1535485004832833537, 1535485004832833536, 1535485004823396353, 1535485004823396352, 1535485004819202048, 1535485004816056321, 1535485004816056320, 1535485004811862016, 1535485004807667712, 1535485004803473409, 1535485004803473408, 1535485004799279104, 1535485004795084801, 1535485004795084800, 1535485004790890496, 1535485004787744768, 1535485004786696192, 1535485004783550464, 1535485004778307585, 1535485004778307584, 1535485004775161856, 1535485004770967552, 1535485004767821824, 1535485004766773248, 1535485004763627520, 1535485004759433217, 1535485004759433216, 1535485004754190337, 1535485004754190336, 1535485004748947456, 1535485004744753153, 1535485004744753152, 1535485004740558849, 1535485004740558848, 1535485004735315968, 1535485004731121664, 1535485004727975936, 1535485004726927360, 1535485004723781633, 1535485004723781632, 1535485004722733056, 1535485004714344448, 1535485004710150145, 1535485004710150144, 1535485004703858689, 1535485004703858688, 1535485004699664384, 1535485004695470080, 1535485004692324353, 1535485004692324352, 1535485004688130048, 1535485004684984320, 1535485004680790017, 1535485004680790016, 1535485004677644288, 1535485004673449985, 1535485004673449984, 1535485004668207105, 1535485004668207104, 1535485004664012800, 1535485004660867072]] 2016-06-01 17:19:44.599 info (corezkregister-1-thread-1-processing-n:172.30.92.66:8983_solr x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.leaderelector watching path /collections/testowa/leader_elect/shard1/election/240110433826439197-core_node3-n_0000000884 know if leader 2016-06-01 17:19:44.603 info (overseerstateupdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.overseer processmessage: queuesize: 1, message = { "operation":"leader", "shard":"shard1", "collection":"testowa"} current state version: 38 2016-06-01 17:19:44.607 info (overseerstateupdate-168013962670440512-172.30.92.66:8983_solr-n_0000000079) [ ] o.a.s.c.o.zkstatewriter going update_collection /collections/testowa/state.json version: 1351 2016-06-01 17:19:44.611 info (zkcallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.zkstatereader cluster state change: [watchedevent state:syncconnected type:nodedatachanged path:/collections/testowa/state.json] collection [testowa] has occurred - updating... (live nodes size: [3]) 2016-06-01 17:19:44.613 info (zkcallback-4-thread-1-processing-n:172.30.92.66:8983_solr) [ ] o.a.s.c.c.zkstatereader updating data [testowa] [1351] [1352] 2016-06-01 17:19:47.272 error (qtp1185255965-22) [ ] o.a.s.s.pkiauthenticationplugin exception trying public key : https://172.30.182.43:8983/solr org.noggit.jsonparser$parseexception: json parse error: char=<,position=0 before='<' after='html> <head> <meta http-equiv="content-' @ org.noggit.jsonparser.err(jsonparser.java:356) @ org.noggit.jsonparser.handlenondoublequotestring(jsonparser.java:712) @ org.noggit.jsonparser.next(jsonparser.java:886) @ org.noggit.jsonparser.nextevent(jsonparser.java:930) @ org.noggit.objectbuilder.<init>(objectbuilder.java:44) @ org.noggit.objectbuilder.getval(objectbuilder.java:37) @ org.apache.solr.common.util.utils.fromjson(utils.java:107) @ org.apache.solr.security.pkiauthenticationplugin.getremotepublickey(pkiauthenticationplugin.java:202) @ org.apache.solr.security.pkiauthenticationplugin.decipherheader(pkiauthenticationplugin.java:155) @ org.apache.solr.security.pkiauthenticationplugin.doauthenticate(pkiauthenticationplugin.java:118) @ org.apache.solr.servlet.solrdispatchfilter.authenticaterequest(solrdispatchfilter.java:283) @ org.apache.solr.servlet.solrdispatchfilter.dofilter(solrdispatchfilter.java:198) @ org.apache.solr.servlet.solrdispatchfilter.dofilter(solrdispatchfilter.java:184) @ org.eclipse.jetty.servlet.servlethandler$cachedchain.dofilter(servlethandler.java:1668) @ org.eclipse.jetty.servlet.servlethandler.dohandle(servlethandler.java:581) @ org.eclipse.jetty.server.handler.scopedhandler.handle(scopedhandler.java:143) @ org.eclipse.jetty.security.securityhandler.handle(securityhandler.java:548) @ org.eclipse.jetty.server.session.sessionhandler.dohandle(sessionhandler.java:226) @ org.eclipse.jetty.server.handler.contexthandler.dohandle(contexthandler.java:1160) @ org.eclipse.jetty.servlet.servlethandler.doscope(servlethandler.java:511) @ org.eclipse.jetty.server.session.sessionhandler.doscope(sessionhandler.java:185) @ org.eclipse.jetty.server.handler.contexthandler.doscope(contexthandler.java:1092) @ org.eclipse.jetty.server.handler.scopedhandler.handle(scopedhandler.java:141) @ org.eclipse.jetty.server.handler.contexthandlercollection.handle(contexthandlercollection.java:213) @ org.eclipse.jetty.server.handler.handlercollection.handle(handlercollection.java:119) @ org.eclipse.jetty.server.handler.handlerwrapper.handle(handlerwrapper.java:134) @ org.eclipse.jetty.server.server.handle(server.java:518) @ org.eclipse.jetty.server.httpchannel.handle(httpchannel.java:308) @ org.eclipse.jetty.server.httpconnection.onfillable(httpconnection.java:244) @ org.eclipse.jetty.io.abstractconnection$readcallback.succeeded(abstractconnection.java:273) @ org.eclipse.jetty.io.fillinterest.fillable(fillinterest.java:95) @ org.eclipse.jetty.io.ssl.sslconnection.onfillable(sslconnection.java:186) @ org.eclipse.jetty.io.abstractconnection$readcallback.succeeded(abstractconnection.java:273) @ org.eclipse.jetty.io.fillinterest.fillable(fillinterest.java:95) @ org.eclipse.jetty.io.selectchannelendpoint$2.run(selectchannelendpoint.java:93) @ org.eclipse.jetty.util.thread.strategy.executeproduceconsume.produceandrun(executeproduceconsume.java:246) @ org.eclipse.jetty.util.thread.strategy.executeproduceconsume.run(executeproduceconsume.java:156) @ org.eclipse.jetty.util.thread.queuedthreadpool.runjob(queuedthreadpool.java:654) @ org.eclipse.jetty.util.thread.queuedthreadpool$3.run(queuedthreadpool.java:572) @ java.lang.thread.run(thread.java:785) 2016-06-01 17:19:47.281 error (qtp1185255965-22) [ ] o.a.s.s.pkiauthenticationplugin decryption failed , key must wrong java.security.invalidkeyexception: no installed provider supports key: (null) @ javax.crypto.cipher.a(unknown source) @ javax.crypto.cipher.init(unknown source) @ javax.crypto.cipher.init(unknown source) @ org.apache.solr.util.cryptokeys.decryptrsa(cryptokeys.java:277) @ org.apache.solr.security.pkiauthenticationplugin.parsecipher(pkiauthenticationplugin.java:172) @ org.apache.solr.security.pkiauthenticationplugin.decipherheader(pkiauthenticationplugin.java:159) @ org.apache.solr.security.pkiauthenticationplugin.doauthenticate(pkiauthenticationplugin.java:118) @ org.apache.solr.servlet.solrdispatchfilter.authenticaterequest(solrdispatchfilter.java:283) @ org.apache.solr.servlet.solrdispatchfilter.dofilter(solrdispatchfilter.java:198) @ org.apache.solr.servlet.solrdispatchfilter.dofilter(solrdispatchfilter.java:184) @ org.eclipse.jetty.servlet.servlethandler$cachedchain.dofilter(servlethandler.java:1668) @ org.eclipse.jetty.servlet.servlethandler.dohandle(servlethandler.java:581) @ org.eclipse.jetty.server.handler.scopedhandler.handle(scopedhandler.java:143) @ org.eclipse.jetty.security.securityhandler.handle(securityhandler.java:548) @ org.eclipse.jetty.server.session.sessionhandler.dohandle(sessionhandler.java:226) @ org.eclipse.jetty.server.handler.contexthandler.dohandle(contexthandler.java:1160) @ org.eclipse.jetty.servlet.servlethandler.doscope(servlethandler.java:511) @ org.eclipse.jetty.server.session.sessionhandler.doscope(sessionhandler.java:185) @ org.eclipse.jetty.server.handler.contexthandler.doscope(contexthandler.java:1092) @ org.eclipse.jetty.server.handler.scopedhandler.handle(scopedhandler.java:141) @ org.eclipse.jetty.server.handler.contexthandlercollection.handle(contexthandlercollection.java:213) @ org.eclipse.jetty.server.handler.handlercollection.handle(handlercollection.java:119) @ org.eclipse.jetty.server.handler.handlerwrapper.handle(handlerwrapper.java:134) @ org.eclipse.jetty.server.server.handle(server.java:518) @ org.eclipse.jetty.server.httpchannel.handle(httpchannel.java:308) @ org.eclipse.jetty.server.httpconnection.onfillable(httpconnection.java:244) @ org.eclipse.jetty.io.abstractconnection$readcallback.succeeded(abstractconnection.java:273) @ org.eclipse.jetty.io.fillinterest.fillable(fillinterest.java:95) @ org.eclipse.jetty.io.ssl.sslconnection.onfillable(sslconnection.java:186) @ org.eclipse.jetty.io.abstractconnection$readcallback.succeeded(abstractconnection.java:273) @ org.eclipse.jetty.io.fillinterest.fillable(fillinterest.java:95) @ org.eclipse.jetty.io.selectchannelendpoint$2.run(selectchannelendpoint.java:93) @ org.eclipse.jetty.util.thread.strategy.executeproduceconsume.produceandrun(executeproduceconsume.java:246) @ org.eclipse.jetty.util.thread.strategy.executeproduceconsume.run(executeproduceconsume.java:156) @ org.eclipse.jetty.util.thread.queuedthreadpool.runjob(queuedthreadpool.java:654) @ org.eclipse.jetty.util.thread.queuedthreadpool$3.run(queuedthreadpool.java:572) @ java.lang.thread.run(thread.java:785) 2016-06-01 17:19:47.288 warn (qtp1185255965-22) [ ] o.a.s.s.pkiauthenticationplugin failed decrypt header, trying after refreshing key looks security plugins working ok, when enabled basic authentication not using superuser , nodes between cannot communicate. idea can wrong?
turns out "blockunknown" property in security.json root of evil. after going through steps scratch simple authentication not working property set. decided make configuration minimal can , worked once removed blockunknown security.json.
i not sure wrong property, after debugging session spotted possible error. internal solr nodes communication failing on fetching public keys of node in cluster, because of property in connection authentication. nodes reason not authenticating.
anyway... have authentication + authorization on ssl , can block unknown hosts on ssl level. brawo ja!
