Recently, I came across an issue with starting ODS managed servers after the backend server reboot. The error captured under OID $DOMAIN_HOME/sysman/log/emoms.log was – “javax.net.ssl.SSLKeyException: [Security:090479]Certificate chain received from hostname – IP failed date validity checks.”
The solution was applied on an OID 11g environment but should be applicable on other versions as well.
1. Let’s take a look at the error message in details. Similar error message can be found in the Node Manager log.
cd $DOMAIN_HOME/sysman/log
view emoms.log
2023-05-31 11:02:52,342 [Thread-309] ERROR sdk.pcs logp.251 - javax.net.ssl.SSLKeyException: [Security:090479]Certificate chain received from hostname - IP failed date validity checks.
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertSent(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)
at com.certicom.tls.record.handshake.ClientStateReceivedServerHello.handle(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessage(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessages(Unknown Source)
at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
at com.certicom.tls.record.WriteHandler.write(Unknown Source)
at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:254)
at weblogic.nodemanager.common.DataFormat.writeCommand(DataFormat.java:247)
at weblogic.nodemanager.client.NMServerClient.sendCmd(NMServerClient.java:318)
at weblogic.nodemanager.client.NMServerClient.sendHello(NMServerClient.java:128)
at weblogic.nodemanager.client.NMServerClient.connect(NMServerClient.java:239)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:200)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:206)
at weblogic.nodemanager.client.NMServerClient.start(NMServerClient.java:94)
at weblogic.nodemanager.mbean.StartRequest.start(StartRequest.java:75)
at weblogic.nodemanager.mbean.StartRequest.execute(StartRequest.java:47)
at weblogic.kernel.WorkManagerWrapper$1.run(WorkManagerWrapper.java:63)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)2. From the Node Manager startup log file, we can find the keystore information, querying which we can obtain the certificate details.
Loading identity key store: FileName=/u01/app/oracle/OID_Middleware/wlserver_10.3/server/lib/DemoIdentity.jks, Type=jks, PassPhraseUsed=true
Loaded node manager configuration properties from '/u01/app/oracle/OID_Middleware/wlserver_10.3/common/nodemanager/nodemanager.properties'
Startup configuration properties loaded from "/u01/app/oracle/OID_Middleware/user_projects/domains/IDMDomain/servers/wls_ods1/data/nodemanager/startup.properties
Starting WebLogic server with command line: /u01/app/oracle/OID_Middleware/user_projects/domains/IDMDomain/bin/startWebLogic.sh
javaHome=/u01/app/oracle/jdk1.7.0_99
JavaHome=/u01/app/oracle/jdk1.7.0_99/jreexport PATH=$PATH:/u01/app/oracle/jdk1.7.0_99/jre/bin
cd /u01/oracle/backup
cp /u01/app/oracle/OID_Middleware/wlserver_10.3/server/lib/DemoIdentity.jks .
cp /u01/app/oracle/OID_Middleware/wlserver_10.3/server/lib/DemoTrust.jks .
keytool -list -v -keystore DemoIdentity.jks3. Query Demo Trust store for the certificate information displayed in Step 2
keytool -list -v -keystore DemoTrust.jks | grep aliasname4. Export the certificate from Demo Identity keystore
Since the certificate was absent in the trust store, the ODS managed servers were unable to connect to Node Manager in order to start.
keytool -exportcert -alias demoidentity -keystore DemoIdentity.jks -file /u01/oracle/backup/trust.crt5. Import trust cert to Demo Trust store
cd /u01/app/oracle/OID_Middleware/wlserver_10.3/server/lib
cp /u01/oracle/backup/trust.crt .
keytool -import -trustcacerts -alias aliasname -file trust.crt -keystore DemoTrust.jks6. Import trust cert to Java trust store
cd /u01/app/oracle/jdk1.7.0_99/jre/lib/security/
cp cacerts cacerts_bkp
keytool -list -v -keystore cacerts | grep aliasname
cp /u01/oracle/backup/trust.crt .
keytool -import -trustcacerts -alias aliasname -file trust.crt -v -keystore cacerts7. Start ODS managed servers
cd /u01/app/oracle/OID_Middleware/asinst_1/bin
./opmnctl status -l
Processes in Instance: asinst_1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
oid1 | oidldapd | 507 | Alive | 1339394116 | 1412000 | 0:08:59 | N/A
oid1 | oidldapd | 476 | Alive | 1339394115 | 419992 | 0:09:00 | N/A
oid1 | oidmon | 446 | Alive | 1339394114 | 560252 | 0:09:00 | LDAPS:3131,LDAP:3060Now you can successfully login to ODSM console at http://hostname:port/odsm/faces/odsm.jspx.
AninditaK 