JMS Client could not establish connection with weblogic server

Issue : 

When an external application tries to establish connection to the weblogic JMS server, the connection does not seem to work.

We observe the below error in the logs :


weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]] for weblogic.jms.C:vmohsitut151:od8:-419rn532v3nc
        at weblogic.jms.frontend.FEConnectionFactory.connectionCreateInternal(FEConnectionFactory.java:371)
        at weblogic.jms.frontend.FEConnectionFactoryImpl.connectionCreateInternal(FEConnectionFactoryImpl.java:79)
        at weblogic.jms.frontend.FEConnectionFactoryImpl.connectionCreateRequest(FEConnectionFactoryImpl.java:62)
        at weblogic.jms.frontend.FEConnectionFactoryImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
        at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: weblogic.messaging.dispatcher.DispatcherException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]] for weblogic.jms.C:vmohsitut151:od8:-419rn532v3nc
        at weblogic.messaging.dispatcher.DispatcherWrapperState.addPeerGoneListener(DispatcherWrapperState.java:412)
        at weblogic.messaging.dispatcher.DispatcherWrapperState.<init>(DispatcherWrapperState.java:79)
        at weblogic.messaging.dispatcher.DispatcherManager.dispatcherAdd(DispatcherManager.java:222)
        at weblogic.messaging.dispatcher.DispatcherManager.addDispatcherReference(DispatcherManager.java:118)
        at weblogic.jms.dispatcher.JMSDispatcherManager.addDispatcherReference(JMSDispatcherManager.java:187)
        at weblogic.jms.frontend.FEConnectionFactory.connectionCreateInternal(FEConnectionFactory.java:368)
        ... 12 more
Caused by: weblogic.rmi.extensions.DisconnectMonitorUnavailableException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]]
        at weblogic.rmi.extensions.DisconnectMonitorListImpl.addDisconnectListener(DisconnectMonitorListImpl.java:83)
        at weblogic.messaging.dispatcher.DispatcherWrapperState.addPeerGoneListener(DispatcherWrapperState.java:409)
        ... 17 more



Observations made : 

This issue does not occur if the source client ( from where the connection to WLS JMS is made ) and Weblogic server are in the same network subnet.

The above issue has been observed under two conditions :

--> If these 2 systems are on different network
or
--> If the two systems are on different subnet in the same network

Solution : 

On the client side, Try to find out which weblogic jar file is being used to make the connection.

If the wlclient.jar file is used for making the connection, then this is the root cause of the issue. Instead of using the wlclient.jar, please use wlthint3client.jar and then try to make a connection to the weblogic JMS.

In our case, wlclient.jar file was being used and we replaced it with wlthint3client.jar

This resolved the issue.

After making the changes, our application file which made connection to JMS looked like below :

#$clp="${clp}:${lib_dir}/wlclient.jar";

$clp="${clp}:${lib_dir}/wlthint3client.jar";

Step by Step process to Install OSB : Oracle Service Bus 12C ( 12.1.3.0.0 version )

This post helps us to install OSB 12.1.3.0.0.

Before proceeding further make sure Weblogic 12.1.3.0.0 is installed. If not, please click here to install the required weblogic server.

Once the installation of weblogic server is completed, please follow the below steps to install the OSB 12.1.3.0.0

The installer to be used for OSB installation is : fmw_12.1.3.0.0_osb.jar and the required installer can be downloaded from edelivery.oracle.com

Execute the below command to start the OSB installer.

<Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_osb.jar

For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\OSB\fmw_12.1.3.0.0_osb.jar

This will initiate the installer. Please proceed as per the steps below.

Click Next on the above screen. Installation location screen would appear. Provide the location where Weblogic Server is installed.

Click Next on the above screen. Installation Type screen would appear. Select Service Bus and click next.

Pre-Requisites would be checked in the below screen. Take appropriate action if any issue is reported. If no issue is reported, Click Next

Summary screen would appear. Analyze the information and click next

Installation progress would be displayed in the below screen

Once done, click finish to close the installer

This completes the installation of OSB : Oracle Service Bus installation ( 12C : 12.1.3.0.0 version )

Step by Step process to Install SOA 12C ( 12.1.3.0.0 version )

This post helps us to install SOA 12.1.3.0.0.

Before proceeding further make sure Weblogic 12.1.3.0.0 is installed. If not, please click here to install the required weblogic server.

Once the installation of weblogic server is completed, please follow the below steps to install the SOA 12.1.3.0.0

The installer to be used for SOA installation is : fmw_12.1.3.0.0_soa.jar and the required installer can be downloaded from edelivery.oracle.com

Execute the below command to start the SOA installer.

<Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_soa.jar

For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\SOA\fmw_12.1.3.0.0_soa.jar

This will initiate the installer. Please proceed as per the steps below.

Click Next on the above screen. In the next screen, you would be prompted for Oracle Home. Please provide the location where Weblogic Server is installed and Click Next

Please select the SOA Suite in the below screen and Click Next

Pre-Requisites would be checked in the below screen. Take appropriate action if any issue is reported. If no issue is reported, Click Next

Summary screen would appear. Analyze the information and click next

Installation progress would be displayed in the below screen

Once done, click next on the below screen

Once done, click finish to close the installer

This completes the installation of SOA 12C ( 12.1.3.0.0 version )

Step by Step process to Install Weblogic 12C ( 12.1.3.0.0 version )

This post helps us to install Weblogic Server 12C (12.1.3.0.0 version ). In the long run this weblogic server home would be used to install SOA, OSB components.

Please use the installer : fmw_12.1.3.0.0_infrastructure.jar for installation of Weblogic server. If this is not used, then it might lead to multiple issues while installing SOA with respect to JRF libraries at a later point of time.

The required installer can be downloaded from edelivery.oracle.com

Below are the steps to be followed for installing WLS 12c version.

1.  Install JDK : jdk-7u67-windows-x64 (Java SE Development Kit ). The version should be : java version "1.7.0_67"
2. Install Weblogic Server.

Below are the steps to be followed for Weblogic Installation.

Run the command as below :

 <Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_infrastructure.jar
For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\Infrastructure\V44416-01\fmw_12.1.3.0.0_infrastructure.jar

This will start the WLS installer. Follow the steps as per the below screen shots

Click Next. In the below screen provide Oracle Home to the location where Weblogic server needs to be installed. Then Click Next

Select Fusion Middleware Infrastructure and Click Next

In the below screen, pre-requisites would be checked. Take appropriate action, if any issue is reported in the below screen

The below step can be skipped by not providing any email address.

Click Next and installation process would begin as shown below.

Once done, review the below information and cick finish to close the installer as per below screen shot.

This will install the weblogic server at the location provided in the installer screen. Navigate to that location and check if the binaries have been installed.

We will be updating additions posts regarding the Installation and Configuration of SOA, OSB 12.1.3 version and the steps for domain creation.

Opatch: LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo : OPatch failed with error code 73

Problem

While running opatch lsinventory, command fails with the below error:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Possible causes are:

   Oracle Home dir. path does not exist in Central Inventory

   Oracle Home is a symbolic link

   Oracle Home inventory is corrupted

LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73

Solution

Attach the Oracle Home to the Central Inventory:

$ cd $ORACLE_HOME/oui/bin 

$ ./attachHome.sh

Now check, command should work fine.

$ORACLE_HOME/OPatch/opatch lsinventory

How to change EM Agent Port ?

EM Agent uses a specific port to receive requests. This port is defined during installation. We faced an issue while installing OID and ODS, both have EMAGENT component:

Processes in Instance: oid1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
oid1                             | oidldapd           |   31821 | Alive    | 1389237283 |  1472620 |  21:24:59 | N/A
oid1                             | oidldapd           |   31790 | Alive    | 1389237282 |   415084 |  21:25:00 | N/A
oid1                             | oidmon             |   31754 | Alive    | 1389237281 |   559676 |  21:25:00 | LDAPS:4040,LDAP:4041
EMAGENT                          | EMAGENT            |   31755 | Alive    | 1389237280 |   106148 |  21:25:00 | N/A



Processes in Instance: ods1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
EMAGENT                          | EMAGENT            |   30455 | Down    |  436375769 |   106148 |   | N/A


If by mistake you give same port to EMAGENT during installation of OID and ODS then it will give you an exception of "Address already in use or bind exception" while starting the component. EMAGENT can be seen as down in one of the instance.

Solution

You have to change EMAGENT port for one of the component.

Steps

1. Stop the EMAgent.
$ $AGENT_HOME/bin/emctl stop agent

In our case $AGENT_HOME=/home/ajNode1/instances/ods1/EMAGENT/EMAGENT

2. Modify the $AGENT_HOME/sysman/config/emd.properties file in the EMAgent home directory:
    For example:
    EMD_URL=http://ajmachine1:5011/emd/main

    Modify the <port number> in the EMD_URL property so the EMAgent uses a new unused port on the target host.
    EMD_URL=http://ajmachine1:5015/emd/main

3. Start the EMAgent.
$ $AGENT_HOME/bin/emctl start agent

Error while creating ovd component on second node

Issue

While doing cluster installation of Oracle Virtual Directory (OVD), we faced this issue. When you run config.sh from $ORACLE_IDM_HOME/bin, it goes successful and creates a ovd component, however it register it as ovd1 instead of ovd2 on second node.

Hence, folder structure created is something like this for 2 nodes:

on Node1: OVD1 server: /home/ajnode1/instances/ovdinst1/config/OVD/ovd1

on Node2: OVD2 server: /home/ajnode2/instances/ovdinst2/config/OVD/ovd1

Due to same name of ovd instance on both nodes, it creates a confusion and it is difficult to monitor.

Solution

It is a bug, and only solution is to manually create the component on second node.

a) Stop ovd2:

/home/ajnode2/instances/ovdinst2/bin/opmnctl stopall

b) Delete the component:

/home/ajnode2/instances/ovdinst2/bin/opmnctl deletecomponent -componentName ovd1

c) Create a new OVD component and name it ovd2 (fill the values accoring to your environment

-bash-4.1$ ./opmnctl createcomponent -adminHost <admin_host> -adminPort <admin_port> -adminUsername <weblogic> -componentName ovd2 -componentType OVD -admin cn=orcladmin -isAdminSSL true -ovdAdminPort <ovd_admin_port> -namespace <dc=your company,dc=com> -ldapPort <Non SSL Port> -ldapSport <SSL Port>

Command requires login to weblogic admin server (localhost):
  Username: weblogic
  Password:


Creating empty component directories...Done
Provisioning OVD files for ovd2
  OVD creation in progress
  Creating OVD specific directories
  Copying Configuration Files
  Registering OVD Ports
  Updating OVD Admin Password
OVD Admin Password:

  Setting CSF values
  Updating Configuration Files
  Adding opmn Snippet
  Setting proxy mbean information
Registering ovd2 component
Command succeeded.


d) Restart the ovd server on second node.