Managed Server Does not Come Up with Empty Initial Replica

While trying to start managed server from a remote machine, we see the below errors

<Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.ldap.EmbeddedLDAPException: Empty initial replica
weblogic.ldap.EmbeddedLDAPException: Empty initial replica
     at weblogic.ldap.EmbeddedLDAP.getInitialReplicaFromAdminServer(EmbeddedLDAP.java:1301)
     at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:221)
     at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
     Truncated. see log file for complete stacktrace

Along with  this error, we have also seen the error which says :

"Unable to Initiate Empty Initial Replica"

One of the possible causes of this issue is that in case your environment is a multi node cluster, then this specific error is observed if the clock of the nodes are not in sync with each other.

Resolution :
Work with your system admin and make sure the clock of all the node of the weblogic domain are in sync with each other.

Then try starting the server. This should fix the issue.

Unable to start weblogic application with stackoverflow error

We have faced an issue where the weblogic managed server comes into RUNNING mode. However, some of the applications targeted to the managed server do not come with the underlying error message in the log files.

<Apr 20, 2017 11:36:04 AM CEST> <Error> <Deployer> <BEA-149265> <Failure occurred in the execution of deployment request with ID "4411720645512325" for task "1". Error is: "weblogic.application.ModuleException: java.lang.StackOverflowError"
weblogic.application.ModuleException: java.lang.StackOverflowError
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:140)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:213)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:208)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.StackOverflowError
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:385)
        at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:344)
        Truncated. see log file for complete stacktrace

Cause of the issue : 
The issue occurs if the default stack size is not sufficient for your application. We need to increase the stack size to resolve this issue.

Solution : 
Please apply the below solution to resolve the issue :

On the admin console --> Click on Summary of servers --> Click on the server name --> Under Configuration --> Server Start

In the arguments section, please add the below parameter and restart the managed server.

-Xss4096k

This should hopefully resolve the issue.

Whole Server Migration - WebLogic 11g

Migration in WebLogic can be categorized broadly into two:

1. Service Migration
2. Server Migration

In Service migration, singleton services like JMS, JTA are migrated from one server to another during server failure.

I will discuss Service Migration in another post.

In Server migration, whole server is migrated i.e the failed server is restarted on another node.

To show this, I have 2 Virtual machines, on which I configured a cluster in a domain. After creating a domain, It looks like this:

Admin and MS1 running on - ajMachine1

MS2 running on - ajMachine2

Below are the high level steps for configuring Whole Server Migration:

1. Configure floating IP addresses for each managed server.

2. Set correct sudo privileges for user

3. Configure the node manager on each Machine

4. Configure leasing mechanism.

5. Configure non-xa data source for automatic migration.

6. Configure cluster migration.

7. Configure automatic server migration for each managed server.

8. Test whole server migration.

Step 1

=====

Configure Floating IP address for each host like this, It can be any IP which you assign. But make sure to add them in /etc/hosts file for each node if you are not using any DNS server.

Below if you see, eth6:1 10.0.0.6 is my floating IP on machine 1 and similarly create another VIP  eth6:1 10.0.0.7 for machine 2

Step 2

=====

In /etc/sudoers file, add these lines:

Step 3

======

In nodemanager.properties file, add the below lines, so when you start managed server using NM, it will mount the associated VIP.

Step 4

=====

Run the below queries in Database:

SQL> create tablespace leasing datafile '/u01/app/oracle/oradata/orcl/leasing.dbf' maxsize 2048m extent management local;

Tablespace created.

SQL> create user leasing identified by oracle123;

User created.

SQL> grant create table to leasing;

Grant succeeded.

SQL> grant create session to leasing;

Grant succeeded.

SQL> alter user leasing default tablespace leasing;

User altered.

SQL> alter user leasing quota unlimited on leasing;

User altered.

SQL> exit

Create ACTIVE Table:

=====================

The ddl to create can be found at $WLSERVER_HOME/server/db/oracle/920/leasing.ddl

[oracle@db tmp]$ sqlplus leasing/Password123

SQL*Plus: Release 11.2.0.1.0 Production on Sat May 21 15:09:25 2016

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> @leasing.ddl

SP2-0734: unknown command beginning "WebLogic S..." - rest of line ignored.

SP2-0734: unknown command beginning "Copyright ..." - rest of line ignored.

DROP TABLE ACTIVE

          *

ERROR at line 1:

ORA-00942: table or view does not exist

Table created.

SQL> desc Active

Name                                      Null?    Type

----------------------------------------- -------- ----------------------------

SERVER                                    NOT NULL VARCHAR2(150)

INSTANCE                                  NOT NULL VARCHAR2(100)

DOMAINNAME                                NOT NULL VARCHAR2(50)

CLUSTERNAME                               NOT NULL VARCHAR2(50)

TIMEOUT                                            DATE

SQL> select * from Active;

no rows selected

Step 5

======

On the Admin Console, create non-xa data source, pointing to leasing schema (created above) and target it to the cluster

Step 6

=====

Navigate to Environments - > Cluster, and click on cluster ->Migration tab.

Update as below:

Step 7

======

Click on MS1 and go to migration tab, and update as below:

Similar for MS2 as well.

Step 8

=====

Now restart all the servers and test the migration.

Kill MS2 and Nodemanager on Node2. After few seconds, you will see, MS2 will get started on Node1

After killing MS2 server on Node2

It starts up on Node1:

readDomain() on Unpacked Weblogic 12C Domain Not Working

We have observed the below issue on 12.1.3 version of weblogic server.

As a part of pack-unpack to create a managed server we have performed the below steps :

--> Created a domain
--> Created a jar file of the domain using the pack utility :

./pack.sh -managed=true -domain=<User_Project_Dir> /domains/<Domain_Name> -template=<User_Project_Dir>/domains/<Domain_Name>.jar -template_name="<Domain_name>"

This will create the jar file at the location.

--> We have tried to unpack the same at the managed server location. Unpack was successfull.

After completion of unpack, we were unable to read the domain.

Connect to WLST :

After connecting to the WLST :
--> readDomain(<Original Domain Directory Location>)

This worked fine

--> readDomain(<domainDirecotry of the unpacked domain>)
This did not work with the below error :

com.oracle.cie.domain.script.jython.WLSTException: com.oracle.cie.domain.script.ScriptException: com.oracle.cie.common.CommonException: fail to load domain from directory
Root Cause: java.lang.reflect.InvocationTargetException

Caused by: com.oracle.cie.domain.script.ScriptException: com.oracle.cie.common.CommonException: fail to load domain from directory
Root Cause: java.lang.reflect.InvocationTargetException
        at com.oracle.cie.domain.script.ScriptExecutor.readTemplate(ScriptExecutor.java:542)
        at com.oracle.cie.domain.script.jython.WLScriptContext.readDomain(WLScriptContext.java:746)
        ... 30 more
Caused by: com.oracle.cie.common.CommonException: fail to load domain from directory
Root Cause: java.lang.reflect.InvocationTargetException


This error was not observed in the 10.3.6 version of weblogic server. This has been reported as a bug with Oracle and currently ( as on 23rd March 2016 ), Oracle is working on resolving the issue.

We will keep this post updated once the issue is resolved.


However, as a temporary work around, please use the pack command with the syntax "managed=false"

This is only a temporary work around and not a permanent solution.

OHS 12.1.3 does not come up after fresh installation

OHS Component 12.1.3 does not come up after fresh installation. We have performed a standard alone installation of OHS. Below is the sequence of steps we have followed :

--> Installed OHS 12.1.3
--> Configured the OHS domain
--> Started Node Manager.
--> Node Manager works fine
--> Tried to start OHS component. OHS doesn't start up with the below error.

./startComponent.sh ohs2
Starting system Component ohs2 ...

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Reading domain from /u01/ofsaa/local/user_projects/domains/ohs12c_domain


Please enter Node Manager password:
Connecting to Node Manager ...
Successfully Connected to Node Manager.
Starting server ohs2 ...
Error Starting server ohs2: weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Server start command for OHS server 'ohs2' failed due to: [Unable to find template directory /u01/ofsaa/local/fmw/ohs/templates/conf within OHS installation at /u01/ofsaa/local/fmw/ohs. Make sure that /u01/ofsaa/local/fmw/ohs contains a valid OHS installation.]. Please check Node Manager log and/or server 'ohs2' log for detailed information.]. Please check Node Manager log for details.
Successfully disconnected from Node Manager.


Exiting WebLogic Scripting Tool.

Done

--> The cause of this issue is that installation of OHS has been done with invalid values of ORACLE_HOME and MW_HOME in .profile

--> Follow the below steps to resolve the issue.
--> Comment the entries of MW_HOME and ORACLE_HOME in .profile
--> Deinstall OHS ( Please be sure to run deinstall.sh in order to perform the deinstall. Just deleting the OHS folder will not work )
--> Reinstall OHS
--> Try to start the OHS now.
--> This should resolve the issue.

keytool error: java.io.IOException: Invalid keystore format

While listing the certificates from .oamkeystore was getting below error:

oracle@aj-oamr2 fmwconfig]$ ls -la .oamkeystore

-rw-r-----. 1 oracle oinstall 10441 Feb  2 11:33 .oamkeystore

[oracle@aj-oamr2 fmwconfig]$ keytool -list -keystore .oamkeystore -storepass <password>
keytool error: java.io.IOException: Invalid keystore format

The problem is with the keystore type which needs to be also defined in the above command.

Example:

[oracle@aj-oamr2 fmwconfig]$ keytool -list -keystore .oamkeystore -storetype JCEKS -storepass <password>

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 12 entries

adminserver, Feb 2, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1): C2:48:94:91:54:A0:BC:5A:11:A7:42:EC:27:9C:8B:57:4C:D0:07:0E
coherence, Feb 2, 2016, SecretKeyEntry,

How to retrieve .oamkeystore password in 11gR2PS3

While attempting to get the keystore password using listCred() wlst command, I got the below error:

wls:/oam_domain/domainRuntime> listCred(map="OAM_STORE",key="jks")
Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listCred


wls:/oam_domain/domainRuntime> help('listCred')
No help for listCred found. Please try help() for available options.

Oracle has removed listCred wlst command and is no longer available in OAM 11.1.2.3.0

To retrive the .oamkeystore password follow the below steps:

1) Log into to EM console, navigate to Weblogic Domain ->Domain Name ->System Mbean Browser 

Under Application Defined Mbean go to om.oracle.jps ->Domain:<domain_name> -> JpsCredentialStore ->JpsCredentialStore 

As shown

2) Click Operation tab and select getPortableCredential

 3) Enter both the parameters as shown

4) Click on Invoke to get the password