Java.lang.OutOfMemoryError" Error When Applying Oracle Weblogic Server Patch

While applying the latest PSU for Weblogic, you might get below error:

./bsu.sh -install -patch_download_dir=/home/oracle/fmw/product/112/utils/bsu/cache_dir -patchlist=S8C2 -prod_dir=/home/oracle/fmw/product/112/wlserver_10.3/

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.createEntry(HashMap.java:897)
        at java.util.HashMap.addEntry(HashMap.java:884)
        at java.util.HashMap.put(HashMap.java:505)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.loadPropertyMap(XBeanDataHandler.java:778)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.<init>(XBeanDataHandler.java:99)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.createDataHandler(XBeanDataHandler.java:559)

Cause:
The memory arguments are in bsu.sh are not sufficient

Solution:
1) Go to <Middleware_home>/utils/bsu
2) Take backup of bsu.sh and edit as below

[oracle@aj-oamr2 bsu]$ diff bsu.sh bsu.sh_bkup
5c5
< MEM_ARGS="-Xms1024m -Xmx1024m"
---
> MEM_ARGS="-Xms256m -Xmx512m"

4) Apply the patch
[oracle@aj-oamr2 bsu]$ ./bsu.sh -install -patch_download_dir=/home/oracle/fmw/product/112/utils/bsu/cache_dir -patchlist=S8C2 -prod_dir=/home/oracle/fmw/product/112/wlserver_10.3/
Checking for conflicts.....
No conflict(s) detected

Installing Patch ID: S8C2..
Result: Success

How to enable Federation Services in OAM 11gR2PS3 ?

By default Identity Federation Service is not enabled. To enable it, you need to go to Available Services and enable it. If the ‘Enable Service’ link is greyed out, try to disable ‘Mobile and Social' and then Enable it.

Looks like a bug, but that is the only way I was able to enable the Federation Services :)

Upgrading OAM to 11gR2PS3(11.1.2.3)

Recently I tried to upgrade OAM from 11gR2 (11.1.2.0) to 11gR2PS3 (11.1.2.3)

Here are the high level steps for upgrading. The steps are simple and self explanatory.  It was a smooth upgrade with no major issues.

1) Upgrade OAM binaries to PS3: This step is a typical OAM installation process and it will update binaries. Give the same Oracle Home as already present for IAM installation.

2) Upgrade oam, IAU and opss schema:

- Can be done by running psa (<MW_HOME>/oracle_common/bin/)

3) Copy Mbean: Run the below command from IAM home

wls:/offline> copyMbeanXmlFiles('/home/oracle/admin/user_projects/domains/oam_domain','/home/oracle/fmw/product/112/iamR2')

LOGGER intialised java.util.logging.Logger@133b514c

 ===============================================

STATUS: SUCCESS

The mbean xml files have been upgraded to new version.

The original mbean xml is saved in "<domain_directory>/output/upgrade".

Please restart the admin and oam servers.

 ===============================================

4) Upgrade OPSS: This step upgrades the OPSS with configuration and policy stores to PS3. It upgrades jps-config.xml and policy stores. 

upgradeOpss(jpsConfig=“/home/oracle/admin/user_projects/domains/oam_domain/config/fmwconfig/jps-config.xml”,jaznData=“/home/oracle/fmw/product/112/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml”, auditStore=“/home/oracle/fmw/product/112/oracle_common/modules/oracle.jps_11.1.1/domain_config/audit-store.xml”, jdbcDriver=“oracle.jdbc.OracleDriver”, url=“jdbc:oracle:thin:@<db_host>:<db_port>/<sid>”, user=“<prefix>_OPSS”, password=“<opss_schema_password>”, upgradeJseStoreType=“true”)

5) Undeploying Coherence: This step un-deploys old coherence library, as with PS3, latest version of coherence gets deployed.

wls:/oam_domain/serverConfig> undeploy('coherence#3.7.1.1@3.7.1.1')

Undeploying application coherence#3.7.1.1@3.7.1.1 ...

<Feb 9, 2016 9:31:16 PM GMT> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating undeploy operation for application, coherence#3.7.1.1@3.7.1.1 [archive: null], to AdminServer oam_server1 .>

.Completed the undeployment of Application with status completed

Current Status of your Deployment:

Deployment command type: undeploy

Deployment State       : completed

Deployment Message     : [Deployer:149194]Operation 'remove' on application 'coherence [LibSpecVersion=3.7.1.1,LibImplVersion=3.7.1.1]' has succeeded on 'AdminServer'

6) Upgrading System Configuration:

a) Go to the directory ORACLE_HOME/common/script_handlers.

b) Remove all the .class files by running the following command:

c) rm *.class

Run the below command from IAM home by running wlst:

upgradeConfig("/home/oracle/admin/user_projects/domains/oam_domain", "sys", “<sys_password>”, “<prefix>_OAM", "jdbc:oracle:thin:@<db_host>:<db_port>/<service_name>”)

Feb 09, 2016 9:14:40 PM oracle.security.am.admin.config.util.GenericFactory locateMethod

INFO: Factory methods not declared for type: DecreaseDeviceStoreCacheRuntimeCacheSize.

Feb 09, 2016 9:14:40 PM oracle.security.am.admin.config.BasicFileConfigurationStore applyUpdates

INFO: Applying configuration update at path /. Current version of configuration 108. Change initiated on this node: true

Feb 09, 2016 9:14:40 PM oracle.security.am.upgrade.framework.psfe.PSFEFramework process

INFO: Plugin oracle.security.am.upgrade.framework.psfe.plugin.PSFEDefaultPlugin successfully processed and validated for featureID:                                 acheSize

Feb 09, 2016 9:14:41 PM oracle.security.am.admin.config.util.GenericFactory locateMethod

INFO: Factory methods not declared for type: VersionUpdate.

Feb 09, 2016 9:14:41 PM oracle.security.am.admin.config.BasicFileConfigurationStore applyUpdates

INFO: Applying configuration update at path /. Current version of configuration 109. Change initiated on this node: true

Feb 09, 2016 9:14:42 PM oracle.security.am.upgrade.framework.psfe.PSFEFramework process

INFO: System configurations have been successfully upgraded to version: 11.1.2.3.0

Restart the services and start exploring new OAM interface !

om.bea.security.providers.authorization.asi.InvocationException: ArmeRUNTIME Exception: null

Below stack trace of exception is seen after starting managed server in oam logs:

Oracle Access Manager - Version 11.1.2.0.0

<Feb 9, 2016 3:09:45 PM GMT> <Error> <oracle.jps.authorization.framework> <JPS-03156> <The exception has been thrown by ARME. The authorization result is set to deny.
com.bea.security.providers.authorization.asi.InvocationException: ArmeRUNTIME Exception: null
        at com.bea.security.providers.authorization.asi.AuthorizationProviderImpl.isAccessAllowed(AuthorizationProviderImpl.java:396)
        at com.bea.security.ssal.micro.MicroAuthorizationManagerWrapper.isAccessAllowed(MicroAuthorizationManagerWrapper.java:73)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed_internal(AuthorizationServiceImpl.java:914)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:745)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:668)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:622)
        at com.bea.security.AuthorizationService.isAccessAllowed(AuthorizationService.java:365)
        at oracle.security.am.common.policy.runtime.provider.oes.proxy.OESRuntimeProxy.wait4OESRuntimeDBPolicyRefreshCompletion(OESRuntimeProxy.java:263)
        at oracle.security.am.common.policy.runtime.provider.oes.proxy.OESRuntimeProxy.init(OESRuntimeProxy.java:193)
        at oracle.security.am.common.policy.runtime.provider.oes.OESPolicyRuntimeProvider.init(OESPolicyRuntimeProvider.java:167)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.getNewInstance(PolicyRuntimeFactory.java:162)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.init(PolicyRuntimeFactory.java:93)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.getPolicyRuntime(PolicyRuntimeFactory.java:84)
        at oracle.security.am.common.policy.util.PolicyComponentLifecycle.initialize(PolicyComponentLifecycle.java:100)

As per Doc ID (1509559.1), these are harmless exceptions and can be ignored.

The policy store is not available; please see the log files for more details

After installing OAM 11gR2, you are getting error while loging into oamconsole.

You did the same mistake which I did of not reading the product documentation before starting with the installation.(http://docs.oracle.com/cd/E27559_01/install.1112/e27301/install.htm#INOAM98374)

There is an additional step in OAM 11gR2 (11.1.2.0) which says before starting up the servers, you need to configure the Security Store.

Follow the below steps to fix this error:

Solution
-----------
1) Delete the Domain
2) Remove the schema from RCU
3) Create the schema again.
4) Create the domain by running config.sh
5) Now the missed step, run the configSecurityStore as below:

$MIDDLWARE_HOME/oracle_common/common/bin/wlst.sh
$ORACLE_HOME/common/tools/configureSecurityStore.py -d $DOMAIN_HOME -c IAM -p <opss_schema_password> -m create

6) Start up the servers.

This should fix the issue.

webgate.so: undefined symbol: nzos_GetSessionMasterSecret

While starting ohs, I was getting below errors in ohs1.log after I configured Webgate 11.2.3.0 with OHS 11.1.1.7


ERROR
--------------

/home/oracle/fmw/products/111/ohs_117/ohs/bin/apachectl startssl: execing httpd
httpd.worker: Syntax error on line 1035 of /home/oracle/admin/ohs_instances/config/OHS/ohs1/httpd.conf: Syntax error on line 4 of /home/oracle/admin/ohs_instances/config/OHS/ohs1/webgate.conf: Cannot load /home/oracle/fmw/products/111/oamWebgateR2/webgate/ohs/lib/webgate.so into server: /home/oracle/fmw/products/111/oamWebgateR2/webgate/ohs/lib/webgate.so: undefined symbol: nzos_GetSessionMasterSecret

ISSUE
-----------

Above error is caused by using incompatible version of Webgate with corresponding OHS
Webgate version 11.1.2.3 requires OHS 11.1.1.9 version. I was on OHS 11.1.1.7

Solution
-------------

For OHS 11.1.1.7 version you have to use Webgate 11.1.2.2.

ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []

You might face this error while starting Database.

SQL> startup
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2261368 bytes
Variable Size            3254783624 bytes
Database Buffers         6341787648 bytes
Redo Buffers               21692416 bytes
Database mounted.
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [],
[], [], [], [], [], [], [], []


Reason: It is due to server crash might be due to hardware or power failure.

Solution:

Recover the database by following the below steps:

Step 1) Shutdown the instance and open in mount state:

SQL>shutdown immediate;
ORACLE instance shut down.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2261368 bytes
Variable Size            3254783624 bytes
Database Buffers         6341787648 bytes
Redo Buffers               21692416 bytes
Database mounted.

Step 2) Recover Database

SQL> recover database;

Media recovery complete.

Step 3) Open the Database

SQL> alter database open;

Database altered.

Step 4) Shutdown and startup normally

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup

ORACLE instance started.

Total System Global Area 9620525056 bytes

Fixed Size                  2261368 bytes

Variable Size            3254783624 bytes

Database Buffers         6341787648 bytes

Redo Buffers               21692416 bytes

Database mounted.

Database opened.