OHS 12.1.3 does not come up after fresh installation

OHS Component 12.1.3 does not come up after fresh installation. We have performed a standard alone installation of OHS. Below is the sequence of steps we have followed :

--> Installed OHS 12.1.3
--> Configured the OHS domain
--> Started Node Manager.
--> Node Manager works fine
--> Tried to start OHS component. OHS doesn't start up with the below error.

./startComponent.sh ohs2
Starting system Component ohs2 ...

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Reading domain from /u01/ofsaa/local/user_projects/domains/ohs12c_domain


Please enter Node Manager password:
Connecting to Node Manager ...
Successfully Connected to Node Manager.
Starting server ohs2 ...
Error Starting server ohs2: weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Server start command for OHS server 'ohs2' failed due to: [Unable to find template directory /u01/ofsaa/local/fmw/ohs/templates/conf within OHS installation at /u01/ofsaa/local/fmw/ohs. Make sure that /u01/ofsaa/local/fmw/ohs contains a valid OHS installation.]. Please check Node Manager log and/or server 'ohs2' log for detailed information.]. Please check Node Manager log for details.
Successfully disconnected from Node Manager.


Exiting WebLogic Scripting Tool.

Done

--> The cause of this issue is that installation of OHS has been done with invalid values of ORACLE_HOME and MW_HOME in .profile

--> Follow the below steps to resolve the issue.
--> Comment the entries of MW_HOME and ORACLE_HOME in .profile
--> Deinstall OHS ( Please be sure to run deinstall.sh in order to perform the deinstall. Just deleting the OHS folder will not work )
--> Reinstall OHS
--> Try to start the OHS now.
--> This should resolve the issue.

keytool error: java.io.IOException: Invalid keystore format

While listing the certificates from .oamkeystore was getting below error:

oracle@aj-oamr2 fmwconfig]$ ls -la .oamkeystore

-rw-r-----. 1 oracle oinstall 10441 Feb  2 11:33 .oamkeystore

[oracle@aj-oamr2 fmwconfig]$ keytool -list -keystore .oamkeystore -storepass <password>
keytool error: java.io.IOException: Invalid keystore format

The problem is with the keystore type which needs to be also defined in the above command.

Example:

[oracle@aj-oamr2 fmwconfig]$ keytool -list -keystore .oamkeystore -storetype JCEKS -storepass <password>

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 12 entries

adminserver, Feb 2, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1): C2:48:94:91:54:A0:BC:5A:11:A7:42:EC:27:9C:8B:57:4C:D0:07:0E
coherence, Feb 2, 2016, SecretKeyEntry,

How to retrieve .oamkeystore password in 11gR2PS3

While attempting to get the keystore password using listCred() wlst command, I got the below error:

wls:/oam_domain/domainRuntime> listCred(map="OAM_STORE",key="jks")
Traceback (innermost last):
  File "<console>", line 1, in ?
NameError: listCred


wls:/oam_domain/domainRuntime> help('listCred')
No help for listCred found. Please try help() for available options.

Oracle has removed listCred wlst command and is no longer available in OAM 11.1.2.3.0

To retrive the .oamkeystore password follow the below steps:

1) Log into to EM console, navigate to Weblogic Domain ->Domain Name ->System Mbean Browser 

Under Application Defined Mbean go to om.oracle.jps ->Domain:<domain_name> -> JpsCredentialStore ->JpsCredentialStore 

As shown

2) Click Operation tab and select getPortableCredential

 3) Enter both the parameters as shown

4) Click on Invoke to get the password

Java.lang.OutOfMemoryError" Error When Applying Oracle Weblogic Server Patch

While applying the latest PSU for Weblogic, you might get below error:

./bsu.sh -install -patch_download_dir=/home/oracle/fmw/product/112/utils/bsu/cache_dir -patchlist=S8C2 -prod_dir=/home/oracle/fmw/product/112/wlserver_10.3/

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.createEntry(HashMap.java:897)
        at java.util.HashMap.addEntry(HashMap.java:884)
        at java.util.HashMap.put(HashMap.java:505)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.loadPropertyMap(XBeanDataHandler.java:778)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.<init>(XBeanDataHandler.java:99)
        at com.bea.cie.common.dao.xbean.XBeanDataHandler.createDataHandler(XBeanDataHandler.java:559)

Cause:
The memory arguments are in bsu.sh are not sufficient

Solution:
1) Go to <Middleware_home>/utils/bsu
2) Take backup of bsu.sh and edit as below

[oracle@aj-oamr2 bsu]$ diff bsu.sh bsu.sh_bkup
5c5
< MEM_ARGS="-Xms1024m -Xmx1024m"
---
> MEM_ARGS="-Xms256m -Xmx512m"

4) Apply the patch
[oracle@aj-oamr2 bsu]$ ./bsu.sh -install -patch_download_dir=/home/oracle/fmw/product/112/utils/bsu/cache_dir -patchlist=S8C2 -prod_dir=/home/oracle/fmw/product/112/wlserver_10.3/
Checking for conflicts.....
No conflict(s) detected

Installing Patch ID: S8C2..
Result: Success

How to enable Federation Services in OAM 11gR2PS3 ?

By default Identity Federation Service is not enabled. To enable it, you need to go to Available Services and enable it. If the ‘Enable Service’ link is greyed out, try to disable ‘Mobile and Social' and then Enable it.

Looks like a bug, but that is the only way I was able to enable the Federation Services :)

Upgrading OAM to 11gR2PS3(11.1.2.3)

Recently I tried to upgrade OAM from 11gR2 (11.1.2.0) to 11gR2PS3 (11.1.2.3)

Here are the high level steps for upgrading. The steps are simple and self explanatory.  It was a smooth upgrade with no major issues.

1) Upgrade OAM binaries to PS3: This step is a typical OAM installation process and it will update binaries. Give the same Oracle Home as already present for IAM installation.

2) Upgrade oam, IAU and opss schema:

- Can be done by running psa (<MW_HOME>/oracle_common/bin/)

3) Copy Mbean: Run the below command from IAM home

wls:/offline> copyMbeanXmlFiles('/home/oracle/admin/user_projects/domains/oam_domain','/home/oracle/fmw/product/112/iamR2')

LOGGER intialised java.util.logging.Logger@133b514c

 ===============================================

STATUS: SUCCESS

The mbean xml files have been upgraded to new version.

The original mbean xml is saved in "<domain_directory>/output/upgrade".

Please restart the admin and oam servers.

 ===============================================

4) Upgrade OPSS: This step upgrades the OPSS with configuration and policy stores to PS3. It upgrades jps-config.xml and policy stores. 

upgradeOpss(jpsConfig=“/home/oracle/admin/user_projects/domains/oam_domain/config/fmwconfig/jps-config.xml”,jaznData=“/home/oracle/fmw/product/112/oracle_common/modules/oracle.jps_11.1.1/domain_config/system-jazn-data.xml”, auditStore=“/home/oracle/fmw/product/112/oracle_common/modules/oracle.jps_11.1.1/domain_config/audit-store.xml”, jdbcDriver=“oracle.jdbc.OracleDriver”, url=“jdbc:oracle:thin:@<db_host>:<db_port>/<sid>”, user=“<prefix>_OPSS”, password=“<opss_schema_password>”, upgradeJseStoreType=“true”)

5) Undeploying Coherence: This step un-deploys old coherence library, as with PS3, latest version of coherence gets deployed.

wls:/oam_domain/serverConfig> undeploy('coherence#3.7.1.1@3.7.1.1')

Undeploying application coherence#3.7.1.1@3.7.1.1 ...

<Feb 9, 2016 9:31:16 PM GMT> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating undeploy operation for application, coherence#3.7.1.1@3.7.1.1 [archive: null], to AdminServer oam_server1 .>

.Completed the undeployment of Application with status completed

Current Status of your Deployment:

Deployment command type: undeploy

Deployment State       : completed

Deployment Message     : [Deployer:149194]Operation 'remove' on application 'coherence [LibSpecVersion=3.7.1.1,LibImplVersion=3.7.1.1]' has succeeded on 'AdminServer'

6) Upgrading System Configuration:

a) Go to the directory ORACLE_HOME/common/script_handlers.

b) Remove all the .class files by running the following command:

c) rm *.class

Run the below command from IAM home by running wlst:

upgradeConfig("/home/oracle/admin/user_projects/domains/oam_domain", "sys", “<sys_password>”, “<prefix>_OAM", "jdbc:oracle:thin:@<db_host>:<db_port>/<service_name>”)

Feb 09, 2016 9:14:40 PM oracle.security.am.admin.config.util.GenericFactory locateMethod

INFO: Factory methods not declared for type: DecreaseDeviceStoreCacheRuntimeCacheSize.

Feb 09, 2016 9:14:40 PM oracle.security.am.admin.config.BasicFileConfigurationStore applyUpdates

INFO: Applying configuration update at path /. Current version of configuration 108. Change initiated on this node: true

Feb 09, 2016 9:14:40 PM oracle.security.am.upgrade.framework.psfe.PSFEFramework process

INFO: Plugin oracle.security.am.upgrade.framework.psfe.plugin.PSFEDefaultPlugin successfully processed and validated for featureID:                                 acheSize

Feb 09, 2016 9:14:41 PM oracle.security.am.admin.config.util.GenericFactory locateMethod

INFO: Factory methods not declared for type: VersionUpdate.

Feb 09, 2016 9:14:41 PM oracle.security.am.admin.config.BasicFileConfigurationStore applyUpdates

INFO: Applying configuration update at path /. Current version of configuration 109. Change initiated on this node: true

Feb 09, 2016 9:14:42 PM oracle.security.am.upgrade.framework.psfe.PSFEFramework process

INFO: System configurations have been successfully upgraded to version: 11.1.2.3.0

Restart the services and start exploring new OAM interface !

om.bea.security.providers.authorization.asi.InvocationException: ArmeRUNTIME Exception: null

Below stack trace of exception is seen after starting managed server in oam logs:

Oracle Access Manager - Version 11.1.2.0.0

<Feb 9, 2016 3:09:45 PM GMT> <Error> <oracle.jps.authorization.framework> <JPS-03156> <The exception has been thrown by ARME. The authorization result is set to deny.
com.bea.security.providers.authorization.asi.InvocationException: ArmeRUNTIME Exception: null
        at com.bea.security.providers.authorization.asi.AuthorizationProviderImpl.isAccessAllowed(AuthorizationProviderImpl.java:396)
        at com.bea.security.ssal.micro.MicroAuthorizationManagerWrapper.isAccessAllowed(MicroAuthorizationManagerWrapper.java:73)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed_internal(AuthorizationServiceImpl.java:914)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:745)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:668)
        at com.bea.security.impl.AuthorizationServiceImpl.isAccessAllowed(AuthorizationServiceImpl.java:622)
        at com.bea.security.AuthorizationService.isAccessAllowed(AuthorizationService.java:365)
        at oracle.security.am.common.policy.runtime.provider.oes.proxy.OESRuntimeProxy.wait4OESRuntimeDBPolicyRefreshCompletion(OESRuntimeProxy.java:263)
        at oracle.security.am.common.policy.runtime.provider.oes.proxy.OESRuntimeProxy.init(OESRuntimeProxy.java:193)
        at oracle.security.am.common.policy.runtime.provider.oes.OESPolicyRuntimeProvider.init(OESPolicyRuntimeProvider.java:167)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.getNewInstance(PolicyRuntimeFactory.java:162)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.init(PolicyRuntimeFactory.java:93)
        at oracle.security.am.common.policy.runtime.PolicyRuntimeFactory.getPolicyRuntime(PolicyRuntimeFactory.java:84)
        at oracle.security.am.common.policy.util.PolicyComponentLifecycle.initialize(PolicyComponentLifecycle.java:100)

As per Doc ID (1509559.1), these are harmless exceptions and can be ignored.

The policy store is not available; please see the log files for more details

After installing OAM 11gR2, you are getting error while loging into oamconsole.

You did the same mistake which I did of not reading the product documentation before starting with the installation.(http://docs.oracle.com/cd/E27559_01/install.1112/e27301/install.htm#INOAM98374)

There is an additional step in OAM 11gR2 (11.1.2.0) which says before starting up the servers, you need to configure the Security Store.

Follow the below steps to fix this error:

Solution
-----------
1) Delete the Domain
2) Remove the schema from RCU
3) Create the schema again.
4) Create the domain by running config.sh
5) Now the missed step, run the configSecurityStore as below:

$MIDDLWARE_HOME/oracle_common/common/bin/wlst.sh
$ORACLE_HOME/common/tools/configureSecurityStore.py -d $DOMAIN_HOME -c IAM -p <opss_schema_password> -m create

6) Start up the servers.

This should fix the issue.

webgate.so: undefined symbol: nzos_GetSessionMasterSecret

While starting ohs, I was getting below errors in ohs1.log after I configured Webgate 11.2.3.0 with OHS 11.1.1.7


ERROR
--------------

/home/oracle/fmw/products/111/ohs_117/ohs/bin/apachectl startssl: execing httpd
httpd.worker: Syntax error on line 1035 of /home/oracle/admin/ohs_instances/config/OHS/ohs1/httpd.conf: Syntax error on line 4 of /home/oracle/admin/ohs_instances/config/OHS/ohs1/webgate.conf: Cannot load /home/oracle/fmw/products/111/oamWebgateR2/webgate/ohs/lib/webgate.so into server: /home/oracle/fmw/products/111/oamWebgateR2/webgate/ohs/lib/webgate.so: undefined symbol: nzos_GetSessionMasterSecret

ISSUE
-----------

Above error is caused by using incompatible version of Webgate with corresponding OHS
Webgate version 11.1.2.3 requires OHS 11.1.1.9 version. I was on OHS 11.1.1.7

Solution
-------------

For OHS 11.1.1.7 version you have to use Webgate 11.1.2.2.