ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []

You might face this error while starting Database.

SQL> startup
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2261368 bytes
Variable Size            3254783624 bytes
Database Buffers         6341787648 bytes
Redo Buffers               21692416 bytes
Database mounted.
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [],
[], [], [], [], [], [], [], []


Reason: It is due to server crash might be due to hardware or power failure.

Solution:

Recover the database by following the below steps:

Step 1) Shutdown the instance and open in mount state:

SQL>shutdown immediate;
ORACLE instance shut down.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 9620525056 bytes
Fixed Size                  2261368 bytes
Variable Size            3254783624 bytes
Database Buffers         6341787648 bytes
Redo Buffers               21692416 bytes
Database mounted.

Step 2) Recover Database

SQL> recover database;

Media recovery complete.

Step 3) Open the Database

SQL> alter database open;

Database altered.

Step 4) Shutdown and startup normally

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup

ORACLE instance started.

Total System Global Area 9620525056 bytes

Fixed Size                  2261368 bytes

Variable Size            3254783624 bytes

Database Buffers         6341787648 bytes

Redo Buffers               21692416 bytes

Database mounted.

Database opened.

IDM Node Installation Overview as part of Fusion Application

Fusion Applications Installation - Part 1

Fusion Applications Installation comprises of 2 parts

Part 1) IDM node Installation

Part 2) FA node Installation

Node 1	idmhost	1. Identity and Access Management components (OID, OVD, OIF, OAM, OIM, SOA) 2. Oracle Database for IDM
Node 2	fahost	1. Fusion Applications Domains 2. Oracle Database for Fusion Apps

Hardware Requirements:

IDM Node:

Memory: Minimum 16GB

Disk Size – 300GB

FA Node:

Memory Requirement: 140GB

Disk Size – 500GB

We will use Oracle Linux x86-64 Operating System for Installation. Software can be downloaded from http://edelivery.oracle.com

1)   Downloading the Installation media

Login to http://edelivery.oracle.com

Download first 17 parts of installable:

2)   Create a staging directory

Unzip all the files in single directory.

More on this will be covered in next post !

Fusion Applications Product and Product offerings

Each product in Fusion Applications is deployed as a Domain. Each domain has J2EE applications which provides certain functionality. These functionality or features provided by the J2EE applications can be considered as product offerings.

A product family is collection of one or more product offerings.

During installation of Fusion Applications you will get the below screen, where you can select a product or product offerings as per your requirement.

Please note, once selected it cannot be changed later during/after installation, so be very sure before you click on Next button.

A brief description of various Product Family as below:

Product Family	Product Offering
Oracle Fusion Customer Relationship Management	Marketing Sales Customer Data Hub Enterprise Contracts
Oracle Fusion Financials	Financials Accounting Hub
Oracle Fusion Human Capital Management	Workforce Deployment Workforce Development Compensation Management
Oracle Fusion Procurement	Procurement
Oracle Fusion Supply Chain Management	•Product Management •Order Orchestration •Material Management and Logistics
Oracle Fusion Project Portfolio Management	Projects

Oracle Fusion Applications- Basic Architecture

Oracle Fusion Applications are built on Fusion Middleware stack.

A very basic deployment architecture can be depicted by the below figure. It will consist of two nodes:

Basic 2 Node Architecture

1. IDM Node
It is used for User and Role Management. All the user and privileges of FA are stored in IDM node. It performs below functions:

IDM Node

- Authentication, Authorization and auditing mechanisms
- Roles and entitlements
- Directory Services
- User Provisioning
- Policy Stores
- Session Management
- Single Sign on / Federation


2) FA Node
It will include best in class Product Families like:
- Oracle Fusion Financials
- Oracle Fusion HCM
- Oracle Fusion Procurement
- Oracle Fusion SCM
- Oracler Fusion CRM

FA Node

Read more on Product and Product offerings in next Post

Test Database Connectivity from Weblogic Server to the Database

This post helps us to establish connectivity to the database from the Weblogic Server.

Go to MWHOME location

cd <WEBLOGIC HOME>/server/bin
set the environment
. ./setWLSEnv.sh

Run the below to test the connectivity.

java utils.dbping ORACLE_THIN <DB_USER_NAME> <DB_PASSWORD> <DB_HOST>:<DB_PORT>:<SID>

This will test the connection and display the results.

The result would be as below for a successful connection :

java utils.dbping ORACLE_THIN <DB_USER_NAME> <DB_PASSWORD> <DB_HOST>:<DB_PORT>:<SID>

**** Success!!! ****

You can connect to the database in your app using:

  java.util.Properties props = new java.util.Properties();
  props.put("user", "<DB_USER>");
  props.put("password", "<PASSWORD>");
  java.sql.Driver d =
    Class.forName("oracle.jdbc.OracleDriver").newInstance();
  java.sql.Connection conn =
    Driver.connect("<DB_USER>", props);

Mac Love

After years of waiting and dreaming I was able to get my love home. It was a life changing experience, so beautiful, full of colors and sound it was. 
Like a new bride it was sitting on my bed, I was too scared to do anything as I did not know how it will react to my touch.
Finally I gathered courage to open it, it prompted me to enter name, and that was the moment when I realized it's mine now.
Amit's Mac pro !! :) :)

JMS Client could not establish connection with weblogic server

Issue : 

When an external application tries to establish connection to the weblogic JMS server, the connection does not seem to work.

We observe the below error in the logs :


weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]] for weblogic.jms.C:vmohsitut151:od8:-419rn532v3nc
        at weblogic.jms.frontend.FEConnectionFactory.connectionCreateInternal(FEConnectionFactory.java:371)
        at weblogic.jms.frontend.FEConnectionFactoryImpl.connectionCreateInternal(FEConnectionFactoryImpl.java:79)
        at weblogic.jms.frontend.FEConnectionFactoryImpl.connectionCreateRequest(FEConnectionFactoryImpl.java:62)
        at weblogic.jms.frontend.FEConnectionFactoryImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
        at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
        at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
        at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
        at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: weblogic.messaging.dispatcher.DispatcherException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]] for weblogic.jms.C:vmohsitut151:od8:-419rn532v3nc
        at weblogic.messaging.dispatcher.DispatcherWrapperState.addPeerGoneListener(DispatcherWrapperState.java:412)
        at weblogic.messaging.dispatcher.DispatcherWrapperState.<init>(DispatcherWrapperState.java:79)
        at weblogic.messaging.dispatcher.DispatcherManager.dispatcherAdd(DispatcherManager.java:222)
        at weblogic.messaging.dispatcher.DispatcherManager.addDispatcherReference(DispatcherManager.java:118)
        at weblogic.jms.dispatcher.JMSDispatcherManager.addDispatcherReference(JMSDispatcherManager.java:187)
        at weblogic.jms.frontend.FEConnectionFactory.connectionCreateInternal(FEConnectionFactory.java:368)
        ... 12 more
Caused by: weblogic.rmi.extensions.DisconnectMonitorUnavailableException: Could not register a DisconnectListener for [Delegate(253822924) [weblogic.iiop.IOR[RMI:weblogic.messaging.dispatcher.DispatcherImpl:0000000000000000] @<ip addres>:48508, <0, null>]]
        at weblogic.rmi.extensions.DisconnectMonitorListImpl.addDisconnectListener(DisconnectMonitorListImpl.java:83)
        at weblogic.messaging.dispatcher.DispatcherWrapperState.addPeerGoneListener(DispatcherWrapperState.java:409)
        ... 17 more



Observations made : 

This issue does not occur if the source client ( from where the connection to WLS JMS is made ) and Weblogic server are in the same network subnet.

The above issue has been observed under two conditions :

--> If these 2 systems are on different network
or
--> If the two systems are on different subnet in the same network

Solution : 

On the client side, Try to find out which weblogic jar file is being used to make the connection.

If the wlclient.jar file is used for making the connection, then this is the root cause of the issue. Instead of using the wlclient.jar, please use wlthint3client.jar and then try to make a connection to the weblogic JMS.

In our case, wlclient.jar file was being used and we replaced it with wlthint3client.jar

This resolved the issue.

After making the changes, our application file which made connection to JMS looked like below :

#$clp="${clp}:${lib_dir}/wlclient.jar";

$clp="${clp}:${lib_dir}/wlthint3client.jar";

Step by Step process to Install OSB : Oracle Service Bus 12C ( 12.1.3.0.0 version )

This post helps us to install OSB 12.1.3.0.0.

Before proceeding further make sure Weblogic 12.1.3.0.0 is installed. If not, please click here to install the required weblogic server.

Once the installation of weblogic server is completed, please follow the below steps to install the OSB 12.1.3.0.0

The installer to be used for OSB installation is : fmw_12.1.3.0.0_osb.jar and the required installer can be downloaded from edelivery.oracle.com

Execute the below command to start the OSB installer.

<Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_osb.jar

For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\OSB\fmw_12.1.3.0.0_osb.jar

This will initiate the installer. Please proceed as per the steps below.

Click Next on the above screen. Installation location screen would appear. Provide the location where Weblogic Server is installed.

Click Next on the above screen. Installation Type screen would appear. Select Service Bus and click next.

Pre-Requisites would be checked in the below screen. Take appropriate action if any issue is reported. If no issue is reported, Click Next

Summary screen would appear. Analyze the information and click next

Installation progress would be displayed in the below screen

Once done, click finish to close the installer

This completes the installation of OSB : Oracle Service Bus installation ( 12C : 12.1.3.0.0 version )

Step by Step process to Install SOA 12C ( 12.1.3.0.0 version )

This post helps us to install SOA 12.1.3.0.0.

Before proceeding further make sure Weblogic 12.1.3.0.0 is installed. If not, please click here to install the required weblogic server.

Once the installation of weblogic server is completed, please follow the below steps to install the SOA 12.1.3.0.0

The installer to be used for SOA installation is : fmw_12.1.3.0.0_soa.jar and the required installer can be downloaded from edelivery.oracle.com

Execute the below command to start the SOA installer.

<Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_soa.jar

For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\SOA\fmw_12.1.3.0.0_soa.jar

This will initiate the installer. Please proceed as per the steps below.

Click Next on the above screen. In the next screen, you would be prompted for Oracle Home. Please provide the location where Weblogic Server is installed and Click Next

Please select the SOA Suite in the below screen and Click Next

Pre-Requisites would be checked in the below screen. Take appropriate action if any issue is reported. If no issue is reported, Click Next

Summary screen would appear. Analyze the information and click next

Installation progress would be displayed in the below screen

Once done, click next on the below screen

Once done, click finish to close the installer

This completes the installation of SOA 12C ( 12.1.3.0.0 version )

Step by Step process to Install Weblogic 12C ( 12.1.3.0.0 version )

This post helps us to install Weblogic Server 12C (12.1.3.0.0 version ). In the long run this weblogic server home would be used to install SOA, OSB components.

Please use the installer : fmw_12.1.3.0.0_infrastructure.jar for installation of Weblogic server. If this is not used, then it might lead to multiple issues while installing SOA with respect to JRF libraries at a later point of time.

The required installer can be downloaded from edelivery.oracle.com

Below are the steps to be followed for installing WLS 12c version.

1.  Install JDK : jdk-7u67-windows-x64 (Java SE Development Kit ). The version should be : java version "1.7.0_67"
2. Install Weblogic Server.

Below are the steps to be followed for Weblogic Installation.

Run the command as below :

 <Location where JDK is installed>/bin/java -jar <Location where WLS installer is located>/fmw_12.1.3.0.0_infrastructure.jar
For Ex :
D:\MWHOME_12C\jdk1.7.0_67\bin\java -jar D:\All_Installers\OFMW\12C_Version\Infrastructure\V44416-01\fmw_12.1.3.0.0_infrastructure.jar

This will start the WLS installer. Follow the steps as per the below screen shots

Click Next. In the below screen provide Oracle Home to the location where Weblogic server needs to be installed. Then Click Next

Select Fusion Middleware Infrastructure and Click Next

In the below screen, pre-requisites would be checked. Take appropriate action, if any issue is reported in the below screen

The below step can be skipped by not providing any email address.

Click Next and installation process would begin as shown below.

Once done, review the below information and cick finish to close the installer as per below screen shot.

This will install the weblogic server at the location provided in the installer screen. Navigate to that location and check if the binaries have been installed.

We will be updating additions posts regarding the Installation and Configuration of SOA, OSB 12.1.3 version and the steps for domain creation.

Opatch: LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo : OPatch failed with error code 73

Problem

While running opatch lsinventory, command fails with the below error:

Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Possible causes are:

   Oracle Home dir. path does not exist in Central Inventory

   Oracle Home is a symbolic link

   Oracle Home inventory is corrupted

LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73

Solution

Attach the Oracle Home to the Central Inventory:

$ cd $ORACLE_HOME/oui/bin 

$ ./attachHome.sh

Now check, command should work fine.

$ORACLE_HOME/OPatch/opatch lsinventory

How to change EM Agent Port ?

EM Agent uses a specific port to receive requests. This port is defined during installation. We faced an issue while installing OID and ODS, both have EMAGENT component:

Processes in Instance: oid1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
oid1                             | oidldapd           |   31821 | Alive    | 1389237283 |  1472620 |  21:24:59 | N/A
oid1                             | oidldapd           |   31790 | Alive    | 1389237282 |   415084 |  21:25:00 | N/A
oid1                             | oidmon             |   31754 | Alive    | 1389237281 |   559676 |  21:25:00 | LDAPS:4040,LDAP:4041
EMAGENT                          | EMAGENT            |   31755 | Alive    | 1389237280 |   106148 |  21:25:00 | N/A



Processes in Instance: ods1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
EMAGENT                          | EMAGENT            |   30455 | Down    |  436375769 |   106148 |   | N/A


If by mistake you give same port to EMAGENT during installation of OID and ODS then it will give you an exception of "Address already in use or bind exception" while starting the component. EMAGENT can be seen as down in one of the instance.

Solution

You have to change EMAGENT port for one of the component.

Steps

1. Stop the EMAgent.
$ $AGENT_HOME/bin/emctl stop agent

In our case $AGENT_HOME=/home/ajNode1/instances/ods1/EMAGENT/EMAGENT

2. Modify the $AGENT_HOME/sysman/config/emd.properties file in the EMAgent home directory:
    For example:
    EMD_URL=http://ajmachine1:5011/emd/main

    Modify the <port number> in the EMD_URL property so the EMAgent uses a new unused port on the target host.
    EMD_URL=http://ajmachine1:5015/emd/main

3. Start the EMAgent.
$ $AGENT_HOME/bin/emctl start agent

Error while creating ovd component on second node

Issue

While doing cluster installation of Oracle Virtual Directory (OVD), we faced this issue. When you run config.sh from $ORACLE_IDM_HOME/bin, it goes successful and creates a ovd component, however it register it as ovd1 instead of ovd2 on second node.

Hence, folder structure created is something like this for 2 nodes:

on Node1: OVD1 server: /home/ajnode1/instances/ovdinst1/config/OVD/ovd1

on Node2: OVD2 server: /home/ajnode2/instances/ovdinst2/config/OVD/ovd1

Due to same name of ovd instance on both nodes, it creates a confusion and it is difficult to monitor.

Solution

It is a bug, and only solution is to manually create the component on second node.

a) Stop ovd2:

/home/ajnode2/instances/ovdinst2/bin/opmnctl stopall

b) Delete the component:

/home/ajnode2/instances/ovdinst2/bin/opmnctl deletecomponent -componentName ovd1

c) Create a new OVD component and name it ovd2 (fill the values accoring to your environment

-bash-4.1$ ./opmnctl createcomponent -adminHost <admin_host> -adminPort <admin_port> -adminUsername <weblogic> -componentName ovd2 -componentType OVD -admin cn=orcladmin -isAdminSSL true -ovdAdminPort <ovd_admin_port> -namespace <dc=your company,dc=com> -ldapPort <Non SSL Port> -ldapSport <SSL Port>

Command requires login to weblogic admin server (localhost):
  Username: weblogic
  Password:


Creating empty component directories...Done
Provisioning OVD files for ovd2
  OVD creation in progress
  Creating OVD specific directories
  Copying Configuration Files
  Registering OVD Ports
  Updating OVD Admin Password
OVD Admin Password:

  Setting CSF values
  Updating Configuration Files
  Adding opmn Snippet
  Setting proxy mbean information
Registering ovd2 component
Command succeeded.


d) Restart the ovd server on second node.

How to find the version of Fusion Application Installed

To find the Fusion Application Version Installed we can query the database by giving the following query:

SQL> select release_name from fusion.ad_product_groups;

RELEASE_NAME
--------------------------------------------------
11.1.8.0.0

Fusion Applications Vs Fusion Middleware

There is a lot of confusion in people regarding Fusion Applications and Fusion Middleware.
Suddenly ORACLE FUSION has become the buzz word. Everyone is talking about it and many wants to get trained or aquire this niche skill.

Oracle has invested billions of dollar in the research and development of Fusion Applications.
Fusion Applications is next generation ERP. It is going to replace EBS R12. As far as I am aware, there won't be any new development or new release of Oracle EBS. There is a phase shift happening towards Fusion Applications. Most of the customers are already using it on SAAS (Software As A Service) environment. Oracle has recently launced the latest version of FA (Fusion Applicattions) for on-premise customers as well.

Following products are offered in Fusion Applications:
1) Oracle Fusion Customer Relationship Management
2) Oracle Fusion Financials
3) Oracle Fusion Governance
4) Oracle Fusion Human Capital Management
5) Oracle Fusion Procurement
6) Oracle Fusion Project Portfolio Management
7) Oracle Fusion Supply Chain Management
8) Oracle Fusion Setup

Fusion Applications is build on top of Fusion Middleware Stack.
It is based on Service-Oriented Architecture (SOA) and includes Application Development Framework (ADF), WebCenter, Business Intelligence (BI), Content Management (UCM/ECM) and Identity Management (IdM).

Fusion Middleware consist of an Application Server (Weblogic) and other technology stack like SOA, BI, Identity Managagement, Content Management. It is a Middleware like Tibco, which is used to integrate applications. With OFM (Oracle Fusion Middleware), we can develop, model business processes, integrate them with various business partners using out of the box plugins or adapters.

Fusion Applications are organized as Pillars where Pillar is standalone subset of Oracle Fusion Applications. In terms of OFMW, these pillars can be related to Domain.

Each Product of FA, is deployed as a Domain. A domain contains multiple J2EE applications related to that product. For example if we consider, Human Capital Managment, it will have all the applications like HCM Absence Administration, HCM Workforce management etc deployed in the HCM domain and targeted on different managed servers of the domain which will be serving the request.

Each product family is based on a common Oracle Fusion Applications data model and services. Product families contain one or more Java EE applications that are specific to Oracle Fusion Applications and deployed to Oracle WebLogic Server.

Oracle Fusion Application 11.1.9 is now available

Oracle has launched the latest version of Fusion Applications which is 11.1.9 (11.1.9.2). It can be downloaded from e-delievery.
Installation steps are almost similar compared to the previous version 11.1.8.

Please see Installation Guide for reference.

Do leave a comment if you need any help in Fusion Apps Installation.

Weblogic Startup error: PS-01050: Opening of wallet based credential store failed

Problem
Following errors are observed while starting AdminServer of OIM Domain

Info: Data source is: opss-DBDS
Feb 01, 2015 9:00:37 PM oracle.security.jps.internal.credstore.ssp.CsfWalletManager openWallet
WARNING: Opening of wallet based credential store failed. Reason java.io.IOException: java.io.FileNotFoundException: /tmp/pki_data1537025641.lck.tmp (Permission denied)
Feb 01, 2015 9:00:37 PM oracle.security.jps.internal.config.util.BootstrapConfigurationUtil getLdapBootstrapCredential
SEVERE: Ldap bootstrap credential retrieval failed. Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOEx
ception: java.io.FileNotFoundException: /tmp/pki_data1537025641.lck.tmp (Permission denied).
Feb 01, 2015 9:00:37 PM oracle.security.jps.internal.credstore.ldap.LdapCredentialStore getBootstrapCredentials
SEVERE: JPS-00071: Ldap bootstrap credential retrieval failed. Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason j
ava.io.IOException: java.io.FileNotFoundException: /tmp/pki_data1537025641.lck.tmp (Permission denied).
oracle.security.jps.JpsException: JPS-00071: Ldap bootstrap credential retrieval failed. Reason:oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credent
ial store failed. Reason java.io.IOException: java.io.FileNotFoundException: /tmp/pki_data1537025641.lck.tmp (Permission denied).
        at oracle.security.jps.internal.config.util.BootstrapConfigurationUtil.getLdapBootstrapCredential(BootstrapConfigurationUtil.java:567)
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore$1.run(LdapCredentialStore.java:144)
        at oracle.security.jps.internal.credstore.ldap.LdapCredentialStore$1.run(LdapCredentialStore.java:142)
        at java.security.AccessController.doPrivileged(Native Method)'

Solution

Check permission on /tmp directory.

chmod -R 777 /tmp