SF For Oracle RAC in an HP/UX Environment

I have been working with clusters since the mid-90’s.  Back then, for Oracle on Solaris, it was required to use Sun Cluster 2.0 for Oracle Parallel Server (OPS).  I was not a fan of Sun Cluster 2.x, it had many issues, and I was hesitant when I was introduced to Symantec VCS. After working with VCS I started to see its simplicity and its benefits.  I loved how easy it was to install, setup and manage.  I became very familiar with it in a Sun Solaris environment. Some of the benefits are:
  • Support for file system-based management. SF Oracle RAC provides a generic clustered file system technology for storing and managing Oracle data files as well as other application data.
  • Support for high-availability of cluster interconnects. The combination of LMX/LLT protocols and the PrivNIC/MultiPrivNIC agents provides maximum bandwidth as well as high availability of the cluster interconnects, including switch redundancy.
  • Use of clustered file system for placement of Oracle Cluster Registry and voting  disks. Clustered file system and volume management technologies provide  robust shared block and raw interfaces for placement of Oracle Cluster Registry and voting disks. In the absence of SF Oracle RAC, separate LUNs need to be configured for OCR and voting disks.
  • Support for a standardized approach toward application and database management. A single-vendor solution for the complete SF Oracle RAC software stack lets you devise a standardized approach toward application and database management. Further, administrators can apply existing expertise of Veritas technologies toward SF Oracle RAC.
  • Increased availability and performance using dynamic multi-pathing (DMP).   DMP provides wide storage array support for protection from failures and performance bottlenecks in the HBAs and SAN switches.
  • Easy administration and monitoring of SF Oracle RAC clusters from a single web console.
  • Support for many types of applications and databases.
  • Improved file system access times using Oracle Disk Manager (ODM).
  • Ability to configure ASM disk groups over CVM volumes to take advantage of dynamic multi-pathing (DMP).
  • Enhanced scalability and availability with access to multiple Oracle RAC  instances per database in a cluster.
  • Support for backup and recovery solutions using volume-level and file system-level snapshot technologies. SF Oracle RAC enables full volume-level snapshots for off-host processing and file system-level snapshots for efficient backup and rollback.
  • Ability to failover applications without downtime using clustered file system technology.
  • Prevention of data corruption in split-brain scenarios with robust SCSI-3 Persistent Reservation (PGR) based I/O fencing.
  • Support for sharing all types of files, in addition to Oracle database files, across nodes.
  • Fast disaster recovery with minimal downtime and interruption to users. Users can transition from a local high availability site to a wide-area disaster recovery  environment with primary and secondary sites. If a node fails, clients that are attached to the failed node can reconnect to a surviving node and resume access to the shared database. Recovery after failure in the SF Oracle RAC environment is far quicker than recovery for a failover database.
  • Verification of disaster recovery configuration using fire drill technology without affecting production systems.
  • Support for a wide range of hardware replication technologies as well as block-level replication using VVR.
Ok, now that we are past the marketing hoopla let us begin the journey.  Recently, I was asked to help a client install and configure SFRAC in an HP-UX environment.  I figured it would be a breeze because it is the same COTS product just a different platform.  Well, there were many trials and tribulations during the install but with regards to the platform and the main difference that you need to watch out for.  It all has to do with one line. In Solaris there is a library that must be copied prior to Oracle Clusterware installation:
  • cp /opt/VRTSvcs/rac/lib/libskgxn2_64.so /opt/ORCLcluster/lib/libskgxn2.so
The dilemma was how to do this in HP-UX.  The systems were HP-UX 11.23 and HW used was PA-RISC.  We did the same effort again on Itanium.  There install documentation did not have any information about how to do this and did mention relinking oracle after installation and upgrades but this library is a PRE-INSTALLATION requirement.  After much pain the secret “sauce” was found.  The steps in the install guide state: Before you run the root.sh script, you need to add the init.cssd.patch.
  1. Open another window on the system where you are running the installer.
  2. Log in as superuser.
  3. Change to the directory where the patch is to be copied:
  • # cd $CRS_HOME/css/admin
  • # cp /opt/VRTSvcs/rac/patch/init.cssd-10gR2.patch .
  1. Run the following command to install the patch:
  • # patch < init.cssd.patch init.cssd
  1. Run the root.sh script. For example:
  • # cd $CRS_HOME
  • # ./root.sh
This starts the CRS daemons on the node where you enter the command. There is a missing step.  Let’s call this step 4ATHIS IS A CRITICALLY IMPORTANT STEP, if you do not do it, Clusterware will NOT install.  So now for the sauce. 4A.   Modify init.cssd.  Find the section that starts with HP-UX and add section in bold.  This is the equivalent of copying the library in Solaris. HP-UX) MACH_HARDWARE=`/bin/uname -m` CLUSTERDIR=/opt/nmapi/nmapi2 if [ "$MACH_HARDWARE" = "ia64" ]; then SO_EXT=so NMAPIDIR_64=$CLUSTERDIR/lib/hpux64 else SO_EXT=sl NMAPIDIR_64=$CLUSTERDIR/lib/pa20_64 fi LD_LIBRARY_PATH=$ORA_CRS_HOME/lib:$NMAPIDIR_64:/usr/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH # Presence of this file indicates that vendor clusterware is installed SKGXNLIB=${NMAPIDIR_64}/libnmapi2.${SO_EXT} if [ -f $SKGXNLIB ]; then USING_VC=1 fi That should do.  As long as you are patched and the “cluvfy” returns “passed” on the checks, then Clusterware should install without a problem.  Enjoy!!