Amely'sOracleDBAblog

OPATCH apply on single instance Oracle Database having RAC setup

Step-1: Log in with Oracle user and take all the databases down on the server:

N.B.- Do not forget to take snapshot/details of all instances and listeners as you need to take them up also after completion of the activity.

login as: oracle
oracle@3.143.73.33's<mailto:oracle@3.143.73.33's> password:
Last login: Thu Apr 23 08:50:18 2015 from 3.143.68.50
[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep pmon
oracle 11771 10500 0 18:27 pts/5 00:00:00 grep pmon
grid 13576 1 0 Feb12 ? 00:23:25 asm_pmon_+ASM
oracle 13934 1 0 Feb12 ? 00:23:57 ora_pmon_drdev1
oracle 14067 1 0 Feb12 ? 00:29:33 ora_pmon_drpkdev2
oracle 14389 1 0 Feb12 ? 00:24:14 ora_pmon_drfibbredb

[oracle@TELYPHONYMEM01 ~]$ db_id
These Instance are Running on the Server
+ASM
drdev1
drpkdev2
drfibbredb
Enter SID Name which you want connect :
drdev1
SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 23 18:27:25 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

INSTANCE_NAME startup Time DB Name OPEN_MODE LOG_MODE SEQUENCE# Log Time
---------------- -------------------- --------- -------------------- ------------ ---------- --------------------
drdev1 12-FEB-15 20:32 PM drdev1 READ WRITE ARCHIVELOG 2555 22-APR-15 21:30 PM

SQL> show user
USER is "SYS"
SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
[oracle@TELYPHONYMEM01 ~]$ db_id
These Instance are Running on the Server
+ASM
drpkdev2
drfibbredb
Enter SID Name which you want connect :
drpkdev2
SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 23 18:28:06 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

INSTANCE_NAME startup Time DB Name OPEN_MODE LOG_MODE SEQUENCE# Log Time
---------------- -------------------- --------- -------------------- ------------ ---------- --------------------
drpkdev2 12-FEB-15 20:33 PM CWOPS READ WRITE ARCHIVELOG 6266 06-MAY-13 12:19 PM
SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> ^[[Aext ext" - rest of line ignored.
SQL> 042: unknown command "
SQL>
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

Step-2: Take the listeners down if you are using local listeners only (No need to and do not take SCAN listeners down)

[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep tns
root 135 2 0 Feb12 ? 00:00:00 [netns]
oracle 15077 1 0 Feb12 ? 00:03:35 /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER_drdev1 -inherit
oracle 29771 10500 0 18:29 pts/5 00:00:00 grep tns
[oracle@TELYPHONYMEM01 ~]$ lsnrctl
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 18:29:15
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Welcome to LSNRCTL, type "help" for information.
LSNRCTL> set current_listener LISTENER_drdev1
Current Listener is LISTENER_drdev1
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=3.143.73.33)(PORT=1875)))
STATUS of the LISTENER
------------------------
Alias LISTENER_drdev1
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 12-FEB-2015 20:37:43
Uptime 69 days 21 hr. 52 min. 2 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/TELYPHONYMEM01/listener_drdev1/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=3.143.73.33)(PORT=1875)))
Services Summary...
Service "drdev1" has 1 instance(s).
Instance "drdev1", status UNKNOWN, has 1 handler(s) for this service...
Service "drfibbredb" has 1 instance(s).
Instance "drfibbredb", status UNKNOWN, has 1 handler(s) for this service...
Service "masterdb" has 1 instance(s).
Instance "masterdb", status UNKNOWN, has 1 handler(s) for this service...
Service "drpkdev2" has 1 instance(s).
Instance "drpkdev2", status UNKNOWN, has 1 handler(s) for this service...
Service "seclrdb" has 1 instance(s).
Instance "seclrdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
LSNRCTL> stop
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=1.***.24.33)(PORT=1875))
The command completed successfully
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=1.***.24.33)(PORT=1875))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
LSNRCTL> exit
You have new mail in /var/spool/mail/oracle
[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep tns
root 135 2 0 Feb12 ? 00:00:00 [netns]
oracle 7021 10500 0 18:30 pts/5 00:00:00 grep tns

Step-3: Log in as Grid user and stop all cluster related resource:

[oracle@TELYPHONYMEM01 ~]$ su - grid
Password:

[grid@TELYPHONYMEM01 ~]$ cd $GRID_HOME
[grid@TELYPHONYMEM01 grid]$ cd bin

[grid@TELYPHONYMEM01 bin]$ crsctl stop resource -all
CRS-2500: Cannot stop resource 'ora.diskmon' as it is not running
CRS-2500: Cannot stop resource 'ora.ons' as it is not running
CRS-2673: Attempting to stop 'ora.cssd' on 'TELYPHONYMEM01'
CRS-2673: Attempting to stop 'ora.evmd' on 'TELYPHONYMEM01'
CRS-2677: Stop of 'ora.evmd' on 'TELYPHONYMEM01' succeeded
CRS-2677: Stop of 'ora.cssd' on 'TELYPHONYMEM01' succeeded
CRS-4000: Command Stop failed, or completed with errors.

[grid@TELYPHONYMEM01 bin]$ ps -ef|grep pmon
grid 5360 11185 0 18:33 pts/5 00:00:00 grep pmon
[grid@TELYPHONYMEM01 bin]$

[grid@TELYPHONYMEM01 bin]$ ps -ef|grep pmon
grid 5360 11185 0 18:33 pts/5 00:00:00 grep pmon

Step-4: Go to the patch directory where you have saved your patch after downloading from Oracle Support and unzip the patch if already not done:

[grid@TELYPHONYMEM01 bin]$ cd /u02/patch
[grid@TELYPHONYMEM01 patch]$ ls -ltr
total 5228
drwxr-xr-x 4 grid oinstall 4096 Jun 14 2013 APR2013_PSU
drwxr-xr-x 4 grid oinstall 4096 Jul 25 2013 JUL2013_PSU
drwxr-xr-x 4 grid oinstall 4096 Oct 19 2013 OCT2013_PSU
drwxr-xr-x 4 grid oinstall 4096 Jan 17 2014 JAN2014_PSU
drwxr-xr-x 5 grid oinstall 4096 Apr 23 2014 APR2014_PSU
drwxr-xr-x 4 grid oinstall 4096 Jul 22 2014 JUL2014_PSU
-rw-r--r-- 1 grid oinstall 5311908 Jul 30 2014 p14551959_112030_Linux-x86-64.zip
drwxr-xr-x 3 grid oinstall 4096 Jul 30 2014 arvind
drwxr-xr-x 4 grid oinstall 4096 Nov 19 16:08 OCT2014_PSU
drwxr-xr-x 4 grid oinstall 4096 Jan 23 19:30 JAN2015_PSU
drwxr-xr-x 4 grid oinstall 4096 Apr 23 18:22 APR2015_PSU
[grid@TELYPHONYMEM01 patch]$ cd APR2015_PSU
[grid@TELYPHONYMEM01 APR2015_PSU]$ ls -ltr
total 612360
-rw-r--r-- 1 grid oinstall 21 Jan 9 2014 README.txt
-rw-r--r-- 1 grid oinstall 0 Jan 9 2014 atp_lfp
drwxr-xr-x 5 grid oinstall 4096 Jan 9 2014 17592127
-rw-r--r-- 1 grid oinstall 450 Feb 24 16:13 bundle.xml
drwxrwxr-x 16 grid oinstall 4096 Mar 3 11:19 20299017
-rw-rw-r-- 1 grid oinstall 63766 Apr 13 13:33 README.html
-rw-rw-r-- 1 grid oinstall 143855 Apr 14 16:52 PatchSearch.xml
-rw-r--r-- 1 grid oinstall 626820215 Apr 23 08:52 p20485830_112030_Linux-x86-64.zip

Step-5: Apply the Opatch to grid home first:

In another session log in as root:

export PATH=$PATH:$HOME/bin:/u02/app/11.2.0/grid/OPatch

opatch auto /u02/patch/APR2015_PSU -ocmrf /u02/app/11.2.0/grid/OPatch/ocm/bin/ocm.rsp -och /u02/app/11.2.0/grid

keep checking log:

after completion :
[grid@TELYPHONYMEM01 APR2015_PSU]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u02/app/11.2.0/grid
Central Inventory : /u02/app/oraInventory
from : /u02/app/11.2.0/grid/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.3.0
Log file location : /u02/app/11.2.0/grid/cfgtoollogs/opatch/opatch2015-04-23_18-45-39PM_1.log
Lsinventory Output file location : /u02/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-04-23_18-45-39PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 11.2.0.3.0
There are 1 product(s) installed in this Oracle Home.

Interim patches (2) :
Patch 20299017 : applied on Thu Apr 23 18:43:39 IST 2015
Unique Patch ID: 18588265
Patch description: "Database Patch Set Update : 11.2.0.3.14 (20299017)"
Created on 2 Mar 2015, 21:45:38 hrs PST8PDT
Sub-patch 19769496; "Database Patch Set Update : 11.2.0.3.13 (19769496)"
Sub-patch 19121548; "Database Patch Set Update : 11.2.0.3.12 (19121548)"
Sub-patch 18522512; "Database Patch Set Update : 11.2.0.3.11 (18522512)"
Sub-patch 18031683; "Database Patch Set Update : 11.2.0.3.10 (18031683)"
Sub-patch 17540582; "Database Patch Set Update : 11.2.0.3.9 (17540582)"
Sub-patch 16902043; "Database Patch Set Update : 11.2.0.3.8 (16902043)"
Sub-patch 16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch 16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch 14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch 14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch 13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch 13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch 13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"

Step-6: Now apply the Opatch to Oracle Home (RDBMS):

Now apply patch to oracle home

as root user

export PATH=$PATH:$HOME/bin:/u01/app/oracle/product/11.2.0/dbhome_1/OPatch
opatch auto /u01/patch/APR2015_PSU -ocmrf /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/ocm/bin/ocm.rsp -oh /u01/app/oracle/product/11.2.0/dbhome_1

keep checking log

[grid@TELYPHONYMEM01 APR2015_PSU]$ su - oracle
Password:
[oracle@TELYPHONYMEM01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/11.2.0/dbhome_1
Central Inventory : /u02/app/oraInventory
from : /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-04-23_18-52-20PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-04-23_18-52-20PM.txt

Installed Top-level Products (1):
Oracle Database 11g 11.2.0.3.0
There are 1 product(s) installed in this Oracle Home.

Interim patches (3) :
Patch 20299017 : applied on Thu Apr 23 18:51:17 IST 2015
Unique Patch ID: 18588265
Patch description: "Database Patch Set Update : 11.2.0.3.14 (20299017)"
Created on 2 Mar 2015, 21:45:38 hrs PST8PDT
Sub-patch 19769496; "Database Patch Set Update : 11.2.0.3.13 (19769496)"
Sub-patch 19121548; "Database Patch Set Update : 11.2.0.3.12 (19121548)"
Sub-patch 18522512; "Database Patch Set Update : 11.2.0.3.11 (18522512)"
Sub-patch 18031683; "Database Patch Set Update : 11.2.0.3.10 (18031683)"
Sub-patch 17540582; "Database Patch Set Update : 11.2.0.3.9 (17540582)"
Sub-patch 16902043; "Database Patch Set Update : 11.2.0.3.8 (16902043)"
Sub-patch 16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch 16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch 14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch 14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch 13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch 13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch 13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Bugs fixed:
13593999, 10350832, 19433746, 14138130, 12919564, 14198511, 13561951
13588248, 13080778, 20134036, 13804294, 16710324, 18031683, 12873183
16992075, 14193240, 14472647, 12880299, 13369579, 14799269, 13840704

Step-7: Start ASM and all Clusterware resources:

[root@TELYPHONYMEM01 ~]# su - grid
[grid@TELYPHONYMEM01 ~]$ cd $GRID_HOME/bin

[grid@TELYPHONYMEM01 bin]$ crsctl start resource -all
check asm instance is up or not
was not up
so following steps:
[grid@TELYPHONYMEM01 bin]$ export ORACLE_SID=+ASM
[grid@TELYPHONYMEM01 bin]$ sqlplus
SQL*Plus: Release 11.2.0.3.0 Production on Thu Apr 23 18:56:51 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter user-name: /as sysasm
Connected to an idle instance.
SQL> startup
ASM instance started
Total System Global Area 1068937216 bytes
Fixed Size 2235208 bytes
Variable Size 1041536184 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL>
SQL>

Step-8: Startup all the databases and run catbundle.sql:

startup all databases.
as soon as db is up run following command in all databases
@?/rdbms/admin/catbundle.sql psu apply;

Step-9: Startup the listener:

take listener up:
------
start the listener and check connectivity:
[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep pmon
oracle 8346 32687 0 19:11 pts/5 00:00:00 grep pmon
grid 10609 1 0 18:57 ? 00:00:00 asm_pmon_+ASM
oracle 15758 1 0 19:09 ? 00:00:00 ora_pmon_drpkdev2
oracle 25144 1 0 19:02 ? 00:00:00 ora_pmon_drdev1
oracle 32441 1 0 19:10 ? 00:00:00 ora_pmon_drfibbredb
[oracle@TELYPHONYMEM01 ~]$ lsnrctl start LISTENER_drdev1
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 19:14:30
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Starting /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/TELYPHONYMEM01/listener_drdev1/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=1.***.24.33)(PORT=1875)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=1.***.24.33)(PORT=1875)))
STATUS of the LISTENER
------------------------
Alias LISTENER_drdev1
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 23-APR-2015 19:14:30
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/TELYPHONYMEM01/listener_drdev1/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=1.***.24.33)(PORT=1875)))
Services Summary...
Service "drdev1" has 1 instance(s).
Instance "drdev1", status UNKNOWN, has 1 handler(s) for this service...
Service "drfibbredb" has 1 instance(s).
Instance "drfibbredb", status UNKNOWN, has 1 handler(s) for this service...
Service "masterdb" has 1 instance(s).
Instance "masterdb", status UNKNOWN, has 1 handler(s) for this service...
Service "drpkdev2" has 1 instance(s).
Instance "drpkdev2", status UNKNOWN, has 1 handler(s) for this service...
Service "seclrdb" has 1 instance(s).
Instance "seclrdb", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep inherit
oracle 31053 1 0 19:14 ? 00:00:00 /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnslsnr LISTENER_drdev1 -inherit
oracle 31556 31637 0 19:14 pts/5 00:00:00 grep inherit

[oracle@TELYPHONYMEM01 ~]$ tnsping drpkdev2
TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 19:15:09
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 1.***.24.33)(PORT = 1875))) (CONNECT_DATA = (sid = drpkdev2)))
OK (0 msec)

[oracle@TELYPHONYMEM01 ~]$ tnsping drdev1
TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 19:15:28
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 1.***.24.33)(PORT = 1875))) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = drdev1)))
OK (0 msec)

[oracle@TELYPHONYMEM01 ~]$ tnsping drfibbredb
TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 19:15:44
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:
TNS-03505: Failed to resolve name
You have mail in /var/spool/mail/oracle

[oracle@TELYPHONYMEM01 ~]$ tnsping drfibbredb
TNS Ping Utility for Linux: Version 11.2.0.3.0 - Production on 23-APR-2015 19:16:07
Copyright (c) 1997, 2011, Oracle. All rights reserved.
Used parameter files:
TNS-03505: Failed to resolve name
[oracle@TELYPHONYMEM01 ~]$ ps -ef|grep pmon
grid 10609 1 0 18:57 ? 00:00:00 asm_pmon_+ASM
oracle 14130 32687 0 19:16 pts/5 00:00:00 grep pmon
oracle 15758 1 0 19:09 ? 00:00:00 ora_pmon_drpkdev2
oracle 25144 1 0 19:02 ? 00:00:00 ora_pmon_drdev1
oracle 32441 1 0 19:10 ? 00:00:00 ora_pmon_drfibbredb
[oracle@TELYPHONYMEM01 ~]$

--------------------------------------------------------------------------------------------------------------------------
================================================================================================================================================
-------------------------------------------------------------------------------------------------------------------------