Exadata Bundle Patch Bug on 11.2.0.3.18ExadataDatabase

One of our team member has shared resolution of bug while patching on Exadata storage server.

Below is the full story

System landscape:

Exadata Machine = Exadata 1/8th Rack X3-2 machine
Platform = Linux x86-64 
Product = Oracle Database 
Version = 11.2.0.3.18ExadataDatabase 

Here we are doing prechek to apply bundle patch on Exadata. ((Quarterly Full Stack Download Patch (QFSDP) Oct 2014)

Go to the patch directory and first execute cleanup to clean previous patch manager utility.

Here patch directory for storage cell is "/bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/"

#./patchmgr -cells /root/cellgroup -cleanup                                                     
2014-12-06 12:01:05 +0800        :Working: DO: Cleanup ...
2014-12-06 12:01:45 +0800        :SUCCESS: DONE: Cleanup

Executing pre-check through patchmgr utility 

#./patchmgr -cells ~/cellgroup -patch_check_prereq -rolling

2014-12-06 12:09:47 +0800        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-12-06 12:10:24 +0800        :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-12-06 12:10:24 +0800        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2014-12-06 12:10:38 +0800 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2014-12-06 12:10:40 +0800        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2014-12-06 12:10:40 +0800        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-12-06 12:12:36 +0800        :SUCCESS: DONE: Check prerequisites on all cells.
2014-12-06 12:12:36 +0800        :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-12-06 12:12:36 +0800 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stdout.
2014-12-06 12:12:36 +0800 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.
2014-12-06 12:12:55 +0800 :WARNING: ACTION REQUIRED: Required fix for bug 17854520 not found in dbhome /u01/app/oracle/product/11.2.0.3/dbhome_1 on dbadm01. Install the fix before performing rolling cell patching.
2014-12-06 12:12:55 +0800 :WARNING: ACTION REQUIRED: Required fix for bug 17854520 not found in dbhome /u01/app/oracle/product/11.2.0.3/dbhome_1 on dbadm02. Install the fix before performing rolling cell patching.
2014-12-06 12:12:55 +0800 :WARNING: Patchmgr plugin complete: Prereq check failed - fix for bug 17854520 is required
2014-12-06 12:12:56 +0800        :FAILED: Details in files <cell_name>.log /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stdout, /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stderr
2014-12-06 12:12:56 +0800        :FAILED: DONE: Execute plugin check for Patch Check Prereq.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

As per above output, pre-check has not been completed successfully and we need to apply bug fix 17854520 on Grid and RDBMS home.


Now let's do the exercise to resolve this issue.



Step:1


You have to raise SR to download this bug fix because when you go to download above bug fix on Oracle support portal, it will ask for password and which can be provided by Oracle on SR.

Below snaps are for your reference.





Step:2

Download bug fix

Step:3


Transfer downloaded file to Exadata and perform required steps to apply patch i.e. Unzip, Owner change for respective database and grid home

Step:4


Go through the Readme and apply bug fix to Grid and RDBMS home as per given instruction.

Step:5

Execute patchmgr cleanup to clean previous patch manager utility

#./patchmgr -cells /root/cellgroup -cleanup

2014-12-06 13:15:57 +0800        :Working: DO: Cleanup ...
2014-12-06 13:15:59 +0800        :SUCCESS: DONE: Cleanup

Step:6

Again execute precheck and check whether it is giving error or not

#./patchmgr -cells ~/cellgroup -patch_check_prereq -rolling

2014-12-06 13:27:56 +0800        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2014-12-06 13:28:26 +0800        :SUCCESS: DONE: Initialize files, check space and state of cell services.
2014-12-06 13:28:26 +0800        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2014-12-06 13:28:41 +0800 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2014-12-06 13:28:42 +0800        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2014-12-06 13:28:42 +0800        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2014-12-06 13:30:36 +0800        :SUCCESS: DONE: Check prerequisites on all cells.
2014-12-06 13:30:36 +0800        :Working: DO: Execute plugin check for Patch Check Prereq ...
2014-12-06 13:30:36 +0800 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /bpatch/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch/patch_11.2.3.3.1.140708/patchmgr.stdout.
2014-12-06 13:30:36 +0800 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.
2014-12-06 13:30:55 +0800 :SUCCESS: Patchmgr plugin complete: Prereq check passed - no exposure to bug 17854520
2014-12-06 13:30:56 +0800        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.

Now your system is ready for patching


We will share couple of Exadata patching articles for all the Exadata components soon.

No comments:

Post a Comment