Apply Patching Oracle Database 19C release Update on OCT 2024 19.25 (p36916690) On Grid Infrastructure Standalone Server.
Posted by Mir Sayeed Hassan on February 12th, 2025
Apply Patching Oracle Database 19C release Update on OCT 2024 19.25 (p36916690) On Grid Infrastructure Standalone Server.
Check the status of Database
[oracle@Paymentdb ~]$ sqlplus / as sysdba SQL*Plus: Release 19.0.0.0.0 - Production on Tue Feb 12 14:27:08 2025 Version 19.23.0.0.0 Copyright (c) 1982, 2023, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.23.0.0.0 SQL> select instance_name, version, open_mode from V$database, v$instance; INSTANCE_NAME VERSION OPEN_MODE --------------------------------------------- PAYMENTDB 19.0.0.0.0 READ WRITE
Prior to this apply the OPATCH on GRID
To Apply the Opatch at GRID & Database., Click on below link:
APPLY THE LATEST OPTACH 6880880 (12.2.0.1.45) on Grid Infrastructure 19C DATABASE
GRID OPATCH APPLY:
Download the Patch from below link
https://support.oracle.com/epmos/faces/PatchSearchResults?_adf.ctrl-state=ym4c1dobw_120&_afrLoop=189225645173189
Create a directory & place the downloaded Patch into it.
[oracle@Paymentdb ~]$ cd /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/
[oracle@Paymentdb GI_RU_19.25.0.0.0_Patch]$ ll total 3290916 -rwxrwxrwx. 1 oracle oinstall 3369893688 Jan 18 13:38 p36916690_190000_Linux-x86-64.zip
Unzip GRID Patch
[oracle@Paymentdb GI_RU_19.25.0.0.0_Patch]$ unzip p36916690_190000_Linux-x86-64.zip
[oracle@Paymentdb GI_RU_19.25.0.0.0_Patch]$ ls 36916690 p36916690_190000_Linux-x86-64.zip PatchSearch.xml
[oracle@Paymentdb GI_RU_19.25.0.0.0_Patch]$ cd 36916690/
[oracle@Paymentdb 36916690]$ ls 36758186 36912597 36917397 36917416 36940756 automation bundle.xml README.html README.txt
Note: Here you can see the sub-multiple patches are exist., You need to apply for each & every patch set as shown below:
To Apply the GRID Patch you need to shutdown the ASM Instance
Stop the ASM Instance
[oracle@Paymentdb ~]$ srvctl status asm ASM is running on Paymentdb
[oracle@Paymentdb ~]$ srvctl stop asm PRCR-1065 : Failed to stop resource ora.asm CRS-2974: unable to act on resource 'ora.asm' on server 'Paymentdb' because that would require stopping or relocating resource 'ora.DATA.dg' but the appropriate force flag was not specified
Note: The resolve the above error & stop the ASM forcefully., use the -f option into it.
[oracle@Paymentdb ~]$ srvctl stop asm -f
[oracle@Paymentdb ~]$ srvctl status asm ASM is not running.
Therefore the ASM Instance is stop successfully.
Check teh Pre-requisite before apply the patch
[oracle@Paymentdb ~]$ cd /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/
[oracle@Paymentdb 36916690]$ ls 36758186 36912597 36917397 36917416 36940756 automation bundle.xml README.html README.txt
Note: You need to check the Pre-requiste for all the individual sub-patches as below above.
[oracle@Paymentdb 36916690]$ cd 36758186
[oracle@Paymentdb 36758186]$ /u01/app/19c/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./ Oracle Interim Patch Installer version 12.2.0.1.45 Copyright (c) 2025, Oracle Corporation. All rights reserved. PREREQ session Oracle Home : /u01/app/oracle/product/19c/dbhome_1 Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/19c/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.45 OUI version : 12.2.0.7.0 Log file location : /u01/app/oracle/product/19c/dbhome_1/cfgtoollogs/opatch/opatch2025-02-11_14-46-40PM_1.log Invoking prereq "checkconflictagainstohwithdetail" Prereq "checkConflictAgainstOHWithDetail" passed. OPatch succeeded.
Note: I have checked for all sub patch number given above., all the pre-requites are OPatch are Succeeded.
Start Applying the Patch on GRID Infrastructure for all the sub patches individual.
[oracle@Paymentdb ~]$ cd /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690
[oracle@Paymentdb 36916690]$ ls 36758186 36912597 36917397 36917416 36940756 automation bundle.xml README.html README.txt
Here you need to apply the Individual SUB-PATCHES as shown below.
Note: Incase if you get the below error while applying the sub patches., Use the below method to resolve and re-run the patch.
ERROR: fuser not found error
Solution
[oracle@Paymentdb db_1]$ touch fuser [root@Paymentdb 35940989]# export OPATCH_NO_FUSER=true [root@Paymentdb 35940989]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19C_19.22/GI_RU_19.22.0.0.0_06-JAN-2024/35940989/33575402/ -oh /u01/app/19c/grid/
Successfully executed.
If the ths error occur : 2024/05/26 18:30:44 CLSRSC-180: An error occurred while executing the command ‘pstree -sp 58410 |’
Solution: install the below RPM Package as per recomendation. & Re-run.
[root@Paymentdb 35940989]# yum install psmisc -y
Apply 1st Sub-Patch
[root@Paymentdb 36758186]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36758186/ -oh /u01/app/19c/grid/ OPatchauto session is initiated at Tue Feb 12 14:55:50 2025 System initialization log file is /u01/app/19c/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-11_02-55-54PM.log. Session log file is /u01/app/19c/grid/cfgtoollogs/opatchauto/opatchauto2025-02-11_02-56-01PM.log The id for this session is 5XFB Executing OPatch prereq operations to verify patch applicability on home /u01/app/19c/grid Patch applicability verified successfully on home /u01/app/19c/grid Executing patch validation checks on home /u01/app/19c/grid Patch validation checks successfully completed on home /u01/app/19c/grid Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_02-56-13PM.log CRS service brought down successfully on home /u01/app/19c/grid Start applying binary patch on home /u01/app/19c/grid Binary patch applied successfully on home /u01/app/19c/grid Running rootadd_rdbms.sh on home /u01/app/19c/grid Successfully executed rootadd_rdbms.sh on home /u01/app/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/19c/grid Postpatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_02-56-45PM.log CRS service started successfully on home /u01/app/19c/grid OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:Paymentdb SIHA Home:/u01/app/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36758186 Log: /u01/app/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-11_14-56-31PM_1.log OPatchauto session completed at Tue Feb 12 14:58:07 2025 Time taken to complete the session 2 minutes, 17 seconds
Apply 2nd Sub-Patch
[root@Paymentdb 36912597]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36912597/ -oh /u01/app/19c/grid/ OPatchauto session is initiated at Tue Feb 12 14:59:34 2025 System initialization log file is /u01/app/19c/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-11_02-59-43PM.log. Session log file is /u01/app/19c/grid/cfgtoollogs/opatchauto/opatchauto2025-02-11_02-59-50PM.log The id for this session is 659G Executing OPatch prereq operations to verify patch applicability on home /u01/app/19c/grid Patch applicability verified successfully on home /u01/app/19c/grid Executing patch validation checks on home /u01/app/19c/grid Patch validation checks successfully completed on home /u01/app/19c/grid Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-01-23PM.log CRS service brought down successfully on home /u01/app/19c/grid Start applying binary patch on home /u01/app/19c/grid Binary patch applied successfully on home /u01/app/19c/grid Running rootadd_rdbms.sh on home /u01/app/19c/grid Successfully executed rootadd_rdbms.sh on home /u01/app/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/19c/grid Postpatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-08-35PM.log CRS service started successfully on home /u01/app/19c/grid OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:Paymentdb SIHA Home:/u01/app/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36912597 Log: /u01/app/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-11_15-02-17PM_1.log OPatchauto session completed at Tue Feb 12 15:09:59 2025 Time taken to complete the session 10 minutes, 17 seconds
Apply 3rd Sub-Patch
[root@Paymentdb 36917397]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36917397/ -oh /u01/app/19c/grid/ OPatchauto session is initiated at Tue Feb 12 15:11:26 2025 System initialization log file is /u01/app/19c/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-11_03-11-31PM.log. Session log file is /u01/app/19c/grid/cfgtoollogs/opatchauto/opatchauto2025-02-11_03-11-37PM.log The id for this session is DGCR Executing OPatch prereq operations to verify patch applicability on home /u01/app/19c/grid Patch applicability verified successfully on home /u01/app/19c/grid Executing patch validation checks on home /u01/app/19c/grid Patch validation checks successfully completed on home /u01/app/19c/grid Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-12-18PM.log CRS service brought down successfully on home /u01/app/19c/grid Start applying binary patch on home /u01/app/19c/grid Binary patch applied successfully on home /u01/app/19c/grid Running rootadd_rdbms.sh on home /u01/app/19c/grid Successfully executed rootadd_rdbms.sh on home /u01/app/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/19c/grid Postpatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-14-06PM.log CRS service started successfully on home /u01/app/19c/grid OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:Paymentdb SIHA Home:/u01/app/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36917397 Log: /u01/app/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-11_15-13-00PM_1.log OPatchauto session completed at Tue Feb 12 15:15:57 2025 Time taken to complete the session 4 minutes, 31 seconds
Apply 4th Sub-Patch
[root@Paymentdb 36917416]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36917416/ -oh /u01/app/19c/grid/ OPatchauto session is initiated at Tue Feb 12 15:17:01 2025 System initialization log file is /u01/app/19c/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-11_03-17-06PM.log. Session log file is /u01/app/19c/grid/cfgtoollogs/opatchauto/opatchauto2025-02-11_03-17-12PM.log The id for this session is Q3JB Executing OPatch prereq operations to verify patch applicability on home /u01/app/19c/grid Patch applicability verified successfully on home /u01/app/19c/grid Executing patch validation checks on home /u01/app/19c/grid Patch validation checks successfully completed on home /u01/app/19c/grid Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-17-55PM.log CRS service brought down successfully on home /u01/app/19c/grid Start applying binary patch on home /u01/app/19c/grid Binary patch applied successfully on home /u01/app/19c/grid Running rootadd_rdbms.sh on home /u01/app/19c/grid Successfully executed rootadd_rdbms.sh on home /u01/app/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/19c/grid Postpatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-19-52PM.log CRS service started successfully on home /u01/app/19c/grid OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:Paymentdb SIHA Home:/u01/app/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36917416 Log: /u01/app/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-11_15-18-40PM_1.log OPatchauto session completed at Tue Feb 12 15:22:11 2025 Time taken to complete the session 5 minutes, 11 seconds
Apply 5th Sub-Patch
On this patch we got the below error.,
Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-23-49PM.log Failed to bring down CRS service on home /u01/app/19c/grid Execution of [SIHAShutDownAction] patch action failed, check log for more details. Failures: Patch Target : Paymentdb->/u01/app/19c/grid Type[siha] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/19c/grid, host: Paymentdb. Command failed: bash -c "LD_LIBRARY_PATH=/u01/app/19c/grid/lib:/u01/app/19c/grid/srvm/lib:/u01/app/19c/grid/lib:/u01/app/19c/grid//lib:/u01/app/19c/grid//srvm/lib /u01/app/19c/grid/perl/bin/perl -I/u01/app/19c/grid/perl/lib -I/u01/app/19c/grid/opatchautocfg/db/dbtmp/bootstrap_Paymentdb/patchwork/crs/install -I/u01/app/19c/grid/opatchautocfg/db/dbtmp/bootstrap_Paymentdb/patchwork/xag /u01/app/19c/grid/opatchautocfg/db/dbtmp/bootstrap_Paymentdb/patchwork/crs/install/roothas.pl -prepatch" Command failure output: Using configuration parameter file: /u01/app/19c/grid/opatchautocfg/db/dbtmp/bootstrap_Paymentdb/patchwork/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-23-49PM.log 2025/02/11 15:23:49 CLSRSC-180: An error occurred while executing the command 'pstree -sp 17473 |' After fixing the cause of failure Run opatchauto resume OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed. bash: zip: command not found opatchauto unable to backup /u01/app/19c/grid/cfgtoollogs/opatchauto bash: zip: command not found opatchauto unable to backup /u01/app/19c/grid/cfgtoollogs/opatchautodb OPatchauto session completed at Tue Feb 12 15:23:50 2025 Time taken to complete the session 0 minute, 53 seconds opatchauto failed with error code 42
Solution: Install rpm packages as per there requirement & resume it.
[root@Paymentdb ~]# yum install zip [root@Paymentdb ~]# yum install psmisc
Resume or apply again above 5th sub-patch
[root@Paymentdb 36940756]# /u01/app/19c/grid/OPatch/opatchauto apply /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36940756/ -oh /u01/app/19c/grid/ OPatchauto session is initiated at Tue Feb 12 15:40:18 2025 System initialization log file is /u01/app/19c/grid/cfgtoollogs/opatchautodb/systemconfig2025-02-11_03-40-23PM.log. Session log file is /u01/app/19c/grid/cfgtoollogs/opatchauto/opatchauto2025-02-11_03-40-29PM.log The id for this session is ZHNV Executing OPatch prereq operations to verify patch applicability on home /u01/app/19c/grid Patch applicability verified successfully on home /u01/app/19c/grid Executing patch validation checks on home /u01/app/19c/grid Patch validation checks successfully completed on home /u01/app/19c/grid Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19c/grid Prepatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-41-10PM.log CRS service brought down successfully on home /u01/app/19c/grid Start applying binary patch on home /u01/app/19c/grid Binary patch applied successfully on home /u01/app/19c/grid Running rootadd_rdbms.sh on home /u01/app/19c/grid Successfully executed rootadd_rdbms.sh on home /u01/app/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/19c/grid Postpatch operation log file location: /u01/app/oracle/crsdata/Paymentdb/crsconfig/hapatch_2025-02-11_03-43-23PM.log CRS service started successfully on home /u01/app/19c/grid OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:Paymentdb SIHA Home:/u01/app/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /u02/source/PATCH_19.25/GI_RU_19.25.0.0.0_Patch/36916690/36940756 Log: /u01/app/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2025-02-11_15-42-20PM_1.log OPatchauto session completed at Tue Feb 12 15:45:16 2025 Time taken to complete the session 4 minutes, 58 seconds
Therefore., applied all the sub-patches successfully.
START THE ASM INSTANCE & VERIFY
Start the ASM Instance
[oracle@Paymentdb ~]$ srvctl start asm
Check the Status of ASM Instance
[oracle@Paymentdb ~]$ srvctl status asm ASM is running on Paymentdb
Login to ASM Instance & Verify the ASM Disks
[oracle@Paymentdb ~]$ . oraenv ORACLE_SID = [PAYMENTDB] ? +ASM The Oracle base remains unchanged with value /u01/app/oracle
[oracle@Paymentdb ~]$ asmcmd
ASMCMD> lsdg State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 512 4096 4194304 153592 153492 0 153492 0 N ARCH/ MOUNTED EXTERN N 512 512 4096 4194304 4194288 4146044 0 4146044 0 N DATA/ MOUNTED EXTERN N 512 512 4096 4194304 10236 10112 0 10112 0 N REDO1/ MOUNTED EXTERN N 512 512 4096 4194304 10236 10148 0 10148 0 N REDO2/
Check the ASM Databas Instance
[oracle@Paymentdb ~]$ sqlplus / as sysasm SQL*Plus: Release 19.0.0.0.0 - Production on Tue Feb 12 15:50:29 2025 Version 19.25.0.0.0 Copyright (c) 1982, 2024, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.25.0.0.0 SQL> select status from v$instance; STATUS -------- STARTED
Verified the GRID Patch applied.
[oracle@Paymentdb ~]$ echo $GRID_HOME /u01/app/19c/grid
[oracle@Paymentdb ~]$ cd $GRID_HOME/OPatch
Check the applied list of patches
[oracle@Paymentdb ~]$ cd $GRID_HOME/OPatch [oracle@Paymentdb OPatch]$ ./opatch lspatches 36940756;TOMCAT RELEASE UPDATE 19.0.0.0.0 (36940756) 36917416;OCW RELEASE UPDATE 19.25.0.0.0 (36917416) 36917397;ACFS RELEASE UPDATE 19.25.0.0.0 (36917397) 36912597;Database Release Update : 19.25.0.0.241015 (36912597) 36758186;DBWLM RELEASE UPDATE 19.0.0.0.0 (36758186)
OPatch succeeded.
Verify with LSINVENTORY
[oracle@Paymentdb OPatch]$ ./opatch lsinventory [oracle@Paymentdb OPatch]$ ./opatch lsinventory Oracle Interim Patch Installer version 12.2.0.1.45 Copyright (c) 2025, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/19c/grid Central Inventory : /u01/app/oraInventory from : /u01/app/19c/grid/oraInst.loc OPatch version : 12.2.0.1.45 OUI version : 12.2.0.7.0 Log file location : /u01/app/19c/grid/cfgtoollogs/opatch/opatch2025-02-11_15-52-02PM_1.log Lsinventory Output file location : /u01/app/19c/grid/cfgtoollogs/opatch/lsinv/lsinventory2025-02-11_15-52-02PM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: Paymentdb ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Grid Infrastructure 19c 19.0.0.0.0 There are 1 products installed in this Oracle Home. CLID: P Interim patches (5) : Patch 36940756 : applied on Tue Feb 12 15:43:12 IRST 2025 Unique Patch ID: 25783343 Patch description: "TOMCAT RELEASE UPDATE 19.0.0.0.0 (36940756)" Created on 12 Aug 2024, 03:44:56 hrs PST8PDT Bugs fixed: 29286300, 30009095, 30729044, 30997459, 31657943, 32107716, 32248633 32625073, 33121445, 33655429, 33846688, 34300543, 34519419, 34816344 34970057, 35229376, 35538576, 35779899, 35926707, 36066696, 36440449 36473821, 36822984 Patch 36917416 : applied on Tue Feb 12 15:19:35 IRST 2025 Unique Patch ID: 25869972 Patch description: "OCW RELEASE UPDATE 19.25.0.0.0 (36917416)" Created on 10 Oct 2024, 00:18:41 hrs PST8PDT Bugs fixed: 13742922, 17787434, 19847322, 23320062, 24611963, 24677421, 24943214 25642320, 25736599, 25738437, 26305045, 26496292, 26615554, 26615581 ------------------------------------------------------------------------------- OPatch succeeded.
====Hence tested & verified in our test env====