Shutdown and Startup the Oracle RAC Database Manually
Posted by Mir Sayeed Hassan on May 14th, 2018
Shutdown and Startup the Oracle RAC Database Manually
=============================
Shutdown the RAC Database Manually
================================
The below scenario is tested in our test env, below are the steps as followed:
Check the status of database
[oracle@test-rac2 ~]$ !sq SQL> select status from v$instance; STATUS ------------ OPEN
Check the status of listener
[oracle@test-rac2 ~]$ lsnrctl status LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 14-MAY-2018 16:23:58 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production Start Date 14-MAY-2018 13:59:02 Uptime 0 days 2 hr. 24 min. 56 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/test-rac2/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.20.0.91)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.20.0.93)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "rac.localdomain" has 1 instance(s). Instance "rac2", status READY, has 1 handler(s) for this service... Service "racXDB.localdomain" has 1 instance(s). Instance "rac2", status READY, has 1 handler(s) for this service... The command completed successfully Check the status of the listener currently running [oracle@test-rac2 ~]$ srvctl status listener -n test-rac2 Listener LISTENER is enabled on node(s): test-rac2 Listener LISTENER is running on node(s): test-rac2
Stop the listener
[oracle@test-rac2 ~]$ srvctl stop listener -n test-rac2
Check the Status of the listener
[oracle@test-rac2 ~]$ srvctl status listener -n test-rac2 Listener LISTENER is enabled on node(s): test-rac2 Listener LISTENER is not running on node(s): test-rac2
Check the Status of the database
[oracle@test-rac2 ~]$ srvctl status database -d rac Instance rac1 is running on node test-rac1 Instance rac2 is running on node test-rac2
Stop the database
[oracle@test-rac2 ~]$ srvctl stop database -d rac
Verify the database stop status
[oracle@test-rac2 ~]$ srvctl status database -d rac Instance rac1 is not running on node test-rac1 Instance rac2 is not running on node test-rac2
Check the status of asm
[oracle@test-rac2 ~]$ srvctl status asm -n test-rac2 ASM is running on test-rac2
Stop the asm
[oracle@test-rac2 ~]$ srvctl stop asm -n test-rac2 PRCR-1014 : Failed to stop resource ora.asm PRCR-1065 : Failed to stop resource ora.asm CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified (I feel before stop the database, we need to stop the asm)
[oracle@test-rac2 ~]$ srvctl status nodeapps -n test-rac2 VIP test-rac2-vip is enabled VIP test-rac2-vip is running on node: test-rac2 Network is enabled Network is running on node: test-rac2 GSD is disabled GSD is not running on node: test-rac2 ONS is enabled ONS daemon is running on node: test-rac2
Stop the nodeapps of node
[oracle@test-rac2 ~]$ srvctl stop nodeapps -n test-rac2 PRCR-1014 : Failed to stop resource ora.net1.network PRCR-1065 : Failed to stop resource ora.net1.network CRS-2529: Unable to act on 'ora.net1.network' because that would require stopping or relocating 'ora.cvu', but the force option was not specified
Login to the asm to check the crsctl
[oracle@test-rac2 ~]$ . oraenv ORACLE_SID = [rac2] ? +ASM2 The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test-rac2 ~]$ crsctl check cluster -all ************************************************************** test-rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** test-rac2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online
[oracle@test-rac2 bin]$ exit logout
Login as root & stop the crs
[root@test-rac2 bin]# cd /u01/app/11.2.0/grid/bin
[root@test-rac2 bin]# pwd /u01/app/11.2.0/grid/bin
[root@test-rac2 bin]# ./crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'test-rac2' CRS-2673: Attempting to stop 'ora.crsd' on 'test-rac2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'test-rac2' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'test-rac2' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'test-rac2' CRS-2673: Attempting to stop 'ora.DATA.dg' on 'test-rac2' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'test-rac2' CRS-2673: Attempting to stop 'ora.oc4j' on 'test-rac2' CRS-2673: Attempting to stop 'ora.cvu' on 'test-rac2' CRS-2677: Stop of 'ora.cvu' on 'test-rac2' succeeded CRS-2672: Attempting to start 'ora.cvu' on 'test-rac1' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'test-rac2' CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.scan3.vip' on 'test-rac2' CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.scan2.vip' on 'test-rac2' CRS-2676: Start of 'ora.cvu' on 'test-rac1' succeeded CRS-2677: Stop of 'ora.scan1.vip' on 'test-rac2' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'test-rac1' CRS-2677: Stop of 'ora.scan2.vip' on 'test-rac2' succeeded CRS-2672: Attempting to start 'ora.scan2.vip' on 'test-rac1' CRS-2677: Stop of 'ora.scan3.vip' on 'test-rac2' succeeded CRS-2672: Attempting to start 'ora.scan3.vip' on 'test-rac1' CRS-2676: Start of 'ora.scan1.vip' on 'test-rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'test-rac1' CRS-2676: Start of 'ora.scan2.vip' on 'test-rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'test-rac1' CRS-2676: Start of 'ora.scan3.vip' on 'test-rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'test-rac1' CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'test-rac1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'test-rac1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'test-rac1' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'test-rac2' CRS-2677: Stop of 'ora.asm' on 'test-rac2' succeeded CRS-2677: Stop of 'ora.oc4j' on 'test-rac2' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'test-rac1' CRS-2674: Start of 'ora.oc4j' on 'test-rac1' failed CRS-2679: Attempting to clean 'ora.oc4j' on 'test-rac1' CRS-2681: Clean of 'ora.oc4j' on 'test-rac1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'test-rac2' CRS-2677: Stop of 'ora.net1.network' on 'test-rac2' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'test-rac2' has completed CRS-2677: Stop of 'ora.crsd' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'test-rac2' CRS-2673: Attempting to stop 'ora.crf' on 'test-rac2' CRS-2673: Attempting to stop 'ora.ctssd' on 'test-rac2' CRS-2673: Attempting to stop 'ora.evmd' on 'test-rac2' CRS-2673: Attempting to stop 'ora.asm' on 'test-rac2' CRS-2677: Stop of 'ora.crf' on 'test-rac2' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'test-rac2' succeeded CRS-2677: Stop of 'ora.evmd' on 'test-rac2' succeeded CRS-2677: Stop of 'ora.asm' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'test-rac2' CRS-2677: Stop of 'ora.ctssd' on 'test-rac2' succeeded CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'test-rac2' CRS-2677: Stop of 'ora.cssd' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'test-rac2' CRS-2677: Stop of 'ora.gipcd' on 'test-rac2' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'test-rac2' CRS-2677: Stop of 'ora.gpnpd' on 'test-rac2' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'test-rac2' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@test-rac2 bin]# ./crsctl check cluster -all CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Check failed, or completed with errors.
Verify the database & listener status
[oracle@test-rac2 ~]$ !sq SQL> exit Disconnected
[oracle@test-rac2 ~]$ lsnrctl status LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 14-MAY-2018 16:51:21 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 111: Connection refused
===========Hence successfully shutdown the RAC Database Manually==========
======================
Startup the database manually
======================
In Starting up the RAC Database, below are the steps as followed.
Login as the root & start the crs process
[root@test-rac2 bin]# cd /u01/app/11.2.0/grid/bin
[root@test-rac2 bin]# ./crsctl start crs CRS-4123: Oracle High Availability Services has been started.
[root@test-rac2 bin]# ./crsctl check cluster -all ************************************************************** test-rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** test-rac2: CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4534: Cannot communicate with Event Manager
As per the above error “CRS-4535: Cannot communicate with Cluster Ready Services” or “CRS-4534: Cannot communicate with Event Manager”, You need to Wait 5 minutes & then check again with “crsctl check cluster -all” command, Hopefully it should work in case if it’s not started then you can start the ora.crsd process to resolve this issue as shown below:
In my scenario, I waited for 5 minutes & check, therefore its started automatically as shown below
[root@test-rac2 bin]# ./crsctl check cluster -all ************************************************************** test-rac1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** test-rac2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online *************************************************************
Or
[root@test-rac2 bin]# ./crsctl start res ora.crsd –init
This will resolve if it’s not started automatically
The below information give the complete details of the cluster status
[root@test-rac2 bin]# ps -ef | grep d.bin root 26828 1 1 17:03 ? 00:00:09 /u01/app/11.2.0/grid/bin/ohasd.bin reboot oracle 26981 1 1 17:03 ? 00:00:08 /u01/app/11.2.0/grid/bin/oraagent.bin oracle 26992 1 0 17:03 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin oracle 27003 1 0 17:03 ? 00:00:05 /u01/app/11.2.0/grid/bin/gpnpd.bin oracle 27013 1 1 17:03 ? 00:00:07 /u01/app/11.2.0/grid/bin/gipcd.bin root 27016 1 1 17:03 ? 00:00:10 /u01/app/11.2.0/grid/bin/orarootagent.bin root 27027 1 2 17:03 ? 00:00:11 /u01/app/11.2.0/grid/bin/osysmond.bin root 27040 1 0 17:03 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdmonitor root 27054 1 0 17:03 ? 00:00:02 /u01/app/11.2.0/grid/bin/cssdagent oracle 27071 1 1 17:03 ? 00:00:06 /u01/app/11.2.0/grid/bin/ocssd.bin root 27144 1 0 17:04 ? 00:00:02 /u01/app/11.2.0/grid/bin/octssd.bin reboot oracle 27195 1 0 17:04 ? 00:00:03 /u01/app/11.2.0/grid/bin/evmd.bin root 27206 1 2 17:04 ? 00:00:10 /u01/app/11.2.0/grid/bin/ologgerd -m test-rac1 -r -d /u01/app/11.2.0/grid/crf/db/test-rac2 root 27332 1 2 17:05 ? 00:00:09 /u01/app/11.2.0/grid/bin/crsd.bin reboot oracle 27438 27195 0 17:06 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log root 27515 1 1 17:06 ? 00:00:06 /u01/app/11.2.0/grid/bin/orarootagent.bin oracle 27525 1 0 17:06 ? 00:00:03 /u01/app/11.2.0/grid/bin/oraagent.bin oracle 27632 1 0 17:06 ? 00:00:00 /u01/app/11.2.0/grid/bin/scriptagent.bin oracle 27708 1 0 17:06 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit oracle 27726 1 0 17:06 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit oracle 27728 1 0 17:06 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit root 28454 6820 0 17:12 pts/1 00:00:00 grep d.bin
Now login in to the asm
[oracle@test-rac2 ~]$ . oraenv ORACLE_SID = [rac2] ? +ASM2 The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test-rac2 ~]$ srvctl status nodeapps -n test-rac2 VIP test-rac2-vip is enabled VIP test-rac2-vip is not running Network is enabled Network is running on node: test-rac2 GSD is disabled GSD is not running on node: test-rac2 ONS is enabled ONS daemon is running on node: test-rac2
[oracle@test-rac2 ~]$ srvctl start nodeapps -n test-rac2 PRKO-2421 : Network resource is already started on node(s): test-rac2 PRKO-2420 : VIP is already started on node(s): test-rac2 PRKO-2422 : ONS is already started on node(s): test-rac2
[oracle@test-rac2 ~]$ srvctl status nodeapps -n test-rac2 VIP test-rac2-vip is enabled VIP test-rac2-vip is running on node: test-rac2 Network is enabled Network is running on node: test-rac2 GSD is disabled GSD is not running on node: test-rac2 ONS is enabled ONS daemon is running on node: test-rac2
[oracle@test-rac2 ~]$ srvctl start asm -n test-rac2 PRCC-1015 : asm was already running on test-rac2 PRCR-1004 : Resource ora.asm is already running
[oracle@test-rac2 ~]$ srvctl status asm -n test-rac2 ASM is running on test-rac2
Verify the asm
[oracle@test-rac2 ~]$ !sq sQL> select status from v$instance; STATUS ------------ STARTED
[oracle@test-rac2 ~]$ . oraenv ORACLE_SID = [+ASM2] ? +ASM2 The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test-rac2 ~]$ asmcmd
ASMCMD> ls DATA/
Start the database by using the oracle user
[oracle@test-rac2 ~]$ srvctl start database -d rac
Verify the database status
[oracle@test-rac2 ~]$ srvctl status database -d rac Instance rac1 is running on node test-rac1 Instance rac2 is running on node test-rac2
Check the status of the listener
[oracle@test-rac2 ~]$ srvctl status listener -n test-rac2 Listener LISTENER is enabled on node(s): test-rac2 Listener LISTENER is running on node(s): test-rac2 Incase if its not started, fallow the below command to start the listener
[oracle@test-rac2 ~]$ srvctl start listener -n test-rac2 PRCC-1015 : LISTENER was already running on test-rac2 PRCR-1004 : Resource ora.LISTENER.lsnr is already running
Verify the database & listener
[oracle@test-rac2 ~]$ !sq SQL> select status from v$instance; STATUS ------ OPEN
[oracle@test-rac2 ~]$ lsnrctl status LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 14-MAY-2018 17:29:48 Copyright (c) 1991, 2013, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) STATUS of the LISTENER ------------------------ Alias LISTENER Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production Start Date 14-MAY-2018 17:21:40 Uptime 0 days 0 hr. 8 min. 8 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/test-rac2/listener/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.20.0.91)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.20.0.93)(PORT=1521))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status READY, has 1 handler(s) for this service... Service "rac.localdomain" has 1 instance(s). Instance "rac2", status READY, has 1 handler(s) for this service... Service "racXDB.localdomain" has 1 instance(s). Instance "rac2", status READY, has 1 handler(s) for this service... The command completed successfully