How to add the ASM DISK into Existing disks in RAC Database on Oracle 11gR2
Posted by Mir Sayeed Hassan on April 25th, 2020
How to add the ASM DISK into Existing disks in RAC DB Server
In the below scenario, we are going to add the new asm disk into the existing disk in RAC DB Server
First Scan the newly added disk into the server
[root@rac1-testdb~]# echo "- - -"> /sys/class/scsi_host/host0/scan [root@rac1-testdb ~]# echo "- - -"> /sys/class/scsi_host/host1/scan [root@rac1-testdb ~]# echo "- - -"> /sys/class/scsi_host/host2/scan [root@rac1-testdb ~]# echo "- - -"> /sys/class/scsi_host/host3/scan
Get the below information from Network team as newly disk added into the server
EU231-ORL131 360060160bfdb4d006266285eeab56b26 EU231-ORL133 360060160bfdb4d006366285e2c6addca EU231-ORL135 360060160bfdb4d006566285e7f04016e EU231-ORL137 360060160bfdb4d006966285edf87edac EU232-ORL132 360060160f8794d00df97285e3b64b4dc EU232-ORL134 360060160f8794d00e197285eb1a9e5a8 EU232-ORL136 360060160f8794d00e297285e92bae42f EU232-ORL138 360060160f8794d00e497285ee405530f
Crosscheck
[root@rac1-testdb~]# multipath -l | grep 6b26 EU231-ORL131 (360060160bfdb4d006266285eeab56b26) dm-1 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep ddca EU231-ORL133 (360060160bfdb4d006366285e2c6addca) dm-0 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep 016e EU231-ORL135 (360060160bfdb4d006566285e7f04016e) dm-3 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep edac EU231-ORL137 (360060160bfdb4d006966285edf87edac) dm-2 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep b4dc EU232-ORL132 (360060160f8794d00df97285e3b64b4dc) dm-4 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep e5a8 EU232-ORL134 (360060160f8794d00e197285eb1a9e5a8) dm-6 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep e42f EU232-ORL136 (360060160f8794d00e297285e92bae42f) dm-5 DGC,VRAID [root@rac1-testdb~]# multipath -l | grep 530f EU232-ORL138 (360060160f8794d00e497285ee405530f) dm-7 DGC,VRAID
Now Add the newly added Multipath location into the Multipath configuration file as shown the given path
vi /etc/multipath.conf
multipath {
wwid 360060160bfdb4d006266285eeab56b26
alias EU231-ORL131
}
multipath {
wwid 360060160bfdb4d006366285e2c6addca
alias EU231-ORL133
}
multipath {
wwid 360060160bfdb4d006566285e7f04016e
alias EU231-ORL135
}
multipath {
wwid 360060160bfdb4d006966285edf87edac
alias EU231-ORL137
}
multipath {
wwid 360060160f8794d00df97285e3b64b4dc
alias EU232-ORL132
}
multipath {
wwid 360060160f8794d00e197285eb1a9e5a8
alias EU232-ORL134
}
multipath {
wwid 360060160f8794d00e297285e92bae42f
alias EU232-ORL136
}
multipath {
wwid 360060160f8794d00e497285ee405530f
alias EU232-ORL138
}
Reload the Multipath
[root@rac2-testdb~]# service multipathd reload Reloading multipathd: [ OK ]
Note: Wait for sometime & issue the below command
Verify the multipath added
[root@rac2-testdb~]# multipath -l | grep ORL [root@rac2-testdb~]# multipath -l | grep ORL EU231-ORL133 (360060160bfdb4d006366285e2c6addca) dm-0 DGC,VRAID EU232-ORL138 (360060160f8794d00e497285ee405530f) dm-7 DGC,VRAID EU231-ORL131 (360060160bfdb4d006266285eeab56b26) dm-1 DGC,VRAID EU232-ORL136 (360060160f8794d00e297285e92bae42f) dm-5 DGC,VRAID EU231-ORL127 (360060160bfdb4d004bb8c65d12a9dc81) dm-481 DGC,VRAID EU232-ORL134 (360060160f8794d00e197285eb1a9e5a8) dm-6 DGC,VRAID EU231-ORL125 (360060160bfdb4d004ab8c65da3317cea) dm-472 DGC,VRAID EU232-ORL132 (360060160f8794d00df97285e3b64b4dc) dm-4 DGC,VRAID EU231-ORL137 (360060160bfdb4d006966285edf87edac) dm-2 DGC,VRAID EU232-ORL128 (360060160f8794d0090e8c65d9bae5d96) dm-647 DGC,VRAID EU232-ORL130 (360060160f8794d0094e8c65dc4937b11) dm-651 DGC,VRAID EU231-ORL135 (360060160bfdb4d006566285e7f04016e) dm-3 DGC,VRAID
Reload the Multipath on Another RAC Node & also verify
[root@rac1-testdb~]# service multipathd reload Reloading multipathd: [ OK ]
[root@rac2-testdb~]# multipath -l | grep ORL ----- Here you see existing disk + new disk EU231-ORL133 (360060160bfdb4d006366285e2c6addca) dm-0 DGC,VRAID EU232-ORL138 (360060160f8794d00e497285ee405530f) dm-7 DGC,VRAID EU231-ORL131 (360060160bfdb4d006266285eeab56b26) dm-1 DGC,VRAID EU232-ORL136 (360060160f8794d00e297285e92bae42f) dm-5 DGC,VRAID EU231-ORL127 (360060160bfdb4d004bb8c65d12a9dc81) dm-481 DGC,VRAID EU232-ORL134 (360060160f8794d00e197285eb1a9e5a8) dm-6 DGC,VRAID EU231-ORL125 (360060160bfdb4d004ab8c65da3317cea) dm-472 DGC,VRAID EU232-ORL132 (360060160f8794d00df97285e3b64b4dc) dm-4 DGC,VRAID EU231-ORL137 (360060160bfdb4d006966285edf87edac) dm-2 DGC,VRAID EU232-ORL128 (360060160f8794d0090e8c65d9bae5d96) dm-647 DGC,VRAID EU232-ORL130 (360060160f8794d0094e8c65dc4937b11) dm-651 DGC,VRAID EU231-ORL135 (360060160bfdb4d006566285e7f04016e) dm-3 DGC,VRAID
Now Start the format of the disk into One of the RAC Node
Below disk are newly added into the server
EU231-ORL131 360060160bfdb4d006266285eeab56b26 EU231-ORL133 360060160bfdb4d006366285e2c6addca EU231-ORL135 360060160bfdb4d006566285e7f04016e EU231-ORL137 360060160bfdb4d006966285edf87edac EU232-ORL132 360060160f8794d00df97285e3b64b4dc EU232-ORL134 360060160f8794d00e197285eb1a9e5a8 EU232-ORL136 360060160f8794d00e297285e92bae42f EU232-ORL138 360060160f8794d00e497285ee405530f
Fallow the below method to format all the above disk as shown below
Note: Here i have discuss one disk, you need to do it for all the newly added disks
[root@rac1-testdb~]# fdisk /dev/mapper/EU231-ORL131
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xff9f2b40.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6527, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527):
Using default value 6527
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
Scan those disks individually
[root@rac1-testdb~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks...
[root@rac1-testdb~]# oracleasm listdisks
Note: You will see existing disk & check the newly disk added into it (Wait for sometime to update the newly disk added)
Create the Oracle ASM Disks
[root@rac1-testdb~]#oracleasm listdisks
[root@rac1-testdb~]# oracleasm createdisk EU231-ORL131 /dev/mapper/EU231-ORL131p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU231ORL133 /dev/mapper/EU231-ORL133p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU231ORL135 /dev/mapper/EU231-ORL135p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU231ORL137 /dev/mapper/EU231-ORL137p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU232ORL132 /dev/mapper/EU232-ORL132p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU232ORL134 /dev/mapper/EU232-ORL134p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU232ORL136 /dev/mapper/EU232-ORL136p1 Writing disk header: done Instantiating disk: done [root@rac1-testdb~]# oracleasm createdisk EU232ORL138 /dev/mapper/EU232-ORL138p1 Writing disk header: done Instantiating disk: done
List the ASM Disks added & verify
[root@rac1-testdb~]#oracleasm listdisks EU231ORL131 EU231ORL133 EU231ORL135 EU231ORL137 EU232ORL132 EU232ORL134 EU232ORL136 EU232ORL138
On another node SCAN DISKS & Verify
[root@rac2-testdb~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "EU231ORL131" Instantiating disk "EU231ORL133" Instantiating disk "EU231ORL135" Instantiating disk "EU231ORL137" Instantiating disk "EU232ORL132" Instantiating disk "EU232ORL134" Instantiating disk "EU232ORL136" Instantiating disk "EU232ORL138"
After the allocation of disks into “+ORL” Disks, Verify by issuing the query at database level
SQL> select group_number,disk_number,state,name,path,mount_status,header_status from V$asm_disk; 0 8 NORMAL /dev/oracleasm/disks/EU231ORL131 CLOSED PROVISIONED 0 7 NORMAL /dev/oracleasm/disks/EU231ORL133 CLOSED PROVISIONED 0 6 NORMAL /dev/oracleasm/disks/EU231ORL135 CLOSED PROVISIONED 0 5 NORMAL /dev/oracleasm/disks/EU231ORL137 CLOSED PROVISIONED 0 4 NORMAL /dev/oracleasm/disks/EU232ORL132 CLOSED PROVISIONED 0 3 NORMAL /dev/oracleasm/disks/EU232ORL134 CLOSED PROVISIONED 0 2 NORMAL /dev/oracleasm/disks/EU232ORL136 CLOSED PROVISIONED 0 1 NORMAL /dev/oracleasm/disks/EU232ORL138 CLOSED PROVISIONED
SQL> select name from v$asm_diskgroup; NAME ------------------------------------------------------------------------------------------ ARCH CLUSTERDG ORADATA ORL
Now login to the ASM as sys / as sysasm & add the newly added disk into failover
[oracle(+ASM1)@rac1-testdb~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Thu Jan 23 14:12:20 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter diskgroup ORL add failgroup E231 disk '/dev/oracleasm/disks/EU231ORL131' NOFORCE; Diskgroup altered. Check the rebalance status SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 3632 12154 11465 0 Wait for some time until its gets completed the rebalance & then verify SQL> select * from v$asm_operation; no rows selected ======================== SQL> alter diskgroup ORL add failgroup E231 disk '/dev/oracleasm/disks/EU231ORL133' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 2398 10675 11338 0 SQL> select * from v$asm_operation; no rows selected ======================== SQL> alter diskgroup ORL add failgroup E231 disk '/dev/oracleasm/disks/EU231ORL135' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 754 13299 9653 1 SQL> select * from v$asm_operation; no rows selected ========================= SQL> alter diskgroup ORL add failgroup E231 disk '/dev/oracleasm/disks/EU231ORL137' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 1964 11382 10280 0 SQL> select * from v$asm_operation; no rows selected ========================== SQL> alter diskgroup ORL add failgroup E232 disk '/dev/oracleasm/disks/EU232ORL132' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 36 11351 5923 1 SQL> select * from v$asm_operation; no rows selected ========================== SQL> alter diskgroup ORL add failgroup E232 disk '/dev/oracleasm/disks/EU232ORL134' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 177 9936 6567 1 SQL> select * from v$asm_operation; no rows selected ========================== SQL> alter diskgroup ORL add failgroup E232 disk '/dev/oracleasm/disks/EU232ORL136' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 1091 10135 9395 0 SQL> select * from v$asm_operation; no rows selected ========================== SQL> alter diskgroup ORL add failgroup E232 disk '/dev/oracleasm/disks/EU232ORL138' NOFORCE; Diskgroup altered. SQL> select * from v$asm_operation; GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE ------------ --------------- ------------ ---------- ---------- ---------- ---------- ---------- ----------- ------------------------------------------------------------------------------------------------------------------------------------ 6 REBAL RUN 11 11 341 10340 7656 1 SQL> select * from v$asm_operation; no rows selected
Finally all the newly disks are added into “+ORL”
6 10 NORMAL ORL_0010 E232 /dev/oracleasm/disks/EU232ORL138 CACHED MEMBER 6 9 NORMAL ORL_0009 E232 /dev/oracleasm/disks/EU232ORL136 CACHED MEMBER 6 8 NORMAL ORL_0008 E232 /dev/oracleasm/disks/EU232ORL134 CACHED MEMBER 6 7 NORMAL ORL_0007 E232 /dev/oracleasm/disks/EU232ORL132 CACHED MEMBER 6 5 NORMAL ORL_0005 E231 /dev/oracleasm/disks/EU231ORL137 CACHED MEMBER 6 3 NORMAL ORL_0003 E231 /dev/oracleasm/disks/EU231ORL135 CACHED MEMBER 6 1 NORMAL ORL_0001 E231 /dev/oracleasm/disks/EU231ORL133 CACHED MEMBER 6 0 NORMAL ORL_0000 E231 /dev/oracleasm/disks/EU231ORL131 CACHED MEMBER
ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 3670004 2466404 0 2466404 0 N ARCH/ MOUNTED HIGH N 512 4096 1048576 12282 11018 4094 2308 0 Y CLUSTERDG/ MOUNTED EXTERN N 512 4096 1048576 13178835 3688894 0 3688894 0 N FRA2/ MOUNTED NORMAL N 512 4096 1048576 106430116 19386134 917501 9234316 0 N ORADATA/ MOUNTED NORMAL N 512 4096 1048576 563189 501401 51199 225101 0 N ORL/
=====Hence tested & verified in our test env=====