Wednesday, February 13, 2013

Removing multipath disks


Removing multipath disks

in one of our RHEL4 blade servers, i need to remove
1x500G HP XP1000 LUN:
PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-19 lx0080apps02vg lvm2 a-   500.00G 100.00G
  /dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G
[root@lx0080 ~]# pvdisplay /dev/dm-19
--- Physical volume ---
  PV Name               /dev/dm-19
  VG Name               lx0080apps02vg
  PV Size               500.00 GB / not usable 1.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              128000
  Free PE               25600
  Allocated PE          102400
  PV UUID               O9b3hw-oSUO-pUpe-oGJw-zEBp-tKXo-3ba2g9
so here's how to do it without causing disruptions on the machine:
btw, this machine hosts a test environment.

CAUTION: DON'T DO THIS IN PRODUCTION ENVIRONMENT ON A REGULAR
BUSINESS DAY (Even if you are sure it won't cause any problems)

make sure no i/o activity on the disk
- i've removed it from nfs exports
- unmounted the FS
remove from VG group
remove from multipath definitions
delete from system

below here are the actual steps.

1. remove from VG group

[root@lx0080 ~]#
 vgremove lx0080apps02vg
Found duplicate PV O9b3hwoSUOpUpeoGJwzEBptKXo3ba2g9: using /dev/sdb1 not /dev/sdq1
Do you really want to remove volume group "lx0080apps02vg" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume "lx0080apps02LV"? [y/n]: y
  Logical volume "lx0080apps02LV" successfully removed
  Volume group "lx0080apps02vg" successfully removed

pvs
PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-19                lvm2 --   500.00G 500.00G 
/dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G
as you can see - /dev/dm-19 is now an orphan. since its the only 500G
LUN in the server, its easy to identify.

2. remove from multipath

from multipath -ll (other entries removed):
mpath8 (360060e8004a51a000000a51a00000200)
[size=500 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 3:0:0:0 sdb 8:16   [active][ready]
 \_ 4:0:0:0 sdq 65:0   [active][ready]
it has two active paths.

multipath -f mpath8

(-f flush a multipath device map specified as parameter, if unused)

[root@lx0080 ~]#
 multipath -f mpath8
[root@lx0080 ~]#
 multipath -ll | grep 360060e8004a51a000000a51a00000200


360060e8004a51a000000a51a00000200 is the UUID of the disk from
scsi_id -gus /block/sdb (or /block/sdq)

now the multipath map is removed. but the disk is still in the system.

[root@lx0080 ~]#
 fdisk -l /dev/sdq
Disk /dev/sdq: 536.8 GB, 536877465600 bytes
255 heads, 63 sectors/track, 65271 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdq1               1       65271   524289276   8e  Linux LVM
[root@lx0080 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 536.8 GB, 536877465600 bytes
255 heads, 63 sectors/track, 65271 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       65271   524289276   8e  Linux LVM
[root@lx0080 ~]#

[root@lx0080 ~]#
 pvs
Found duplicate PV O9b3hwoSUOpUpeoGJwzEBptKXo3ba2g9: using /dev/sdb1 not /dev/sdq1
  PV         VG             Fmt  Attr PSize   PFree
  /dev/dm-17 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-18 lx0080apps00vg lvm2 a-   200.00G  11.50G
  /dev/dm-20 lx0080apps01vg lvm2 a-   250.00G  50.00G
  /dev/dm-21 lx0080apps01vg lvm2 a-   100.00G  50.00G
  /dev/dm-22 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-23 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/dm-24 lx0080apps01vg lvm2 a-   200.00G  10.00G
  /dev/sda2  vg00           lvm2 a-    67.88G   4.47G
  /dev/sdb1                 lvm2 --   500.00G 500.00G

3. delete from the system

[root@lx0080 ~]#
 echo 1 > /sys/block/sdb/device/delete
[root@lx0080 ~]#
 echo 1 > /sys/block/sdq/device/delete
[root@lx0080 ~]#
 dmesg
Synchronizing SCSI cache for disk sdb:
Synchronizing SCSI cache for disk sdq:
[root@lx0080 ~]# fdisk -l /dev/sdb
[root@lx0080 ~]#
 fdisk -l /dev/sdq


here are some entries generated thru syslog:

- when multipath -f was run:
Apr 22 09:17:20 lx0080 multipathd: dm map mpath8 removed
Apr 22 09:17:20 lx0080 udevd[2444]: udev done!
- when deleting sdb
Apr 22 09:22:52 lx0080 kernel: Synchronizing SCSI cache for disk sdb:
Apr 22 09:22:52 lx0080 multipathd: remove sdb path checker
Apr 22 09:22:52 lx0080 udevd[2444]: udev done!

- when deleting sdq
Apr 22 09:22:57 lx0080 kernel: Synchronizing SCSI cache for disk sdq:
Apr 22 09:22:57 lx0080 multipathd: remove sdq path checker
Apr 22 09:22:57 lx0080 udevd[2444]: udev done!
Apr 22 09:23:02 lx0080 udevd[2444]: udev done!

our storage admin can now remove the disk definition from storage
server.

1 comment: