-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathEMC_storage.txt
210 lines (164 loc) · 7.45 KB
/
EMC_storage.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
// =========== PowerPath ===========
# rpm -qa | grep device
device-mapper-event-libs-1.02.77-3.el6.x86_64
device-mapper-multipath-0.4.9-62.el6.x86_64
device-mapper-1.02.77-3.el6.x86_64
device-mapper-event-1.02.77-3.el6.x86_64
device-mapper-multipath-libs-0.4.9-62.el6.x86_64
device-mapper-libs-1.02.77-3.el6.x86_64
device-mapper-persistent-data-0.1.4-1.el6.x86_64
# rpm -qa | grep device-mapper-multipath
device-mapper-multipath-libs-0.4.9-62.el6.x86_64
device-mapper-multipath-0.4.9-62.el6.x86_64
# rpm -ivh EMCPower.LINUX-5.6.0.00.00-143.RHEL6.x86_64.rpm
Preparing... ########################################### [100%]
1:EMCpower.LINUX ########################################### [100%]
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.
# rpm -qa | grep EMCpower
EMCpower.LINUX-5.6.0.00.00-143.x86_64
# emcpreg -install
=========== EMC PowerPath Registration ===========
Do you have a new registration key or keys to enter?[n] y
Enter the registration keys(s) for your product(s),
one per line, pressing Enter after each key.
After typing all keys, press Enter again.
Key (Enter if done): BGP3-EB4K-RFRB-QS6R-MP94-G7GV
1 key(s) successfully added.
Key successfully installed.
Key (Enter if done):
1 key(s) successfully registered.
Stop the multipathd service.
# multipath -F
# /etc/init.d/multipathd stop
# chkconfig multipathd off
# modprobe -r scsi_dh_emc
Start/stop the PowerPath service.
# /etc/init.d/PowerPath start
Starting PowerPath: done
# /etc/init.d/PowerPath stop
Stopping PowerPath: done
# fdisk -l | grep emcpower
Disk /dev/emcpowera: 107.4 GB, 107374182400 bytes
NOTE:
1.PowerPath can not co-exist with DM-Multipath Service.
2."the closed source EMC powerpath module might not integrate itself correctly with the cgroups devices controller".
Because even if the "devices" controller is enabled, the powperpath disk is allowed by devices controller too.
Disable "devices" controller in /etc/libvirt/qemu.conf
cgroup_controllers = [ "cpu", "memory" ]
# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=FCN00123500005 [S2YCUI-EMC]
Logical device ID=6006016002213200006E15B43D38E211 [LUN 0]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0;
Owner: default=SP B, current=SP B Array failover mode: 4
==============================================================================
--------------- Host --------------- - Stor - -- I/O Path -- -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
2 lpfc sdb SP A0 active alive 0 0
2 lpfc sdc SP B0 active alive 0 0
// =========== MultiPath ===========
# yum install device-mapper.x86_64 device-mapper-multipath.x86_64
# modprobe dm-multipath
# modprobe dm-round-robin
# lsmod | grep multipath
dm_multipath 17711 2 dm_round_robin
dm_mod 82743 14 dm_multipath,dm_mirror,dm_log
# service multipathd start
Starting multipathd daemon: [ OK ]
# chkconfig multipathd on //ensure sure that the multipath daemon starts on bootup
# /sbin/mpathconf --enable
# ls -l /etc/multipath.conf
-rw-------. 1 root root 2754 Jan 9 18:55 /etc/multipath.conf
# multipath -v2 //prints out multipathed paths that show which devices are multipathed.
create: mpatha (0QEMU QEMU HARDDISK test) undef QEMU,QEMU HARDDISK
size=10G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 3:0:0:0 sdb 8:16 undef ready running
`-+- policy='round-robin 0' prio=1 status=undef
`- 4:0:0:0 sdc 8:32 undef ready running
Jan 09 18:56:29 | mpathb: ignoring map
# multipath -ll
mpatha (0QEMU QEMU HARDDISK test) dm-2 QEMU,QEMU HARDDISK
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 3:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 4:0:0:0 sdc 8:32 active ready running
Configure multipath.
# vim /etc/multipath.conf
...
# service multipathd reload
# multipath -ll
mpathb (36006016002213200006e15b43d38e211) dm-0 DGC,RAID 5 ----创建了一个RAID 5
size=100G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 2:0:1:0 sdc 8:32 active ready running ----多路径下的两个盘符sdb和sdc
`-+- policy='round-robin 0' prio=0 status=enabled
`- 2:0:0:0 sdb 8:16 active ready running
# ls -l /dev/mapper/*
crw-rw----. 1 root root 10, 58 Nov 27 15:52 /dev/mapper/control
lrwxrwxrwx. 1 root root 7 Nov 27 17:28 /dev/mapper/mpathb -> ../dm-0
lrwxrwxrwx. 1 root root 7 Nov 27 15:52 /dev/mapper/VolGroup-eserved_LV -> ../dm-2
lrwxrwxrwx. 1 root root 7 Nov 27 15:52 /dev/mapper/VolGroup-ogVol_root -> ../dm-3
lrwxrwxrwx. 1 root root 7 Nov 27 15:52 /dev/mapper/VolGroup-ogVol_swap -> ../dm-1
# fdisk -l | grep mpath
Disk /dev/mapper/mpathb: 107.4 GB, 107374182400 bytes
# multipath -F //删除现有路径 两个新的路径就会被删除
# multipath -v2 //格式化路径 格式化后又出现
multipath磁盘的基本操作
# pvcreate /dev/mapper/mpathb
Physical volume "/dev/mapper/mpathb" successfully created
//# vgcreate test /dev/mapper/mpathb
//# lvcreate -L 1g -n lv1 test
# fdisk /dev/mapper/mpathb //分区时用这个目录/dev/mapper/mpathb
# mkfs.ext4 /dev/mapper/mpathab #对mpath1b分区格式化成ext4文件系统
# mount /dev/mapper/mpathab /mnt/
dd往磁盘读写数据，然后用iostat观察各通道的流量和状态，以判断failover或负载均衡方式是否正常：
# dd if=/dev/zero of=/dev/mapper/mpathb
# iostat -k 2
// ==multipath.conf配置文件
multipath.conf主要包括blacklist、multipaths、devices三部份的配置
blacklist配置
blacklist {
devnode "^sda"
}
Multipaths部分配置multipaths和devices两部份的配置。
multipaths {
multipath {
wwid **************** #此值multipath -v3可以看到
alias iscsi-dm0 #映射后的别名,可以随便取
path_grouping_policy multibus #路径组策略
path_checker tur #决定路径状态的方法
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
}
}
Devices部分配置
devices {
device {
vendor "iSCSI-Enterprise" #厂商名称
product "Virtual disk" #产品型号
path_grouping_policy multibus #默认的路径组策略
getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
path_checker readsector0 #决定路径状态的方法
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
failback immediate #故障恢复的模式
no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值
rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
}
}
How to debug multipathd ?
1.Log to console:
# multipathd -d
2.Log to syslog:
# multipathd -v4
3.Use strace:
# strace -f multipathd
how to get the wwid.
# scsi_id --whitelisted --replace-whitespace --device=/dev/sdX
Refference:
http://www.datacentersky.com/depth-analysis-the-path-of-the-how-to-configure-multi-in-the-linux-environment.html
http://www.linuxfly.org/post/513/
http://christophe.varoqui.free.fr/faq.html