반응형
multipath IO overview
multipath는 서버에서 스토리지까지 경로가 여러개일 경우, 다수의 경로를 통해 스토리지에 접속하는 기능을 제공한다. 스토리지 업체에 따라 전용 Multipath Driver를 사용할 수도 있지만, 그게 아닐 경우 리눅스 표준인 Device Mapper Multipath Driver를 사용한다. 이번 실험실에서는 Rocky Linux 9에서 multipathd 데몬을 통해 하나의 Device 장치에 2개의 경로로 접속하는 설정을 수행해볼 예정이다.
실제 실험환경은 Nutanix AHV 가상화 위에서 진행되었으며, 스토리지 제품이 없어 스토리지 역할을 할 iSCSI Target은 다른 가상머신에 구현 했다. iSCSI Target을 구성하는 방법은 targetcli (iSCSI Target 생성) 를 참고 바란다.
iSCSI 프로토콜은 기본 3260/TCP 포트를 사용하지만 이 실험에서는 Firewalld / SELinux를 종료해 놓고 실험을 진행했다.
Lab 환경
OS : Rocky 9.3 - Basic Server Installed (multipath, iSCSI Target)
구성도
iSCSI 연결 - Install iscsiadm 설치
[root@multipathio ~]# yum install iscsi-initiator-utils
Last metadata expiration check: 0:05:04 ago on Tue 02 Apr 2024 05:34:02 PM KST.
Dependencies resolved.
=====================================================================================================================================================================
Package Architecture Version Repository Size
=====================================================================================================================================================================
Installing:
iscsi-initiator-utils x86_64 6.2.1.4-3.git2a8f9d8.el9 baseos 378 k
Installing dependencies:
iscsi-initiator-utils-iscsiuio x86_64 6.2.1.4-3.git2a8f9d8.el9 baseos 94 k
isns-utils-libs x86_64 0.101-4.el9 baseos 100 k
Transaction Summary
=====================================================================================================================================================================
Install 3 Packages
Total download size: 572 k
Installed size: 2.1 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): isns-utils-libs-0.101-4.el9.x86_64.rpm 512 kB/s | 100 kB 00:00
(2/3): iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64.rpm 466 kB/s | 94 kB 00:00
(3/3): iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64.rpm 1.6 MB/s | 378 kB 00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 731 kB/s | 572 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : isns-utils-libs-0.101-4.el9.x86_64 1/3
Installing : iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64 2/3
Running scriptlet: iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64 2/3
Created symlink /etc/systemd/system/sockets.target.wants/iscsiuio.socket → /usr/lib/systemd/system/iscsiuio.socket.
Installing : iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 3/3
Running scriptlet: iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 3/3
Created symlink /etc/systemd/system/remote-fs.target.wants/iscsi.service → /usr/lib/systemd/system/iscsi.service.
Created symlink /etc/systemd/system/sockets.target.wants/iscsid.socket → /usr/lib/systemd/system/iscsid.socket.
Created symlink /etc/systemd/system/sysinit.target.wants/iscsi-onboot.service → /usr/lib/systemd/system/iscsi-onboot.service.
Verifying : isns-utils-libs-0.101-4.el9.x86_64 1/3
Verifying : iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64 2/3
Verifying : iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 3/3
Installed:
iscsi-initiator-utils-6.2.1.4-3.git2a8f9d8.el9.x86_64 iscsi-initiator-utils-iscsiuio-6.2.1.4-3.git2a8f9d8.el9.x86_64 isns-utils-libs-0.101-4.el9.x86_64
Complete!
[root@multipathio ~]# iscsiadm
Try `iscsiadm --help' for more information.
# Rocky 9에서 자동으로 iqn 이름이 생성되지 않아 직접 iqn 이름을 생성해줘야 한다.
[root@multipathio ~]# vi /etc/iscsi/initiatorname.iscsi
[root@multipathio ~]# systemctl restart iscsid
[root@multipathio ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2024-04.local.rocky9:multipath
Target에 ACL 등록
# Target 측에서 Client iqn을 등록 해주어야 한다.
/> cd iscsi/iqn.2024-05.local.rocky:iscsitarget/tpg1/acls/
/iscsi/iqn.20...get/tpg1/acls> create iqn.2024-04.local.rocky9:multipath
Created Node ACL for iqn.2024-04.local.rocky9:multipath
Created mapped LUN 0.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- LUN01 .................................................... [/dev/target-volume/target-lun01 (30.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2024-05.local.rocky:iscsitarget ............................................................................... [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 2]
| | o- iqn.1994-05.com.redhat:e33825e1bd50 .................................................................. [Mapped LUNs: 1]
| | | o- mapped_lun0 ................................................................................. [lun0 block/LUN01 (rw)]
| | o- iqn.2024-04.local.rocky9:multipath ................................................................... [Mapped LUNs: 1]
| | o- mapped_lun0 ................................................................................. [lun0 block/LUN01 (rw)]
| o- luns .......................................................................................................... [LUNs: 1]
| | o- lun0 ............................................... [block/LUN01 (/dev/target-volume/target-lun01) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
iSCSI 연결 - Target Login
# iSCSI Target에 Login (Network 경로가 2개라 2개의 경로를 모두 찾아 Login 해야한다.)
[root@multipathio ~]# iscsiadm -m discovery --type sendtargets --portal 192.168.70.6
192.168.70.6:3260,1 iqn.2024-05.local.rocky:iscsitarget
[root@multipathio ~]# iscsiadm -m discovery --type sendtargets --portal 192.168.70.7
192.168.70.7:3260,1 iqn.2024-05.local.rocky:iscsitarget
[root@multipathio ~]# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2024-05.local.rocky:iscsitarget, portal: 192.168.70.6,3260]
Logging in to [iface: default, target: iqn.2024-05.local.rocky:iscsitarget, portal: 192.168.70.7,3260]
Login to [iface: default, target: iqn.2024-05.local.rocky:iscsitarget, portal: 192.168.70.6,3260] successful.
Login to [iface: default, target: iqn.2024-05.local.rocky:iscsitarget, portal: 192.168.70.7,3260] successful.
# 연결된 Device 확인
[root@multipathio ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 49G 0 part
├─rl-root 253:0 0 45.1G 0 lvm /
└─rl-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 30G 0 disk
sdc 8:32 0 30G 0 disk
sr0 11:0 1 1024M 0 rom
[root@multipathio ~]# lsscsi
[0:0:0:0] disk NUTANIX VDISK 0 /dev/sda
[1:0:0:0] cd/dvd QEMU QEMU DVD-ROM 2.5+ /dev/sr0
[3:0:0:0] disk LIO-ORG LUN01 4.0 /dev/sdb
[4:0:0:0] disk LIO-ORG LUN01 4.0 /dev/sdc
# 두개의 Device의 wwid 값이 같음을 확인(타겟 Device가 사실상 같은 Device임을 의미)
[root@multipathio ~]# /lib/udev/scsi_id -g -u -d /dev/sdb
36001405d6a192f9887a4ad79108c7cf7
[root@multipathio ~]# /lib/udev/scsi_id -g -u -d /dev/sdc
36001405d6a192f9887a4ad79108c7cf7
Multipath IO 구성
# Multipath 패키지 설치
[root@multipathio ~]# yum install device-mapper-multipath
Last metadata expiration check: 2:10:14 ago on Wed 03 Apr 2024 07:41:20 AM KST.
Dependencies resolved.
==============================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================
Installing:
device-mapper-multipath x86_64 0.8.7-22.el9 baseos 143 k
Installing dependencies:
device-mapper-multipath-libs x86_64 0.8.7-22.el9 baseos 267 k
Transaction Summary
==============================================================================================================================================================================================================
Install 2 Packages
Total download size: 410 k
Installed size: 1.2 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): device-mapper-multipath-libs-0.8.7-22.el9.x86_64.rpm 1.7 MB/s | 267 kB 00:00
(2/2): device-mapper-multipath-0.8.7-22.el9.x86_64.rpm 904 kB/s | 143 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 550 kB/s | 410 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : device-mapper-multipath-libs-0.8.7-22.el9.x86_64 1/2
Installing : device-mapper-multipath-0.8.7-22.el9.x86_64 2/2
Running scriptlet: device-mapper-multipath-0.8.7-22.el9.x86_64 2/2
Created symlink /etc/systemd/system/sysinit.target.wants/multipathd.service → /usr/lib/systemd/system/multipathd.service.
Created symlink /etc/systemd/system/sockets.target.wants/multipathd.socket → /usr/lib/systemd/system/multipathd.socket.
Verifying : device-mapper-multipath-libs-0.8.7-22.el9.x86_64 1/2
Verifying : device-mapper-multipath-0.8.7-22.el9.x86_64 2/2
Installed:
device-mapper-multipath-0.8.7-22.el9.x86_64 device-mapper-multipath-libs-0.8.7-22.el9.x86_64
Complete!
[root@multipathio ~]# multipath -ll
60325.731757 | /etc/multipath.conf does not exist, blacklisting all devices.
60325.731805 | You can run "/sbin/mpathconf --enable" to create
60325.731816 | /etc/multipath.conf. See man mpathconf(8) for more details
60325.746961 | DM multipath kernel driver not loaded
# multipath.conf example file 을 /etc/multipath.conf로 복사
[root@multipathio ~]# cp /usr/share/doc/device-mapper-multipath/multipath.conf /etc/
# vi, nano, vim 등 편집기를 통해 multipath.conf 파일 수정
[root@multipathio ~]# vi /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
}
# rocky 9에서 multipath v0.8.7가 설치되었는데, wwid "*"는 parsing error 가 발생
# wwid ".*" 정규표현식을 통해 전체 wwid를 blacklist에 등록
# devnode로 blacklist에 등록하면, 예외처리시 wwid로 예외처리가 안됨.
# devnode blacklist는 devnode로 예외처리, wwid blacklist는 wwid로 예외처리를 함
# For example, a WWID exception will not apply to devices specified
# by a devnode blacklist entry, even if the blacklisted device is associated with
# that WWID.
blacklist {
wwid ".*"
}
blacklist_exceptions {
wwid 36001405d6a192f9887a4ad79108c7cf7
}
multipaths {
multipath {
wwid 36001405d6a192f9887a4ad79108c7cf7
alias multipathio
path_grouping_policy multibus
failback manual
path_selector "round-robin 0"
rr_weight priorities
no_path_retry 5
}
}
# 저장 후 multipathd 서비스 시작
[root@multipathio ~]# systemctl start multipathd
[root@multipathio ~]# systemctl enable multipathd
[root@multipathio ~]# mpathconf --enable
[root@multipathio ~]# multipath -ll
multipathio (36001405d6a192f9887a4ad79108c7cf7) dm-2 LIO-ORG,LUN01
size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 3:0:0:0 sdb 8:16 active ready running
`- 4:0:0:0 sdc 8:32 active ready runningMultipath 구성된 Volume 마운트
[root@multipathio ~]# fdisk -l | grep /dev/
Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
/dev/sda1 * 2048 2099199 2097152 1G 83 Linux
/dev/sda2 2099200 104857599 102758400 49G 8e Linux LVM
Disk /dev/mapper/rl-root: 45.13 GiB, 48452599808 bytes, 94633984 sectors
Disk /dev/mapper/rl-swap: 3.87 GiB, 4156555264 bytes, 8118272 sectors
Disk /dev/sdb: 30 GiB, 32212254720 bytes, 62914560 sectors
Disk /dev/sdc: 30 GiB, 32212254720 bytes, 62914560 sectors
Disk /dev/mapper/multipathio: 30 GiB, 32212254720 bytes, 62914560 sectors
[root@multipathio ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 49G 0 part
├─rl-root 253:0 0 45.1G 0 lvm /
└─rl-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 30G 0 disk
└─multipathio 253:2 0 30G 0 mpath
sdc 8:32 0 30G 0 disk
└─multipathio 253:2 0 30G 0 mpath
sr0 11:0 1 1024M 0 rom
[root@multipathio ~]# mkfs.xfs /dev/mapper/multipathio
meta-data=/dev/mapper/multipathio isize=512 agcount=4, agsize=1966080 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=1 inobtcount=1 nrext64=0
data = bsize=4096 blocks=7864320, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@multipathio ~]# mount /dev/mapper/multipathio /mnt
[root@multipathio ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs tmpfs 716M 8.6M 708M 2% /run
/dev/mapper/rl-root xfs 46G 1.4G 44G 4% /
/dev/sda1 xfs 960M 223M 738M 24% /boot
tmpfs tmpfs 358M 0 358M 0% /run/user/0
/dev/mapper/multipathio xfs 30G 247M 30G 1% /mnt
# multipath로 구성된 volume은 device mapper (dm-2)로 동작하는 것을 확인 할 수 있다.
[root@multipathio ~]# dmsetup info /dev/dm-2
Name: multipathio
State: ACTIVE
Read Ahead: 4096
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 2
Number of targets: 1
UUID: mpath-36001405d6a192f9887a4ad79108c7cf7
반응형
'Linux' 카테고리의 다른 글
| Linux LVM on CentOS 7 (0) | 2024.04.03 |
|---|---|
| iscsiadm (iSCSI Initiator 설정) (0) | 2024.04.03 |
| targetcli (iSCSI Target 생성) (0) | 2024.04.03 |
