【笔记】HCIE Storage v3.0 OceanStor Pacific分布式存储架构

OceanStor Pacific分布式存储的架构和实验。

软件架构

分布式存储，可以理解为多节点服务器，通过网络将节点连接在一起，然后通过软件部署，形成统一的存储池。

节点角色

  • 管理：负责整个集群的EC算法，重构规则，故障冗余，监控其他节点的状态。数据流量：管理网络
  • 计算：实际运行业务的节点，存储控制的机头，通过写入数据时写入到VBS节点，VBS按照EC或者副本的方式进行数据切分。数据流向：存储前端网络
  • 存储：提供存储资源的节点，每个硬盘都会有一个OSD进程，数据在VBS进行切分后，写入到OSD节点，OSD和OSD之间重构，数据同步。数据流量：存储后端网络

节点角色与可部署的组件参考：https://support.huawei.com/hedex/hdx.do?docid=EDOC1100376801&id=pro_des_000110，不同的文件系统协议部署的组件稍有不同

关键组件

  • FSM（FusionStorage Manager）：管理模块，提供告警、监控、日志等操作维护功能
  • MDC（MetaData Controller）：元数据控制软件，实现控制分布式集群的状态、控制数据分布规则以及数据重建规则等。至少部署3个形成MDC集群，可选择3、5、7、9个节点部署MDC。
  • ZK（ZooKeeper）：为MDC集群提供选主仲裁，保存集群信息。由ZK集群在多个MDC中选举主MDC，主MDC监控其他MDC，主MDC故障后再选举新的主MDC。有几个MDC就会有几个ZK，MDC在哪里ZK就会在哪里
  • OSD（Object Storage Device）：管理硬盘的进程，处理VBS下发的I/O消息，执行具体的I/O数据存储空间操作。一块盘一个OSD进程，节点加入存储池时，启动OSD进程。
  • VBS：虚拟块存储管理组件（块服务独有），执行元数据管理。通过SCSI或iSCSI接口提供分布式存储接入服务，使计算资源能过通过VBS访问存储系统，从而把卷映射给客户的应用服务器使用。选择节点启用VBS后，该节点上启动VBS进程，部署VBS进程的节点形成VBS集群
  • EDS：块的元数据、索引，接收来自VBS的I/O业务，并执行具体的I/O操作。可以执行各服务相关的特性操作（比如快照），同时还可以管理存储空间，建立数据与存储空间之间的索引关系，确保每块数据都可以通过索引找到对应的存储位置。同时数据存储到物理空间之前，进行数据重删压缩处理。加入存储池的节点开启服务时，启动EDS进程、

组件交互

  1. MDC
    1. MDC一般是主备管理，一个主，多个备
    2. MDC通过心跳检测机制对OSD和VBS进行状态监控，会与OSD、VBS进行消息交互
    3. MDC模块启动后，会向ZK注册选主，先注册的为主MDC，ZK通过心跳监控MDC的状态，当主的挂掉后，再重新选主
    4. 主MDC与其他MDC维持心跳关系，由主MDC确定其他MDC是否为归属MDC，并告知归属MDC管理哪些存储池，主MDC决定归属MDC故障后的接替者
  2. OSD
    1. OSD启动时，向主MDC确认归属MDC是哪个，向归属MDC汇报自己的状态，归属MDC再将状态转发给VBS
    2. OSD与归属MDC维持心跳关系，每秒向归属MDC汇报自己的状态；若归属MDC在5秒内没有收到OSD的心跳，MDC会人为OSD故障，并把OSD踢出存储池，更新OSD view到每个OSD和VBS，在5分钟内没有收到心跳就会数据重建
  3. VBS
    1. VBS启动时会查询主MDC，向主MDC注册（主MDC维持了一个VBS表，并同步给归属MDC，以便于其他MDC能将OSD的变化转发给VBS），向主MDC确定是否为主VBS
    2. 主VBS从主MDC获取I/O view，主VBS向OSD要元数据，其他VBS向主VBS获取元数据
    3. 一套FS只有一个主VBS，如果主VBS故障，其他VBS会重新向主MDC确认新的主VBS

网络平面

  1. 管理平面：FSM和FSA的通信
  2. 存储平面：用于MDC、VBS、OSD的通信
  3. 业务平面：用于OS与VBS的通信

关键I/O流程（块）

写I/O流程

  1. 上层应用下发I/O请求到VBS模块
  2. VBS收到后， 根据算法分发将数据转发到对应节点的EDS模块
  3. EDS根据I/O大小
    1. 小I/O（<512KB），EDS直接存在SSD缓存盘中，同时该EDS节点的内存会存储一份数据，EDS会返回写成功给VBS，再由VBS返回给应用。待内存中的数据聚合到更大的块后，系统将数据异步写入OSD中
    2. 大I/O（≥512KB），EDS将数据直接写入到OSD中做数据持久化

读I/O流程

  1. 上层应用下发读I/O请求到存储的VBS模块中
  2. VBS收到I/O请求后：
    1. 如果系统开了直通读，VBS将I/O请求发送给OSD，OSD处理步骤跳转第三步
    2. 如果系统没有开启直通读，VBS将根据算法，将I/O转发到指定节点，该节点的EDS模块在内存的写缓存中查找是否有指定数据，然后返回给VBS，读I/O流程结束；否则执行第三步
  3. 读请求发送给OSD，OSD收到请求后：
    1. 在SSD的写缓存上查找数据，找到后返回给VBS，读I/O流程结束
    2. 在SSD的读缓存上查找数据，找到后返回给VBS，读I/O流程结束
    3. 从硬盘上查找应用所需数据，找到后直接返回给VBS，同时增加该I/O数据的访问因子。如果访问因子达到一定值后会被缓存到SSD的读缓存中，方便下次快速读取

参考：
https://www.cnblogs.com/LongShineW/p/15334207.html

使用存储空间流程（块）

通过iSCSI协议


这个过程比较容易理解。

  1. 存储系统上创建卷
  2. 在存储上创建主机，将主机和应用服务器联系在一起（如果卷映射多个主机就创建主机组）
  3. 将卷映射给主机/主机组，相当于逻辑捆绑在一起
  4. 从应用侧扫描存储空间

通过SCSI协议

  1. 存储系统上创建卷
  2. 为应用服务器启动VBS服务
  3. 创建卷和VBS的关系。卷、VBS、主机成为逻辑上的整体
  4. 应用可以直接扫描存储空间并使用

iSCSI实验

主机网络配置

IP地址

业务主机给2个地址，1个是管理地址，1个用于存储地址
本次实验环境：
业务侧：
172.111.0.0/16是管理网段
172.112.0.0/16是存储业务网段
存储侧：
172.3.0.0/16（3台节点，分别为172.3.0.1、172.3.0.2、172.3.0.3）

主机添加路由

1
2
3
4
5
6
7
8
9
10
11
12
13
分布式存储的前端业务接口地址网段是172.3.0.0/16
[root@localhost ~]# route add -net 172.3.0.0 netmask 255.255.0.0 gw 172.112.0.254 dev ens224
[root@localhost ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.111.0.254   0.0.0.0         UG    100    0        0 ens192
172.3.0.0       172.112.0.254   255.255.0.0     UG    0      0        0 ens224
172.111.0.0     0.0.0.0         255.255.0.0     U     100    0        0 ens192
172.112.0.0     0.0.0.0         255.255.0.0     U     101    0        0 ens224

route永久生效
[root@localhost ~]# cat /etc/sysconfig/network-scripts/route-ens224
172.3.0.0/16 via 172.112.0.254

添加存储业务口路由

上面主机侧加了路由，数据包可以到达存储，但是存储侧却不知道去往业务112网段从哪里走，所以，存储侧也需要配置对应的路由让数据包回到业务主机

部署的时候将业务口进行了绑定，绑定端口的名字叫storage_data0，地址配置如图所示。

连通性测试

从业务主机上ping存储前端地址：

配置iscsi IP


如果没有开启，请点击开启：


另外2个存储节点同样操作。

发起业务连接

业务主机创建启动器

安装iscsi服务，查看iqn号：

1
2
[root@localhost ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:266aa8ea83e6

存储侧添加该业务主机的启动器：

回到业务主机，发起连接：

1
2
3
4
5
6
7
8
9
10
11
[root@localhost ~]# iscsiadm -m discovery -t st -p 172.3.0.1
172.3.0.1:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131073
[root@localhost ~]# iscsiadm -m discovery -t st -p 172.3.0.2
172.3.0.2:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131074
[root@localhost ~]# iscsiadm -m discovery -t st -p 172.3.0.3
172.3.0.3:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131075
[root@localhost ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131073, portal: 172.3.0.1,3260]
Logging in to [iface: default, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131074, portal: 172.3.0.2,3260]
Logging in to [iface: default, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131075, portal: 172.3.0.3,3260]
[root@localhost ~]# iscsiadm -m node -o update -n node.startup -v automatic

存储侧创建主机并映射LUN



创建卷：

创建映射：

建立业务

扫盘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
[root@localhost ~]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware   Model: Virtual disk     Rev: 2.0
      Type:   Direct-Access                    ANSI SCSI revision: 06
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: NECVMWar Model: VMware SATA CD00 Rev: 1.00
      Type:   CD-ROM                           ANSI SCSI revision: 05
.Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 8 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 10 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 11 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 12 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 13 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 14 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 15 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 16 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 17 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 18 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 19 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 20 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 21 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 22 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 23 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 24 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 25 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 26 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 27 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 28 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 29 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 30 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 31 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 32 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 33 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 33 0 0 1 ...
OLD: Host: scsi33 Channel: 00 Id: 00 Lun: 01
      Vendor: Huawei   Model: VBS fileIO       Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 34 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 34 0 0 1 ...
OLD: Host: scsi34 Channel: 00 Id: 00 Lun: 01
      Vendor: Huawei   Model: VBS fileIO       Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 35 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 35 0 0 1 ...
OLD: Host: scsi35 Channel: 00 Id: 00 Lun: 01
      Vendor: Huawei   Model: VBS fileIO       Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.


[root@localhost ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk
├─sda1        8:1    0    1G  0 part /boot/efi
├─sda2        8:2    0    1G  0 part /boot
└─sda3        8:3    0   50G  0 part
  └─cl-root 253:0    0   50G  0 lvm  /
sdb           8:16   0    5G  0 disk
sdc           8:32   0    5G  0 disk
sdd           8:48   0    5G  0 disk
sr0          11:0    1 1024M  0 rom

多路径配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[root@localhost ~]# rpm -qa |grep multi
device-mapper-multipath-0.8.3-3.el8.x86_64
device-mapper-multipath-libs-0.8.3-3.el8.x86_64
初始配置的时候是没有默认配置文件
[root@localhost ~]# cp /usr/share/doc/device-mapper-multipath/multipath.conf /etc/
[root@localhost ~]# vim /etc/multipath.conf
blacklist {
       devnode "sda"
}

devices{
       device{
              vendor                              "Huawei"
              product                             "VBS fileIO"
              path_grouping_policy                 multibus
              prio                                 const
              path_selector                       "service-time 0"
              path_checker                         tur
              failback                             immediate
              no_path_retry                       "10"
       }
}
[root@localhost ~]# systemctl enable  multipathd.service --now
[root@localhost ~]# multipath -ll
mpatha (36888603000000004b04fe6f621110518) dm-1 Huawei,VBS fileIO
size=5.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 33:0:0:1 sdb 8:16 active ready running
  |- 34:0:0:1 sdd 8:48 active ready running
  `- 35:0:0:1 sdc 8:32 active ready running

格式化分区并挂载

清理环境

停止多路径

1
2
[root@localhost ~]#  systemctl stop multipathd.socket
[root@localhost ~]#  systemctl stop multipathd.service

登出iscsi会话

1
2
3
4
5
6
7
[root@localhost ~]# iscsiadm -m node -u
Logging out of session [sid: 1, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131073, portal: 172.3.0.1,3260]
Logging out of session [sid: 2, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131074, portal: 172.3.0.2,3260]
Logging out of session [sid: 3, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131075, portal: 172.3.0.3,3260]
Logout of [sid: 1, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131073, portal: 172.3.0.1,3260] successful.
Logout of [sid: 2, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131074, portal: 172.3.0.2,3260] successful.
Logout of [sid: 3, target: iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131075, portal: 172.3.0.3,3260] successful.

删除iscsi会话

1
2
3
4
5
6
7
8
9
10
[root@localhost ~]# iscsiadm -m node
172.3.0.1:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131073
172.3.0.2:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131074
172.3.0.3:3260,1 iqn.2012-10.com.huawei.dsware:b04fe6f621110518.vbs.131075

[root@localhost ~]# iscsiadm -m node -o delete -p 172.3.0.1
[root@localhost ~]# iscsiadm -m node -o delete -p 172.3.0.2
[root@localhost ~]# iscsiadm -m node -o delete -p 172.3.0.3
[root@localhost ~]# iscsiadm -m node
iscsiadm: No records found

存储侧取消映射

  1. 卷/主机取消映射
  2. 删除卷
  3. 移除业务主机启动器
  4. 删除主机
  5. 删除启动器

SCSI实验

这就体现出VBS的好处了。
在业务主机侧安装VBS，相当于安装了个代理，VBS直接与OSD访问，速度上比iscsi快。
但是过程上比较麻烦，业务主机安装VBS的话，需要安装一些软件包（要配置好yum仓库，依赖包都在仓库里），从官网找到对应存储版本的tools压缩包（例：OceanStor-Pacific_8.2.0_Tools.tar.gz），在业务主机中解压运行对应OS的脚本即可（下面有详细步骤）；目前还不支持在Windows上安装。

查询是否安装过VBS

1
2
3
4
5
6
7
8
9
10
11
12
如果该计算节点曾经创建过VBS，确认待创建VBS服务的计算节点的存储软件是否被卸载。
在计算节点执行命令rpm -qa | grep clouda || rpm -qa --dbpath=/opt/fusionstorage/deploymanager/clouda/rpm/ | grep clouda，如果回显存在clouda安装包，说明存储软件未被卸载，需要执行2。
在计算节点执行命令ls -l /opt/fusionstorage/deploymanager/clouda/clear_all_for_fs_manual.sh查看clear_all_for_fs_manual.sh脚本是否存在。
如果回显如下，表示脚本存在。
[root@localhost ~]# ls -l /opt/fusionstorage/deploymanager/clouda/clear_all_for_fs_manual.sh
-r-x------. 1 root root 258 Apr 21 10:46 /opt/fusionstorage/deploymanager/clouda/clear_all_for_fs_manual.sh
执行以下命令卸载软件。
cd /opt/fusionstorage/deploymanager/clouda
bash clear_all_for_fs_manual.sh
否则，执行以下命令卸载软件。
cd /opt/fusionstorage/deploymanager/clouda
bash clear_all_for_fs_node.sh all

安装操作系统依赖包

兼容性查询

不是所有的系统都支持安装VBS，官网兼容性查询：https://info.support.huawei.com/storage/comp/#/oceanstor-pacific
兼容性查询

配置yum源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@localhost install_lib]# mount /dev/cdrom /media/
mount: /media: WARNING: device write-protected, mounted read-only.
[root@localhost install_lib]# rm -rf /etc/yum.repos.d/*
[root@localhost install_lib]# vim /etc/yum.repos.d/media.repo
[media]
name=media
baseurl=file:///media/AppStream
gpgcheck=0
enabled=1

[media1]
name=media1
baseurl=file:///media/BaseOS
gpgcheck=0
enabled=1


[root@localhost media]# dnf list
media                                                                                                                                                          25 MB/s | 2.2 MB     00:00
Last metadata expiration check: 0:00:01 ago on Thu 30 May 2024 04:28:50 PM CST.
Installed Packages
NetworkManager.x86_64                                                                      1:1.22.8-4.el8                                                                           @anaconda
NetworkManager-config-server.noarch                                                        1:1.22.8-4.el8                                                                           @anaconda
NetworkManager-libnm.x86_64                                                                1:1.22.8-4.el8                                                                           @anaconda
NetworkManager-libreswan.x86_64                                                            1.2.10-4.el8                                                                             @AppStream
NetworkManager-team.x86_64                                                                 1:1.22.8-4.el8                                                                           @anaconda
NetworkManager-tui.x86_64                                                                  1:1.22.8-4.el8

将操作系统依赖包上传至业务主机

如果计算节点的操作系统不是分布式存储OS操作系统，请参考：
https://support.huawei.com/hedex/api/pages/EDOC1100340534/YZM1124Y/03/resources/block_config_000218.html
VBS节点操作系统依赖包脚本
将将OceanStor-Pacific_8.2.0_Tools.tar.gz软件包上传到计算节点，并解压软件包：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[root@localhost ~]# tar -zxvf OceanStor-Pacific_8.2.0_Tools.tar.gz
[root@localhost ~]# ls
anaconda-ks.cfg  ccm_tool     disktools                           install_lib     network                               ofed      sha256sum_sync      SnapshotTool  vmware-tools-distrib
broadcastConfig  crldata.crl  FusionStoragePatch_PreDownload.zip  multipath.conf  OceanStor-Pacific_8.2.0_Tools.tar.gz  osd_tool  sha256sum_sync.cms  vmware
[root@localhost ~]# cd install_lib/
[root@localhost install_lib]# ls
install_lib_for_alma.sh     install_lib_for_debian.sh    install_lib_for_kylin.sh     install_lib_for_openeuler.sh  install_lib_for_suse.sh
install_lib_for_bclinux.sh  install_lib_for_euler.sh     install_lib_for_neokylin.sh  install_lib_for_rhel.sh       install_lib_for_ubuntu.sh
install_lib_for_centos.sh   install_lib_for_kylinsec.sh  install_lib_for_oel.sh       install_lib_for_rocky.sh      install_lib_for_uniontech.sh

[root@localhost install_lib]# vim install_lib_for_centos.sh
……

[root@localhost install_lib]# cp install_lib_for_centos.sh /tmp/
[root@localhost install_lib]# cd /tmp/
[root@localhost tmp]# sh install_lib_for_centos.sh compute
Prepare :  check roles [ compute ] ... done
Prepare :  check OS [ CentOS Linux release 8.2.2004 (Core)  ] ... done
Prepare :  Starting Install all libs ...
Install :  attr... done
Install :  numactl... done
Install :  rpm... done
Install :  traceroute... done
Install :  net-tools... done
Install :  unzip... done
Install :  zip... done
Install :  lsof... done
Install :  dos2unix... done
Install :  expect... done
Install :  pciutils... done
Install :  sysstat... done
Install :  wget... done
Install :  curl... done
Install :  lsscsi... done
Install :  bc... done
Install :  rsync... done
Install :  acl... done
Install :  libnl3... done
Install :  python3... done
Install :  NetworkManager... done
Install :  ncurses-compat-libs... done
Install all packages done.
 All libraries installed successfully.

增加VBS节点






1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
配置Bond0（安装VBS的节点必须要配置网卡绑定）
[root@localhost ~]# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.111.0.21  netmask 255.255.0.0  broadcast 172.111.255.255
        inet6 fe80::f70b:7411:ea7:db86  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:ab:36:60  txqueuelen 1000  (Ethernet)
        RX packets 40993  bytes 203927333 (194.4 MiB)
        RX errors 0  dropped 24  overruns 0  frame 0
        TX packets 39547  bytes 3273318 (3.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.112.0.21  netmask 255.255.0.0  broadcast 172.112.255.255
        inet6 fe80::9a9:bf06:6703:f9f9  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:ab:03:1f  txqueuelen 1000  (Ethernet)
        RX packets 3549  bytes 8309534 (7.9 MiB)
        RX errors 0  dropped 33  overruns 0  frame 0
        TX packets 3176  bytes 264776 (258.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:ab:2b:47  txqueuelen 1000  (Ethernet)
        RX packets 3  bytes 180 (180.0 B)
        RX errors 0  dropped 2  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1688  bytes 128348 (125.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1688  bytes 128348 (125.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@localhost network-scripts]# nmcli connection delete ens224
Connection 'ens224' (106aa30e-63c8-4e38-8c8f-5e046247df58) successfully deleted.
[root@localhost network-scripts]# nmcli connection add ifname bond0 con-name bond0 type bond mode active-backup
[root@localhost network-scripts]# nmcli connection modify bond0 ipv4.addresses 172.3.0.10/16 autoconnect yes ipv4.method manual
[root@localhost network-scripts]# nmcli connection add ifname ens224 con-name ens224 type bond-slave master bond0
Connection 'ens224' (f8ac2986-71b5-4c39-b69d-7a6c937ca111) successfully added.
[root@localhost network-scripts]# nmcli connection add ifname ens256  con-name ens256 type bond-slave master bond0
Connection 'ens256' (2f6790d2-c2e8-4b58-9bff-0250de2f9ce1) successfully added.
[root@localhost network-scripts]# nmcli connection up bond0
Connection successfully activated (master waiting for slaves) (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)
[root@localhost network-scripts]# ifconfig
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet 172.3.0.10  netmask 255.255.0.0  broadcast 172.112.255.255
        inet6 fe80::963e:242a:ec4c:d6cd  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:ab:03:1f  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26  bytes 2940 (2.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



创建卷并挂载



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
[root@localhost network-scripts]# rescan-scsi-bus.sh
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware   Model: Virtual disk     Rev: 2.0
      Type:   Direct-Access                    ANSI SCSI revision: 06
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 3 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
      Vendor: NECVMWar Model: VMware SATA CD00 Rev: 1.00
      Type:   CD-ROM                           ANSI SCSI revision: 05
Scanning host 4 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 5 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 6 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 7 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 8 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 9 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 10 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 11 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 12 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 13 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 14 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 15 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 16 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 17 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 18 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 19 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 20 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 21 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 22 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 23 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 24 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 25 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 26 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 27 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 28 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 29 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 30 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 31 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 32 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning host 36 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
 Scanning for device 36 65535 5 0 ...
OLD: Host: scsi36 Channel: 65535 Id: 05 Lun: 00
      Vendor: Huawei   Model: VBS fileIO       Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 05
Scanning host 37 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
0 new or changed device(s) found.
0 remapped or resized device(s) found.
0 device(s) removed.
[root@localhost network-scripts]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  100G  0 disk
├─sda1        8:1    0    1G  0 part /boot/efi
├─sda2        8:2    0    1G  0 part /boot
└─sda3        8:3    0   50G  0 part
  └─cl-root 253:0    0   50G  0 lvm  /
sde           8:64   0    5G  0 disk

清理环境

  1. 解除映射并删除



  1. 删除VBS

  1. 移除节点