Linux6.5_x64+oracle11.2.0.4+RoseHA9.0一、设备信息及规划Dell R730服务器：2*E5-2643v3 3.4GHz6C/4*16G DDR4memory/3*300GB15K RPM SAS/Raid/2*2Port1000M/dual power supply/DVD/2*single port FC HBA card/3year overseas warranty其中磁盘0、1做Raid1,磁盘2做热备Dell MD3800f：7*600GB15K RPM SAS/2*16G FC,2U MD38xxF,8G CF cache,4*5m fiber/3year overseas warranty其中磁盘0、1、2、3、4、5做Raid10，磁盘6做热备Lun1=1.7T做数据存储,Lun2=3G做oracle仲裁,Lun3=2G预留BOSS仲裁。
二、R730安装配置1、开机设置Raid1服务器开机，系统自检，加载到PERC卡自检界面的时候，按<CTRL+R>进入PERC BIOS管理界面查看磁盘列表，各个磁盘状态为“Ready”高亮选中需要管理的PERC卡，按F2，选择弹出菜单里的“Create New VD”。
然后根据提示，选择slot0和slot1两块盘，创建Raid1。
高亮选中已创建的磁盘组，按F2，选择弹出菜单里的“Manage Ded.HS”选中对应的热备盘热备盘添加成功清除磁盘数据将Raid信息清除，然后重建Raid信息，选中ID:0的虚拟磁盘后按F2在选项中选择Fast Init就能清除数据。
2、HBA卡安装注意事项在安装HBA卡时就记住WWN号码和端口顺序，MD3800f存储每次刷新时显示顺序都会不一样。
3、linux 6.5安装（略）三、Dell MD3800f初始安装配置1、连接管理网口在电脑中安装Dell PowerVault MD Storage Manager，开机并连接到管理网口，将电脑的IP设置成与192.168.129.101同一网段的IP，通过Dell PowerVault MD Storage Manager连接到DELL MD3800f测试。
在Linux主机中也必须安装Dell PowerVault MD Storage Manager软件，否则会造成控制器自动切换问题。
Linux主机不需要配置同网段IP，只要光纤连接后使用软件的自动搜索即可。
RAID控制器模块网络配置可使用DHCP服务器（默认设置）进行分配。
但是，如果DHCP服务器在150秒内不可用，RAID控制器模块会分配静态IP地址。
初次连接或每次断开后，都需要等待150秒才能ping通。
2、查找添加存储在软件的“设置”菜单中添加存储阵列，可以使用自动搜索也可以使用带外管理手动输入主备控制器的IP地址进行添加。
右键点击添加的存储，选择“管理存储阵列”，将打开新的窗口对存储进行管理。
点击“存储和备份服务”，7块硬盘情况下，使用前6块创建磁盘组，使用Raid10，映射关系为0-1,2-3,4-5。
4、创建并初始化虚拟磁盘不分配。
新的虚拟磁盘初始化需要大概2个小时，选中虚拟磁盘后可以在右侧看到进度。
理盘”。
为磁盘组分配热备用物理磁盘6、添加主机和主机组选择存储阵列，在右键选项中选择定义主机填写主机名选择FC接口类型和主机端口标识，可通过下列命令确定FC HBA卡的WWN号[root@BOSS1~]#cat/sys/class/fc_host/host1/port_name0x21000024ff1111ca[root@BOSS1~]#cat/sys/class/fc_host/host2/port_name0x21000024ff1111d0确定是主机组命名主机组7、为主机组添加映射右键选择主机组“添加LUN映射”8、MD3800f存储摘要四、Linux多路径软件DMMP安装配置[root@BOSS1~]#fdisk-lDisk/dev/sdb:1771.7GB,1771674009600bytes255heads,63sectors/track,215393cylindersDisk/dev/sdc:3221MB,3221225472bytes100heads,62sectors/track,1014cylindersDisk/dev/sde:1771.7GB,1771674009600bytes255heads,63sectors/track,215393cylindersDisk/dev/sdd:2147MB,2147483648bytes67heads,62sectors/track,1009cylindersDisk/dev/sdg:2147MB,2147483648bytes67heads,62sectors/track,1009cylindersDisk/dev/sdf:3221MB,3221225472bytes100heads,62sectors/track,1014cylinders将MD3800f的卷映射到主机上以后，可以看到六块盘，需要通过linux自带的多路径软件聚合成三块。
主要用到下面这两个包device-mapper-1.02.63-4.el5.x86_64.rpmdevice-mapper-multipath-0.4.7-46.el5.x86_64.rpm查看包的安装情况[root@sms01Server]#rpm-q device-mapper-*package device-mapper-1.02.63-4.el5.i386.rpm is not installed package device-mapper-1.02.63-4.el5.x86_64.rpm is not installed package device-mapper-event-1.02.63-4.el5.x86_64.rpm is not installed package device-mapper-multipath-0.4.7-46.el5.x86_64.rpm is not installed安装底层模块[root@sms01Server]#rpm-Uvh device-mapper-1.02.63-4.el5.x86_64.rpm检查安装是否正常[root@sms01Server]#lsmod|grep dm_multipat安装多路径工具包[root@sms01]#rpm-Uvh device-mapper-multipath-0.4.7-46.el5.x86_64.rpm设置开机自启动[root@sms01Server]#chkconfig multipathd on接着通过命令：multipath–ll查看多路径状态查看模块是否加载成功[root@liujing~]#multipath–ll查看多路径状态Mar1019:18:28|/etc/multipath.conf does not exist,blacklisting all devices.Mar1019:18:28|A sample multipath.conf file is located atMar1019:18:28| /usr/share/doc/device-mapper-multipath-0.4.9/multipath.confMar1019:18:28|You can run/sbin/mpathconf to create or modify /etc/multipath.confMar1019:18:28|DM multipath kernel driver not loaded----DM模块没有加载如果模块没有加载成功请使用下列命初始化DM,或重启系统---Use the following commands to initialize and start DM for the first time:#modprobe dm-multipath#modprobe dm-round-robin#service multipathd start#multipath-v2初始化完了之后再通过multipath-ll命令查看是否加载成功[root@liujing~]#multipath-llMar1019:21:14|/etc/multipath.conf does not exist,blacklisting all devices.Mar1019:21:14|A sample multipath.conf file is located atMar1019:21:14| /usr/share/doc/device-mapper-multipath-0.4.9/multipath.confMar1019:21:14|You can run/sbin/mpathconf to create or modify /etc/multipath.confDM multipath kernel driver not loaded----这个提示没了说明DM模块已加载成功。
从上面的提示可以看到，DM模块是成功加载，但是/etc/下没有multipath.conf 配置文件编辑配置文件[root@sms01Server]#vi/etc/multipath.conf在linux6.5中需要手动创建此文件#cp/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf将blacklist{devnode"*"}注释掉并确保下面不被注释defaults{user_friendly_names yes}开启服务[root@sms01Server]#service multipathd startmultipath基本操作命令#/etc/init.d/multipathd start#开启mulitipath服务#multipath-F#删除现有没有使用的路径#multipath-v2#格式化路径，检测路径，合并路径双机中，如果映射3个卷，可能在主机是dm-0，dm-1，dm-2的顺序，而在另一台是dm-1，dm-0，dm-2的顺序。
为了保证双机的设备名一致，需在两台机器的multipath.conf配置文件中增加wwid对应名称的配置。
检测并查看WWN号[root@BOSS1Packages]#multipath-v2[root@BOSS1Packages]#multipath-llmpathd(3600a0980009883c8000001fd574c51d9)dm-2DELL,MD38xxfsize=2.0G features='0'hwhandler='0'wp=rw|-+-policy='round-robin0'prio=1status=active|`-1:0:0:3sdd8:48active ready running`-+-policy='round-robin0'prio=1status=enabled`-2:0:0:3sdg8:96active ready runningmpathc(3600a098000988461000002725747151e)dm-1DELL,MD38xxfsize=3.0G features='0'hwhandler='0'wp=rw|-+-policy='round-robin0'prio=1status=active|`-1:0:0:2sdc8:32active ready running`-+-policy='round-robin0'prio=1status=enabled`-2:0:0:2sdf8:80active ready runningmpathb(3600a0980009884610000026f574714fd)dm-0DELL,MD38xxfsize=1.6T features='0'hwhandler='0'wp=rw|-+-policy='round-robin0'prio=1status=active|`-1:0:0:1sdb8:16active ready running`-+-policy='round-robin0'prio=1status=enabled`-2:0:0:1sde8:64active ready running配置文件中增加对应multipaths{multipath{wwid3600a0980009884610000026f574714fdalias mpathb}multipath{wwid3600a098000988461000002725747151ealias mpathc}multipath{wwid3600a0980009883c8000001fd574c51d9alias mpathd}}在使用fdisk-l查看的时候，会看到多出的聚合后设备Disk/dev/mapper/mpathb:1771.7GB,1771674009600bytes255heads,63sectors/track,215393cylindersUnits=cylinders of16065*512=8225280bytesSector size(logical/physical):512bytes/512bytesI/O size(minimum/optimal):512bytes/512bytesDisk identifier:0x00000000Disk/dev/mapper/mpathc:3221MB,3221225472bytes255heads,63sectors/track,391cylindersUnits=cylinders of16065*512=8225280bytesSector size(logical/physical):512bytes/512bytesI/O size(minimum/optimal):512bytes/512bytesDisk identifier:0x00000000Disk/dev/mapper/mpathd:2147MB,2147483648bytes255heads,63sectors/track,261cylindersUnits=cylinders of16065*512=8225280bytesSector size(logical/physical):512bytes/512bytesI/O size(minimum/optimal):512bytes/512bytesDisk identifier:0x00000000使用fdisk对/dev/mapper/mpathb进行分区，[root@sms02starboss]#fdisk/dev/mapper/mpathbDevice contains neither a valid DOS partition table,nor Sun,SGI or OSF disklabelBuilding a new DOS disklabel with disk identifier0xac956c3a. Changes will remain in memory only,until you decide to write them. After that,of course,the previous content won't be recoverable. Warning:invalid flag0x0000of partition table4will be corrected by w(rite)WARNING:DOS-compatible mode is deprecated.It's strongly recommended to switch off the mode(command'c')and change display units tosectors(command'u').Command(m for help):n------------------------新建分区Command actione extendedp primary partition(1-4)p-----------------------------主分区Partition number(1-4):1First cylinder(1-1016,default1):Using default value1Last cylinder,+cylinders or+size{K,M,G}(1-1016,default1016): Using default value1016Command(m for help):w---------------------写入列表相当于保存The partition table has been altered!Calling ioctl()to re-read partition table.Syncing disks.分区后会出现/dev/mapper/mpathbp1如果是超过2T需要用GPT进行分区，oracle不能用超过2T的分区[root@sms02starboss]#parted/dev/mapper/mpathb//用parted命令分区GNU Parted 2.1Using/dev/mapper/mpathbWelcome to GNU Parted!Type'help'to view a list of commands. (parted)mklabel GPT//建立磁盘labelWarning:The existing disk label on/dev/mapper/mpathb will be destroyed and alldata on this disk will be lost.Do you want to continue?Yes/No?y(parted)mkpart//创建分区Partition name?[]?ora01//分区名称File system type?[ext2]?ext4//文件系统Start?1End?2577GB//大小(parted)p//查看是否成功Model:Linux device-mapper(multipath)(dm)Disk/dev/mapper/mpathb:2577GBSector size(logical/physical):512B/512BPartition Table:gptNumber Start End Size File system Name Flags11049kB2577GB2577GB ora01(parted)quitInformation:You may need to update/etc/fstab.使用mkfs.ext4/dev/mapper/mpathbp1格式化只能用/dev/mapper/mpathbp1挂载多路径测试：启用多路径软件前设备识别/dev/sdb和/dev/sdc，拔下一个控制器上的FC线，/dev/sdb消失。