NVMe SW RAID 구성

# lspci | grep -i nvme
06:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
08:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
09:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

# ls -1 /dev/nvme*
/dev/nvme0
/dev/nvme0n1
/dev/nvme1
/dev/nvme1n1
/dev/nvme2
/dev/nvme2n1
/dev/nvme3
/dev/nvme3n1

# mdadm --examine /dev/nvme[0-3]n1
mdadm: No md superblock detected on /dev/nvme0n1.
mdadm: No md superblock detected on /dev/nvme1n1.
mdadm: No md superblock detected on /dev/nvme2n1.
mdadm: No md superblock detected on /dev/nvme3n1.

# (참고) 사용중이면 다음과 같이 표시됨
# mdadm --examine /dev/nvme[0-3]n1
/dev/nvme0n1:
   MBR Magic : aa55
Partition[0] :   2000407216 sectors at         2048 (type fd)
...

# /dev/nvme0n1 ~ /dev/nvme3n1 총 4개의 디스크에 대해 다음 명령어 수행

# fdisk /dev/nvme0n1
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/nvme0n1: 1024.2 GB, 1024209543168 bytes, 2000409264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xb1aad99b

        Device Boot      Start         End      Blocks   Id  System
/dev/nvme0n1p1            2048  2000409263  1000203608   83  Linux

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): p

Disk /dev/nvme0n1: 1024.2 GB, 1024209543168 bytes, 2000409264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xb1aad99b

        Device Boot      Start         End      Blocks   Id  System
/dev/nvme0n1p1            2048  2000409263  1000203608   fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

# ls -1 /dev/nvme* | grep p
/dev/nvme0n1p1
/dev/nvme1n1p1
/dev/nvme2n1p1
/dev/nvme3n1p1

# mdadm --create /dev/md0 --level=stripe --raid-devices=4 /dev/nvme[0-3]n1p1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 nvme3n1p1[3] nvme2n1p1[2] nvme1n1p1[1] nvme0n1p1[0]
      4000285696 blocks super 1.2 512k chunks

unused devices: <none>

# mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Thu Aug 29 13:25:32 2019
        Raid Level : raid0
        Array Size : 4000285696 (3814.97 GiB 4096.29 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Thu Aug 29 13:25:32 2019
             State : clean
    Active Devices : 4
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 0

        Chunk Size : 512K

Consistency Policy : none

              Name : kisti100g:0  (local to host kisti100g)
              UUID : 02770dbe:a0593132:5dfeaec8:e1938f8b
            Events : 0

    Number   Major   Minor   RaidDevice State
       0     259        4        0      active sync   /dev/nvme0n1p1
       1     259        5        1      active sync   /dev/nvme1n1p1
       2     259        6        2      active sync   /dev/nvme2n1p1
       3     259        7        3      active sync   /dev/nvme3n1p1

# mdadm -E /dev/nvme[0-3]n1p1
/dev/nvme0n1p1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 02770dbe:a0593132:5dfeaec8:e1938f8b
           Name : kisti100g:0  (local to host kisti100g)
  Creation Time : Thu Aug 29 13:25:32 2019
     Raid Level : raid0
   Raid Devices : 4

 Avail Dev Size : 2000143024 (953.74 GiB 1024.07 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : clean
    Device UUID : e4dc278f:ef805f56:b944e294:1f7ed526

    Update Time : Thu Aug 29 13:25:32 2019
  Bad Block Log : 512 entries available at offset 8 sectors
       Checksum : 65d21e87 - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
...

# mkfs.xfs /dev/md0
# mkdir /DMC
# chmod 777 DMC
# mount /dev/md0 /DMC

# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  1.6G   49G   4% /
devtmpfs                  63G     0   63G   0% /dev
tmpfs                     63G     0   63G   0% /dev/shm
tmpfs                     63G   11M   63G   1% /run
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/sda2               1014M  146M  869M  15% /boot
/dev/mapper/centos-home   14T   33M   14T   1% /home
tmpfs                     13G     0   13G   0% /run/user/0
/dev/md0                 3.8T   34M  3.8T   1% /DMC

# mdadm -E -s -v >> /etc/mdadm.conf
# mdadm --detail --scan --verbose >> /etc/mdadm.conf

# vi /etc/fstab
# 마지막에 다음 줄 추가
/dev/md0    /DMC    xfs    defaults    0 0