ceph分布式安装 andrew 2023-08-11 2023-08-16 1.环境配置
1 2 3 cd /etc/yum.repos.d/mkdir /tmp/bakmv * /tmp/bak/
#配置系统源码，epel源：
1 2 3 4 5 curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo yum install wget -y wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo yum -y install yum-plugin-priorities.noarch
#配置ceph源：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 cat << EOF | tee /etc/yum.repos.d/ceph.repo [Ceph] name=Ceph packages for $basearch baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/\$basearch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc priority=1 [Ceph-noarch] name=Ceph noarch packages baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/noarch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/SRPMS enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc EOF
#关闭防火墙：
1 2 3 systemctl stop firewalld systemctl disable firewalld systemctl status firewalld
#配置主机名称：
1 2 3 4 5 6 ceph1节点： hostnamectl --static set-hostname ceph1 ceph2节点： hostnamectl --static set-hostname ceph2 ceph3节点： hostnamectl --static set-hostname ceph3
#所有节点配置hosts文件：
1 2 3 4 5 6 /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.231 ceph1 192.168.0.232 ceph2 192.168.0.233 ceph3
#所有节点NTP配置：
1 2 3 4 5 6 7 8 9 yum -y install ntp ntpdate vi /etc/ntp.conf restrict 127.0.0.1 restrict ::1 restrict 192.168.3.0 mask 255.255.255.0 //ceph1的网段与掩码 server 127.127.1.0 fudge 127.127.1.0 stratum 8
在ceph2、ceph3及所有客户机节点新建NTP文件。
1 2 3 4 5 6 7 vi /etc/ntp.conf server 192.168.3.166 systemctl start ntpd systemctl enable ntpd systemctl status ntpd
#ssh配置，在ceph1节点生成公钥，并发放到各个主机/客户机节点。：
1 2 3 ssh-keygen -t rsa for i in {1..3}; do ssh-copy-id ceph$i ; done for i in {1..3}; do ssh-copy-id client$i ; done
#在所有节点，关闭SELinux
1 2 sed -i 's/enforcing/disabled/' /etc/selinux/config setenforce 0
安装Ceph软件1 2 3 4 5 6 7 8 yum -y install ceph ceph -v命令查看版本: [root@ceph1 ~] ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) [root@ceph2 ~] ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) [root@ceph3 ~] ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
1 yum -y install ceph-deploy
1 2 3 mkdir cluster cd cluster ceph-deploy new ceph1 ceph2 ceph3
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 [root@ceph1 ~] [root@ceph1 cluster] [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy new ceph1 ceph2 ceph3 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] func : <function new at 0x7ffb7dc07de8> [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7ffb7d58c6c8> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : ['ceph1' , 'ceph2' , 'ceph3' ] [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph1][DEBUG ] find the location of an executable [ceph1][INFO ] Running command : /usr/sbin/ip link show [ceph1][INFO ] Running command : /usr/sbin/ip addr show [ceph1][DEBUG ] IP addresses found: [u'192.168.0.231' ] [ceph_deploy.new][DEBUG ] Resolving host ceph1 [ceph_deploy.new][DEBUG ] Monitor ceph1 at 192.168.0.231 [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph2][DEBUG ] connected to host: ceph1 [ceph2][INFO ] Running command : ssh -CT -o BatchMode=yes ceph2 [ceph2][DEBUG ] connected to host: ceph2 [ceph2][DEBUG ] detect platform information from remote host [ceph2][DEBUG ] detect machine type [ceph2][DEBUG ] find the location of an executable [ceph2][INFO ] Running command : /usr/sbin/ip link show [ceph2][INFO ] Running command : /usr/sbin/ip addr show [ceph2][DEBUG ] IP addresses found: [u'192.168.0.232' ] [ceph_deploy.new][DEBUG ] Resolving host ceph2 [ceph_deploy.new][DEBUG ] Monitor ceph2 at 192.168.0.232 [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph3][DEBUG ] connected to host: ceph1 [ceph3][INFO ] Running command : ssh -CT -o BatchMode=yes ceph3 [ceph3][DEBUG ] connected to host: ceph3 [ceph3][DEBUG ] detect platform information from remote host [ceph3][DEBUG ] detect machine type [ceph3][DEBUG ] find the location of an executable [ceph3][INFO ] Running command : /usr/sbin/ip link show [ceph3][INFO ] Running command : /usr/sbin/ip addr show [ceph3][DEBUG ] IP addresses found: [u'192.168.0.233' ] [ceph_deploy.new][DEBUG ] Resolving host ceph3 [ceph_deploy.new][DEBUG ] Monitor ceph3 at 192.168.0.233 [ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph1' , 'ceph2' , 'ceph3' ] [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.0.231' , '192.168.0.232' , '192.168.0.233' ] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
3.2 初始化密钥
1 ceph-deploy mon create-initial
3.3 将ceph.client.admin.keyring拷贝到各个节点上
1 ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3
3.4 查看是否配置成功。
4 部署MGR节点
1 ceph-deploy mgr create ceph1 ceph2 ceph3
查看MGR是否部署成功。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 ceph -s [root@ceph1 cluster] ``` 5 部署OSD节点 ```bash ceph-deploy osd create --data /dev/sdb ceph1 ceph-deploy osd create --data /dev/sdc ceph1 ceph-deploy osd create --data /dev/sdd ceph1 ceph-deploy osd create --data /dev/sdb ceph2 ceph-deploy osd create --data /dev/sdc ceph2 ceph-deploy osd create --data /dev/sdd ceph2 ceph-deploy osd create --data /dev/sdb ceph3 ceph-deploy osd create --data /dev/sdc ceph3 ceph-deploy osd create --data /dev/sdd ceph3
创建成功后，查看是否正常
6 验证Ceph
1 ceph osd pool create vdbench 10 10
创建块设备
1 2 3 4 5 rbd create image01 --size 200--pool vdbench --image-format 2 --image-feature layering rbd ls --pool vdbench [root@ceph1 cluster] [root@ceph1 cluster] image01
#PG 分 配 计 算
二.熟练 ceph 的用户管理及授权
1 2 3 4 5 6 7 root@ceph-deploy:~ [client.admin] key = AQBnFaNj1iyBMBAAd+9hKWXaNw3GYxT9PEXvrQ== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
#列 出 指 定 用 户 信 息
1 2 3 4 5 6 7 8 root@ceph-deploy:~ [osd.10] key = AQB+I6Njk4KWNBAAL09FFayLKF44IgUQ1fjKYQ== caps mgr = "allow profile osd" caps mon = "allow profile osd" caps osd = "allow *" exported keyring for osd.10
#: 列 出 用 户
1 2 3 4 5 6 7 8 9 10 11 cephadmin@ceph-deploy:~$ ceph auth list mds.ceph-mgr1 key: AQAdRbFjOwBXIRAAUTdwElBzYPHW+4uFicFC7Q== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx osd.0 key: AQC0IqNjbcgKIxAA+BCNpQeZiMujR+r+69Miig== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow *
#可以结合使用-o 文件名选项和 ceph auth list 将输出保存到某个文件。
1 [cephadmin@ceph-deploy ceph-cluster]$ ceph auth list -o 123.key
#ceph auth add
1 2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=testpool2' added key for client.tom
##验证 key
1 2 3 4 5 6 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get client.tom [client.tom] key = AQD2vbJj8fIiDBAArtJBzQiuPy8nDWPSFVs0bw== caps mon = "allow r" caps osd = "allow rwx pool=testpool2" exported keyring for client.tom
##创建用户
1 2 3 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create client.jack mon 'allow r' osd 'allow rwx pool=testpool2' [client.jack] key = AQC/vrJj5kenHhAAGeRJpY64feS4Dn6DD/R8VA==
##再次创建用户
1 2 3 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get-or-create client.jack mon 'allow r' osd 'allow rwx pool=testpool2' [client.jack] key = AQC/vrJj5kenHhAAGeRJpY64feS4Dn6DD/R8VA==
#ceph auth get-or-create-key:
1 2 3 [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get-or-create-key client.jack mon 'allow r' osd 'allow rwx pool=mypool' AQAtr8dfi37XMhAADbHWEZ0shY1QZ5A8eBpeoQ==
用户有 key 就显示没有就创建
1 2 3 4 5 6 7 8 9 10 11 12 13 14 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get client.jack [client.jack] key = AQC/vrJj5kenHhAAGeRJpY64feS4Dn6DD/R8VA== caps mon = "allow r" caps osd = "allow rwx pool=testpool2" exported keyring for client.jack cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth caps client.jack mon 'allow r' osd 'allow rw pool=testpool2' updated caps for client.jack cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get client.jack [client.jack] key = AQC/vrJj5kenHhAAGeRJpY64feS4Dn6DD/R8VA== caps mon = "allow r" caps osd = "allow rw pool=testpool2" exported keyring for client.jack
#删 除 用 户
1 2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth del client.tom updated
#导出 keyring 至指定文件
1 2 3 [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get client.user1 -o ceph.client.user1.keyring exported keyring for client.user1
#验证指定用户的 keyring 文件：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 [cephadmin@ceph-deploy ceph-cluster]$ cat ceph.client.user1.keyring [client.user1] key = AQAUUchfjpMqGRAARV6h0ofdDEneuaRnxuHjoQ== caps mon = "allow r" caps osd = "allow * pool=mypool" [cephadmin@ceph-deploy ceph-cluster]$ ceph auth del client.user1 Updated [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get client.user1 Error ENOENT: failed to find client.user1 in keyring [cephadmin@ceph-deploy ceph-cluster]$ ceph auth import -i ceph.client.user1.keyring imported keyring [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get client.user1 exported keyring for client.user1
#将多 用 户 导 出 至 秘 钥 环 ：
1 2 $ ceph-authtool --create-keyring ceph.client.user.keyring creating ceph.client.user.keyring
#把指定的 admin 用户的 keyring 文件内容导入到 user 用户的 keyring 文件：
1 2 3 $ceph -authtool ./ceph.client.user.keyring--import-keyring ./ceph.client.admin.keyring importing contents of ./ceph.client.admin.keyring into ./ceph.client.user.keyring
#验证 keyring 文件：
1 2 3 4 5 6 7 [cephadmin@ceph-deploy ceph-cluster]$ ceph-authtool -l ./ceph.client.user.keyring [client.admin] key = AQAGDKJfQk/dAxAA3Y+9xoE/p8in6QjoHeXmeg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
#再导入一个其他用户的 keyring：
1 2 3 [cephadmin@ceph-deploy ceph-cluster]$ ceph-authtool ./ceph.client.user.keyring --import-keyring ./ceph.client.user1.keyring importing contents of ./ceph.client.user1.keyring into ./ceph.client.user.keyring
#再次验证 keyring 文件是否包含多个用户的认证信息：
1 2 3 4 5 6 7 8 9 10 11 [cephadmin@ceph-deploy ceph-cluster]$ ceph-authtool -l ./ceph.client.user.keyring [client.admin] key = AQAGDKJfQk/dAxAA3Y+9xoE/p8in6QjoHeXmeg== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.user1] key = AQAUUchfjpMqGRAARV6h0ofdDEneuaRnxuHjoQ== caps mon = "allow r" caps osd = "allow * pool=mypool"
三. 使用普通客户挂载块存储
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 cephadmin@ceph-deploy:~/ceph-cluster$ceph osd pool create rbd-data1 32 32 cephadmin@ceph-deploy:~/ceph-cluster$ceph osd pool application enable rbd-data1 rbd cephadmin@ceph-deploy:~/ceph-cluster$rbd pool init -p rbd-data1 cephadmin@ceph-deploy:~/ceph-cluster$rbd create data-img1 --size 3G --pool rbd-data1 --image-format 2 --image-feature layering cephadmin@ceph-deploy:~/ceph-cluster$rbd create data-img2 --size 5G --pool rbd-data1 --image-format 2 --image-feature layering cephadmin@ceph-deploy:~/ceph-cluster$rbd ls --pool rbd-data1 cephadmin@ceph-deploy:~/ceph-cluster$rbd ls --pool rbd-data1 -l --format json --pretty-format ceph auth add client.shijie mon 'allow r' osd 'allow rwx pool=rbd-data1' ceph auth get client.shijie cephadmin@ceph-deploy:~/ceph-cluster$ ceph-authtool --create-keyring ceph.client.shijie.keyring cephadmin@ceph-deploy:~/ceph-cluster$ ceph auth get client.shijie -o ceph.client.shijie.keyring cephadmin@ceph-deploy:~/ceph-cluster$cat ceph.client.shijie.keyring cephadmin@ceph-deploy:~/ceph-cluster$ scp ceph.client.shijie.keyring root@172.31.6.109:/etc/ceph/ cephadmin@ceph-deploy:~/ceph-cluster$rbd ls -p rbd-data1 -l root@ceph-node4:/ /dev/rbd0 root@ceph-node4:/ rbd0 root@ceph-node4:/ root@ceph-node4:/ root@ceph-node4:/ 48374db8541a7fa375c00611373051ef21690e89adfd4c156b3f6ffb0dbe95a2 root@ceph-node4:/data ibdata1 ib_logfile0 ib_logfile1 mysql performance_schema test root@ceph-node4:/etc/ceph /dev/rbd1 root@ceph-node4:/etc/ceph root@ceph-node4:/etc/ceph root@ceph-node4:/etc/ceph root@ceph-node4:/etc/ceph root@ceph-node4:/etc/ceph root@ceph-node4:/data1 auth.log lost+found 四. 使用普通用户挂载 cephfs（可以通过 secret 或者 secretfile 的形式多主机同时挂载） Ceph FS 需要运行 Meta Data Services(MDS)服务，其守护进程为 ceph-mds，ceph-mds进程管理与 cephFS 上存储的文件相关的元数据，并协调对 ceph 存储集群的访问。 cephadmin@ceph-deploy:~/ceph-cluster$ apt-cache madison ceph-mds ceph-mds | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages cephadmin@ceph-deploy:~/ceph-cluster$ apt install ceph-mds=16.2.10-1bionic [cephadmin@ceph-deploy ceph-cluster]$ ceph osd pool create cephfs-metadata 32 32 pool 'cephfs-metadata' created [cephadmin@ceph-deploy ceph-cluster]$ ceph osd pool create cephfs-data 64 64 pool 'cephfs-data' created [cephadmin@ceph-deploy ceph-cluster]$ ceph fs new mycephfs cephfs-metadata cephfs-data new fs with metadata pool 7 and data pool 8 cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs ls name: mycephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data ] cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs mycephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 10 13 12 0 POOL TYPE USED AVAIL cephfs-metadata metadata 146k 18.9T cephfs-data data 0 18.9T MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) cephadmin@ceph-deploy:~$ ceph mds stat mycephfs:1 {0=ceph-mgr1=up:active} cephadmin@ceph-deploy:~/ceph-cluster$ceph auth add client.yanyan mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data' [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get client.yanyan exported keyring for client.yanyan [client.yanyan] key = AQCxpdhfjQt1OxAAGe0mqTMveNu2ZMEem3tb0g== caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rwx pool=cephfs-data" [cephadmin@ceph-deploy ceph-cluster]$ ceph auth get client.yanyan -o ceph.client.yanyan.keyring exported keyring for client.yanyan [cephadmin@ceph-deploy ceph-cluster]$ ceph auth print-key client.yanyan > yanyan.key [cephadmin@ceph-deploy ceph-cluster]$ cat ceph.client.yanyan.keyring [client.yanyan] key = AQCxpdhfjQt1OxAAGe0mqTMveNu2ZMEem3tb0g== caps mds = "allow rw" caps mon = "allow r" caps osd = "allow rwx pool=cephfs-data" [cephadmin@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.yanyan.keyring yanyan.key root@172.31.6.109:/etc/ceph/ root@ceph-node4:~ cluster: id : 7c088d6f-06b0-4584-b23f-c0f150af51d4 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 24m) mgr: ceph-mgr1(active, since 65m) mds: 1/1 daemons up osd: 16 osds: 16 up (since 24m), 16 in (since 13d) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 329 pgs objects: 296 objects, 218 MiB usage: 948 MiB used, 60 TiB / 60 TiB avail pgs: 329 active+clean root@ceph-node4:~ root@ceph-node4:/etc/ceph root@ceph-node4:/etc/ceph Filesystem Size Used Avail Use% Mounted on udev 955M 0 955M 0% /dev 172.31.6.101:6789,172.31.6.102:6789,172.31.6.103:6789:/ 19T 0 19T 0% /data root@ceph-node3:~ root@ceph-node3:~ root@ceph-node3:~ Filesystem Size Used Avail Use% Mounted on udev 955M 0 955M 0% /dev 172.31.6.101:6789,172.31.6.102:6789,172.31.6.103:6789:/ 19T 0 19T 0% /data root@ceph-node3:~ root@ceph-node3:~ root@ceph-node3:/data auth.log root@ceph-node3:/data root@ceph-node3:/data root@ceph-node3:/data root@ceph-node4:/ Jan 4 22:25:01 ceph-node3 CRON[4365]: pam_unix(cron:session): session closed for user root 12345678 testlog root@ceph-node4:/ ceph 380928 1 libceph 315392 1 ceph fscache 65536 1 ceph libcrc32c 16384 5 nf_conntrack,nf_nat,xfs,raid456,libceph root@ceph-node4:/ filename: /lib/modules/4.15.0-130-generic/kernel/fs/ceph/ceph.ko license: GPL description: Ceph filesystem for Linux author: Patience Warnick <patience@newdream.net> author: Yehuda Sadeh <yehuda@hq.newdream.net> author: Sage Weil <sage@newdream.net> alias : fs-cephsrcversion: CB79D9E4790452C6A392A1C depends: libceph,fscache retpoline: Y intree: Y name: ceph vermagic: 4.15.0-130-generic SMP mod_unload signat: PKCS signer: sig_key: sig_hashalgo: md4
五.实现 MDS 服务的多主一备高可用架构
#当前mds服务器状态
1 2 3 4 5 6 7 8 9 10 11 12 13 14 cephadmin@ceph-deploy:~$ ceph mds stat mycephfs:1 {0=ceph-mgr1=up:active} 将 ceph-mgr2 和 ceph-mon2 和 ceph-mon3 作为 mds 服务角色添加至 ceph 集群，最后实两主两备的 mds 高可用和高性能结构。 [root@ceph-mgr2 ~] [root@ceph-mon2 ~] [root@ceph-mon3 ~] cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mgr2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mon2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph-deploy mds create ceph-mon3
#验证 mds 服务器当前状态：
1 2 [cephadmin@ceph-deploy ceph-cluster]$ ceph mds stat mycephfs-1/1/1 up {0=ceph-mgr1=up:active}, 3 up:standby
#验证 ceph集群当前状态
1 2 3 4 5 6 7 8 cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 16 14 12 0 POOL TYPE USED AVAIL cephfs-metadata metadata 282k 18.9T cephfs-data data 564k 18.9T STANDBY MDS
#当前的文件系统状态:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs get mycephfs Filesystem 'mycephfs' (1) fs_name mycephfs epoch 28 flags 12 created 2023-01-01T19:09:29.258956+0800 modified 2023-01-05T14:58:00.369468+0800 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 406 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs ,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0up {0=144106} failed damaged stopped data_pools [8] metadata_pool 7 inline_data disabled balancer standby_count_wanted 1 [mds.ceph-mgr1{0:144106} state up:active seq 27 addr [v2:172.31.6.104:6800/428364709,v1:172.31.6.104:6801/428364709] compat {c=[1],r=[1],i=[7ff]}] 目前有四个 mds 服务器，但是有一个主三个备，可以优化一下部署架构，设置为为两主两备。 cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs set mycephfs max_mds 2 cephadmin@ceph-deploy:~/ceph-cluster$ ceph fs status mycephfs - 0 clients RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mgr1 Reqs: 0 /s 16 14 12 0 1 active ceph-mon2 Reqs: 0 /s 10 13 11 0 POOL TYPE USED AVAIL cephfs-metadata metadata 354k 18.9T cephfs-data data 564k 18.9T STANDBY MDS ceph-mgr2 ceph-mon3 MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
安装完成后ceph -s提示：“mon is allowing insecure global_id reclaim”。
解决方案：禁用不安全模式
1 ceph config set mon auth_allow_insecure_global_id_reclaim false
输入完成后需要等待5-10秒。
PG数量计算公式：
（总OSD数量*100）/ 副本数（复制分数，默认为3）= PG数量
一般情况下结果取2的N次方，尽量先设置小点，后期可以增大。如果先设置的比较大的话后期减小风险较高，所以尽量取小的2的N次方结果。
1 ceph osd pool create rbd 128 128
初始化存储池
1.设置存储池副本数
1 2 3 4 [root@controller-1 ~] size: 2 [root@controller-1 ~] set pool 1 size to 1
升级client的虚拟机内核到5版本
修改client下文件权限
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 chmod +r /etc/ceph/ceph.client.admin.keyringclient节点创建设备镜像，单位是M rbd create --size 4096 --pool rbd img client节点映射镜像到主机： rbd map img --name client.admin client节点格式化块设备 mkfs.ext4 -m 0 /dev/rbd/rbd/foo client节点挂载mount块设备 mkdir /mnt/ceph-block-devicemount /dev/rbd/rbd/foo /mnt/ceph-block-device -o discard 创建文件系统时报不支持EC数据池问题 当拥有一个ceph_metadata元数据副本类型池和一个ceph_data数据EC类型池时，建立文件系统： ceph fs new cephfs cephfs_metadata cephfs_data N版可能会报： Error EINVAL: pool 'cephfs_data' (id '11' ) is an erasure-coded pool. Use of an EC pool for the default data pool is discouraged; see the online CephFS documentation for more information. Use --force to override. 加上—force后： ceph fs new cephfs cephfs_metadata cephfs_data --force 也可能会报： Error EINVAL: pool 'cephfs_data' (id '11' ) is an erasure-coded pool, with no overwrite support 这是需要手动设置ceph_data池 allow_ec_overwrites=true ceph osd pool set cephfs_data allow_ec_overwrites true 再执行 ceph fs new cephfs cephfs_metadata cephfs_data –forc 就可以创建文件系统成功。
