在使用儲存服務的時候，時常會根據使用者的權限(付的錢？XD)來決定這個使用者可以使用哪些硬體支援，常見的像是 SSD, HDD 或者 Tape 等等。
本篇就是專門介紹在 Ceph 中透過 Crush Rule 將硬體支援歸納出多種不同的 Partitions，並且配置每個 RGW 使用者可以儲存的 Placement tags 與 default Placement，以達到如上述限制使用者可儲存的硬體支援。
前置條件
- 需已建置一個 Ceph 叢集
- Ceph 叢集需提供 RGW 服務
環境
| 參數 |
數值 |
| Operating System |
CentOS 7 |
| Host Name |
ceph |
| Private Network |
172.17.1.100 |
| Ceph Version |
14.2.2 |
配置
確認當前 Crush Map 包含兩個 root
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| $ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-9 3.00000 root slow
-10 1.00000 host OSD04
3 hdd 1.00000 osd.3 up 1.00000 1.00000
-11 1.00000 host OSD05
4 hdd 1.00000 osd.4 up 1.00000 1.00000
-12 1.00000 host OSD06
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-1 3.00000 root default
-3 1.00000 host OSD01
0 hdd 1.00000 osd.0 up 1.00000 1.00000
-4 1.00000 host OSD02
1 hdd 1.00000 osd.1 up 1.00000 1.00000
-5 1.00000 host OSD03
2 hdd 1.00000 osd.2 up 1.00000 1.00000
|
建立一個新的 Crush Rule 名為 slow 並且 Type 為 replicated
1
| $ ceph osd crush rule create-replicated slow slow host
|
確認當前已存在的 Crush Rule
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| $ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "slow",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -9,
"item_name": "slow"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
|
在 default Zonegroup 增加新的 Placement target 名為 slow 並且賦予 tag 為 slow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| $ radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id slow --tags slow
[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
},
{
"key": "slow",
"val": {
"name": "slow",
"tags": [
"slow"
],
"storage_classes": [
"STANDARD"
]
}
}
]
|
替 default-placement Placement target 新增名為 default-placement tag
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| $ radosgw-admin zonegroup placement modify --rgw-zonegroup default --placement-id default-placement --tags default-placement
[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [
"default-placement"
],
"storage_classes": [
"STANDARD"
]
}
},
{
"key": "slow",
"val": {
"name": "slow",
"tags": [
"slow"
],
"storage_classes": [
"STANDARD"
]
}
}
]
|
在 default Zone 增加新的 placement pool，並且命名為 slow。建置過程中需配置 index-pool, data-pool 與 –data-extra-pool
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
| $ radosgw-admin zone placement add --rgw-zone default --placement-id slow --index-pool slow.rgw.buckets.index --data-pool slow.rgw.buckets.data --data-extra-pool slow.rgw.buckets.non-ec
{
"id": "67798670-9929-4675-a595-a27b10576b5e",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "slow",
"val": {
"index_pool": "slow.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "slow.rgw.buckets.data"
}
},
"data_extra_pool": "slow.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": "",
"realm_id": ""
}
|
配置完成後，建立相對應的 pools
1
2
3
4
5
6
7
8
9
10
11
| $ ceph osd pool create slow.rgw.buckets.index 8 8
pool 'slow.rgw.buckets.index' created
$ ceph osd pool create slow.rgw.buckets.data 8 8
pool 'slow.rgw.buckets.data' created
$ ceph osd pool create slow.rgw.buckets.non-ec 8 8
pool 'slow.rgw.buckets.non-ec' created
|
更改新建立 pools 的 Crush Rule
1
2
3
4
5
6
7
8
9
10
11
| $ ceph osd pool set slow.rgw.buckets.index crush_rule slow
set pool 6 crush_rule to slow
$ ceph osd pool set slow.rgw.buckets.data crush_rule slow
set pool 7 crush_rule to slow
$ ceph osd pool set slow.rgw.buckets.non-ec crush_rule slow
set pool 8 crush_rule to slow
|
確認新建立 pool 的 Crush Rule
1
2
3
4
5
6
7
| $ ceph osd pool ls detail
...
pool 5 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 31 flags hashpspool stripe_width 0 application rgw
pool 6 'slow.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 68 flags hashpspool stripe_width 0
pool 7 'slow.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 70 flags hashpspool stripe_width 0
pool 8 'slow.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 72 flags hashpspool stripe_width 0
|
crush_rule 為 1 表示已更改為 slow Crush Rule，若讀者實際操作可能值會與範例不同
重新啟動 RGW
1
| $ systemctl restart ceph-radosgw@radosgw.gateway
|
建立測試使用者
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
| $ radosgw-admin user create --uid only-slow --display-name "only-slow"
{
"user_id": "only-slow",
"display_name": "only-slow",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "only-slow",
"access_key": "91W6AWWF1J9OIPBVQZG7",
"secret_key": "GsQzXqK9nfXmAvuiXF1hg2VIfrvffppeqaCaoyMJ"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
|
將使用者的 metadata 匯出並且更改 default_placement 與 placement_tags 再匯入
1
2
3
4
5
6
7
8
9
10
11
| $ radosgw-admin metadata get user:only-slow > only-slow.json
$ vim only-slow.json
...
"op_mask": "read, write, delete",
"default_placement": "slow",
"default_storage_class": "",
"placement_tags": ["slow"],
...
$ radosgw-admin metadata put user:only-slow < only-slow.json
|
結果
利用 boto3 套件進行測試，測試程式碼如下：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| import boto3
bucket = "test2"
#location = "default:default-placement"
location = "default:slow"
s3 = boto3.client(
's3',
endpoint_url="http://172.17.1.100:7480",
aws_access_key_id="91W6AWWF1J9OIPBVQZG7",
aws_secret_access_key="GsQzXqK9nfXmAvuiXF1hg2VIfrvffppeqaCaoyMJ",
)
s3.create_bucket(
Bucket=bucket,
CreateBucketConfiguration={'LocationConstraint':location},
)
|
location 為 <region>:<placement-id>，使用 default:slow 可成功建立，而 default:defualt-placement 會得到 AccessDenied 結果。
參考