Ceph Object Storage Placement 介紹

在使用儲存服務的時候,時常會根據使用者的權限(付的錢?XD)來決定這個使用者可以使用哪些硬體支援,常見的像是 SSD, HDD 或者 Tape 等等。

本篇就是專門介紹在 Ceph 中透過 Crush Rule 將硬體支援歸納出多種不同的 Partitions,並且配置每個 RGW 使用者可以儲存的 Placement tags 與 default Placement,以達到如上述限制使用者可儲存的硬體支援。

前置條件

  • 需已建置一個 Ceph 叢集
  • Ceph 叢集需提供 RGW 服務

環境

參數 數值
Operating System CentOS 7
Host Name ceph
Private Network 172.17.1.100
Ceph Version 14.2.2

配置

確認當前 Crush Map 包含兩個 root

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ ceph osd tree

ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-9 3.00000 root slow
-10 1.00000 host OSD04
3 hdd 1.00000 osd.3 up 1.00000 1.00000
-11 1.00000 host OSD05
4 hdd 1.00000 osd.4 up 1.00000 1.00000
-12 1.00000 host OSD06
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-1 3.00000 root default
-3 1.00000 host OSD01
0 hdd 1.00000 osd.0 up 1.00000 1.00000
-4 1.00000 host OSD02
1 hdd 1.00000 osd.1 up 1.00000 1.00000
-5 1.00000 host OSD03
2 hdd 1.00000 osd.2 up 1.00000 1.00000

建立一個新的 Crush Rule 名為 slow 並且 Type 為 replicated

1
$ ceph osd crush rule create-replicated slow slow host

確認當前已存在的 Crush Rule

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ ceph osd crush rule dump

[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "slow",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -9,
"item_name": "slow"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]

default Zonegroup 增加新的 Placement target 名為 slow 並且賦予 tag 為 slow

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id slow --tags slow

[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
},
{
"key": "slow",
"val": {
"name": "slow",
"tags": [
"slow"
],
"storage_classes": [
"STANDARD"
]
}
}
]

default-placement Placement target 新增名為 default-placement tag

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ radosgw-admin zonegroup placement modify --rgw-zonegroup default --placement-id default-placement --tags default-placement

[
{
"key": "default-placement",
"val": {
"name": "default-placement",
"tags": [
"default-placement"
],
"storage_classes": [
"STANDARD"
]
}
},
{
"key": "slow",
"val": {
"name": "slow",
"tags": [
"slow"
],
"storage_classes": [
"STANDARD"
]
}
}
]

default Zone 增加新的 placement pool,並且命名為 slow。建置過程中需配置 index-pool, data-pool 與 –data-extra-pool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
$ radosgw-admin zone placement add --rgw-zone default --placement-id slow --index-pool slow.rgw.buckets.index --data-pool slow.rgw.buckets.data --data-extra-pool slow.rgw.buckets.non-ec

{
"id": "67798670-9929-4675-a595-a27b10576b5e",
"name": "default",
"domain_root": "default.rgw.meta:root",
"control_pool": "default.rgw.control",
"gc_pool": "default.rgw.log:gc",
"lc_pool": "default.rgw.log:lc",
"log_pool": "default.rgw.log",
"intent_log_pool": "default.rgw.log:intent",
"usage_log_pool": "default.rgw.log:usage",
"reshard_pool": "default.rgw.log:reshard",
"user_keys_pool": "default.rgw.meta:users.keys",
"user_email_pool": "default.rgw.meta:users.email",
"user_swift_pool": "default.rgw.meta:users.swift",
"user_uid_pool": "default.rgw.meta:users.uid",
"otp_pool": "default.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": "default.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "default.rgw.buckets.data"
}
},
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_type": 0
}
},
{
"key": "slow",
"val": {
"index_pool": "slow.rgw.buckets.index",
"storage_classes": {
"STANDARD": {
"data_pool": "slow.rgw.buckets.data"
}
},
"data_extra_pool": "slow.rgw.buckets.non-ec",
"index_type": 0
}
}
],
"metadata_heap": "",
"realm_id": ""
}

配置完成後,建立相對應的 pools

1
2
3
4
5
6
7
8
9
10
11
$ ceph osd pool create slow.rgw.buckets.index 8 8

pool 'slow.rgw.buckets.index' created

$ ceph osd pool create slow.rgw.buckets.data 8 8

pool 'slow.rgw.buckets.data' created

$ ceph osd pool create slow.rgw.buckets.non-ec 8 8

pool 'slow.rgw.buckets.non-ec' created

更改新建立 pools 的 Crush Rule

1
2
3
4
5
6
7
8
9
10
11
$ ceph osd pool set slow.rgw.buckets.index crush_rule slow

set pool 6 crush_rule to slow

$ ceph osd pool set slow.rgw.buckets.data crush_rule slow

set pool 7 crush_rule to slow

$ ceph osd pool set slow.rgw.buckets.non-ec crush_rule slow

set pool 8 crush_rule to slow

確認新建立 pool 的 Crush Rule

1
2
3
4
5
6
7
$ ceph osd pool ls detail

...
pool 5 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 31 flags hashpspool stripe_width 0 application rgw
pool 6 'slow.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 68 flags hashpspool stripe_width 0
pool 7 'slow.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 70 flags hashpspool stripe_width 0
pool 8 'slow.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 72 flags hashpspool stripe_width 0

crush_rule 為 1 表示已更改為 slow Crush Rule,若讀者實際操作可能值會與範例不同

重新啟動 RGW

1
$ systemctl restart ceph-radosgw@radosgw.gateway

建立測試使用者

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
$ radosgw-admin user create --uid only-slow --display-name "only-slow"

{
"user_id": "only-slow",
"display_name": "only-slow",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
{
"user": "only-slow",
"access_key": "91W6AWWF1J9OIPBVQZG7",
"secret_key": "GsQzXqK9nfXmAvuiXF1hg2VIfrvffppeqaCaoyMJ"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

將使用者的 metadata 匯出並且更改 default_placementplacement_tags 再匯入

1
2
3
4
5
6
7
8
9
10
11
$ radosgw-admin metadata get user:only-slow > only-slow.json
$ vim only-slow.json

...
"op_mask": "read, write, delete",
"default_placement": "slow",
"default_storage_class": "",
"placement_tags": ["slow"],
...

$ radosgw-admin metadata put user:only-slow < only-slow.json

結果

利用 boto3 套件進行測試,測試程式碼如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import boto3

bucket = "test2"
#location = "default:default-placement"
location = "default:slow"

s3 = boto3.client(
's3',
endpoint_url="http://172.17.1.100:7480",
aws_access_key_id="91W6AWWF1J9OIPBVQZG7",
aws_secret_access_key="GsQzXqK9nfXmAvuiXF1hg2VIfrvffppeqaCaoyMJ",
)

s3.create_bucket(
Bucket=bucket,
CreateBucketConfiguration={'LocationConstraint':location},
)

location 為 <region>:<placement-id>,使用 default:slow 可成功建立,而 default:defualt-placement 會得到 AccessDenied 結果。

參考

評論

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×