I have test environment to determine what would be best settings. I have
been told that enabling zlib compression would be good to save iops on
storage. But doing the test now, I get worse results.
[@test2 ~]# pr -m -t mail04-mdbox-vdb-append-64kb-6.log
mail04-mdbox-vdb-append-64kb-8.log |less
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
1 1 6 1/ 1 1 1 6 1/ 1
0 0 9 1/ 1 0 0 8 1/ 1
0 0 7 1/ 1 0 0 8 1/ 1
0 0 10 1/ 1 0 0 9 1/ 1
0 0 7 1/ 1 0 0 6 1/ 1
0 0 8 1/ 1 0 0 8 1/ 1
0 0 9 1/ 1 0 0 8 1/ 1
0 0 10 1/ 1 0 0 8 1/ 1
0 0 9 1/ 1 0 0 7 1/ 1
0 0 11 1/ 1 0 0 7 1/ 1
29 3 1093 ms/cmd avg 43 2 1232 ms/cmd avg
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
0 0 10 1/ 1 0 0 9 1/ 1
0 0 9 1/ 1 0 0 7 1/ 1
0 0 10 1/ 1 0 0 7 1/ 1
0 0 9 1/ 1 0 0 6 1/ 1
0 0 11 1/ 1 0 0 8 1/ 1
0 0 9 1/ 1 0 0 7 1/ 1
0 0 8 1/ 1 0 0 6 1/ 1
0 0 11 1/ 1 0 0 7 1/ 1
0 0 9 1/ 1 0 0 5 1/ 1
0 0 9 1/ 1 0 0 9 1/ 1
0 0 1063 ms/cmd avg 0 0 1413 ms/cmd avg
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
0 0 8 1/ 1 0 0 8 1/ 1
0 0 10 1/ 1 0 0 8 1/ 1
0 0 7 1/ 1 0 0 8 1/ 1
0 0 10 1/ 1 0 0 6 1/ 1
0 0 10 1/ 1 0 0 7 1/ 1
0 0 11 1/ 1 0 0 8 1/ 1
0 0 7 1/ 1 0 0 8 1/ 1
0 0 11 1/ 1 0 0 6 1/ 1
0 0 8 1/ 1 0 0 7 1/ 1
0 0 10 1/ 1 0 0 8 1/ 1
0 0 1077 ms/cmd avg 0 0 1382 ms/cmd avg
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
0 0 7 1/ 1 0 0 6 1/ 1
0 0 9 1/ 1 0 0 4 1/ 1
0 0 6 1/ 1 0 0 7 1/ 1
0 0 9 1/ 1 0 0 7 1/ 1
0 0 8 1/ 1 0 0 7 1/ 1
0 0 10 1/ 1 0 0 5 1/ 1
0 0 9 1/ 1 0 0 8 1/ 1
0 0 7 1/ 1 0 0 6 1/ 1
0 0 9 1/ 1 0 0 6 1/ 1
0 0 7 1/ 1 0 0 8 1/ 1
0 0 1246 ms/cmd avg 0 0 1524 ms/cmd avg
...
...
...
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
0 0 8 1/ 1 0 0 5 1/ 1
0 0 6 1/ 1 0 0 4 1/ 1
0 0 6 1/ 1 0 0 4 1/ 1
0 0 7 1/ 1 0 0 4 1/ 1
0 0 5 1/ 1 0 0 5 1/ 1
0 0 5 1/ 1 0 0 4 1/ 1
0 0 6 1/ 1 0 0 4 1/ 1
0 0 7 1/ 1 0 0 5 1/ 1
0 0 6 1/ 1 0 0 4 1/ 1
0 0 7 1/ 1 0 0 5 1/ 1
0 0 1596 ms/cmd avg 0 0 2308 ms/cmd avg
Logi Sele Appe Logi Sele Appe
100% 100% 100% 100% 100% 100%
0 0 7 1/ 1 0 0 4 1/ 1
0 0 1583 ms/cmd avg 0 0 4 1/ 1
0 0 2331 ms/cmd avg
Totals:
Logi Sele Appe Totals:
100% 100% 100% Logi Sele Appe
1 1 1978 100% 100% 100%
1 1 1145
mdbox format is a cross between mbox and sdbox. The idea is that it keeps up to mdbox_rotate_size sized mbox files which contain mails. Using zlib here will not help, because zlib is not applied to the full mdbox file, but individual mails within. Also not sure why you think that adding compression would make things faster? =) I am not fully sure how cephfs works but you might get better results with sdbox format. Aki> On 14 January 2019 at 17:30 Marc Roos <M.Roos at f1-outsourcing.eu> wrote: > > > > I have test environment to determine what would be best settings. I have > been told that enabling zlib compression would be good to save iops on > storage. But doing the test now, I get worse results. > > [@test2 ~]# pr -m -t mail04-mdbox-vdb-append-64kb-6.log > mail04-mdbox-vdb-append-64kb-8.log |less > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 1 1 6 1/ 1 1 1 6 1/ 1 > 0 0 9 1/ 1 0 0 8 1/ 1 > 0 0 7 1/ 1 0 0 8 1/ 1 > 0 0 10 1/ 1 0 0 9 1/ 1 > 0 0 7 1/ 1 0 0 6 1/ 1 > 0 0 8 1/ 1 0 0 8 1/ 1 > 0 0 9 1/ 1 0 0 8 1/ 1 > 0 0 10 1/ 1 0 0 8 1/ 1 > 0 0 9 1/ 1 0 0 7 1/ 1 > 0 0 11 1/ 1 0 0 7 1/ 1 > 29 3 1093 ms/cmd avg 43 2 1232 ms/cmd avg > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 0 0 10 1/ 1 0 0 9 1/ 1 > 0 0 9 1/ 1 0 0 7 1/ 1 > 0 0 10 1/ 1 0 0 7 1/ 1 > 0 0 9 1/ 1 0 0 6 1/ 1 > 0 0 11 1/ 1 0 0 8 1/ 1 > 0 0 9 1/ 1 0 0 7 1/ 1 > 0 0 8 1/ 1 0 0 6 1/ 1 > 0 0 11 1/ 1 0 0 7 1/ 1 > 0 0 9 1/ 1 0 0 5 1/ 1 > 0 0 9 1/ 1 0 0 9 1/ 1 > 0 0 1063 ms/cmd avg 0 0 1413 ms/cmd avg > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 0 0 8 1/ 1 0 0 8 1/ 1 > 0 0 10 1/ 1 0 0 8 1/ 1 > 0 0 7 1/ 1 0 0 8 1/ 1 > 0 0 10 1/ 1 0 0 6 1/ 1 > 0 0 10 1/ 1 0 0 7 1/ 1 > 0 0 11 1/ 1 0 0 8 1/ 1 > 0 0 7 1/ 1 0 0 8 1/ 1 > 0 0 11 1/ 1 0 0 6 1/ 1 > 0 0 8 1/ 1 0 0 7 1/ 1 > 0 0 10 1/ 1 0 0 8 1/ 1 > 0 0 1077 ms/cmd avg 0 0 1382 ms/cmd avg > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 0 0 7 1/ 1 0 0 6 1/ 1 > 0 0 9 1/ 1 0 0 4 1/ 1 > 0 0 6 1/ 1 0 0 7 1/ 1 > 0 0 9 1/ 1 0 0 7 1/ 1 > 0 0 8 1/ 1 0 0 7 1/ 1 > 0 0 10 1/ 1 0 0 5 1/ 1 > 0 0 9 1/ 1 0 0 8 1/ 1 > 0 0 7 1/ 1 0 0 6 1/ 1 > 0 0 9 1/ 1 0 0 6 1/ 1 > 0 0 7 1/ 1 0 0 8 1/ 1 > 0 0 1246 ms/cmd avg 0 0 1524 ms/cmd avg > ... > ... > ... > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 0 0 8 1/ 1 0 0 5 1/ 1 > 0 0 6 1/ 1 0 0 4 1/ 1 > 0 0 6 1/ 1 0 0 4 1/ 1 > 0 0 7 1/ 1 0 0 4 1/ 1 > 0 0 5 1/ 1 0 0 5 1/ 1 > 0 0 5 1/ 1 0 0 4 1/ 1 > 0 0 6 1/ 1 0 0 4 1/ 1 > 0 0 7 1/ 1 0 0 5 1/ 1 > 0 0 6 1/ 1 0 0 4 1/ 1 > 0 0 7 1/ 1 0 0 5 1/ 1 > 0 0 1596 ms/cmd avg 0 0 2308 ms/cmd avg > Logi Sele Appe Logi Sele Appe > 100% 100% 100% 100% 100% 100% > 0 0 7 1/ 1 0 0 4 1/ 1 > 0 0 1583 ms/cmd avg 0 0 4 1/ 1 > 0 0 2331 ms/cmd avg > Totals: > Logi Sele Appe Totals: > 100% 100% 100% Logi Sele Appe > 1 1 1978 100% 100% 100% > 1 1 1145 > > > > > > >
>mdbox format is a cross between mbox and sdbox.> >The idea is that it keeps up to mdbox_rotate_size sized mbox files >which contain mails. Using zlib here will not help, because zlib is >not applied to the full mdbox file, but individual mails within. > >Also not sure why you think that adding compression would make things >faster? =) atop shows a saturated lvm/disk, some were telling me that enabling compression would alleviate on the disk io's in exchange of higher cpu utilization. >I am not fully sure how cephfs works but you might get better results >with sdbox format. > I think the one message / one file, is bad for cephfs storage, as I understand ceph creates quite a bit of overhead there. https://www.mail-archive.com/ceph-users at lists.ceph.com/msg48054.html >> On 14 January 2019 at 17:30 Marc Roos <M.Roos at f1-outsourcing.eu> wrote: >> >> >> >> I have test environment to determine what would be best settings. I have >> been told that enabling zlib compression would be good to save iops on >> storage. But doing the test now, I get worse results. >> >> [@test2 ~]# pr -m -t mail04-mdbox-vdb-append-64kb-6.log >> mail04-mdbox-vdb-append-64kb-8.log |less >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 1 1 6 1/ 1 1 1 6 1/ 1 >> 0 0 9 1/ 1 0 0 8 1/ 1 >> 0 0 7 1/ 1 0 0 8 1/ 1 >> 0 0 10 1/ 1 0 0 9 1/ 1 >> 0 0 7 1/ 1 0 0 6 1/ 1 >> 0 0 8 1/ 1 0 0 8 1/ 1 >> 0 0 9 1/ 1 0 0 8 1/ 1 >> 0 0 10 1/ 1 0 0 8 1/ 1 >> 0 0 9 1/ 1 0 0 7 1/ 1 >> 0 0 11 1/ 1 0 0 7 1/ 1 >> 29 3 1093 ms/cmd avg 43 2 1232 ms/cmd avg >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 0 0 10 1/ 1 0 0 9 1/ 1 >> 0 0 9 1/ 1 0 0 7 1/ 1 >> 0 0 10 1/ 1 0 0 7 1/ 1 >> 0 0 9 1/ 1 0 0 6 1/ 1 >> 0 0 11 1/ 1 0 0 8 1/ 1 >> 0 0 9 1/ 1 0 0 7 1/ 1 >> 0 0 8 1/ 1 0 0 6 1/ 1 >> 0 0 11 1/ 1 0 0 7 1/ 1 >> 0 0 9 1/ 1 0 0 5 1/ 1 >> 0 0 9 1/ 1 0 0 9 1/ 1 >> 0 0 1063 ms/cmd avg 0 0 1413 ms/cmd avg >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 0 0 8 1/ 1 0 0 8 1/ 1 >> 0 0 10 1/ 1 0 0 8 1/ 1 >> 0 0 7 1/ 1 0 0 8 1/ 1 >> 0 0 10 1/ 1 0 0 6 1/ 1 >> 0 0 10 1/ 1 0 0 7 1/ 1 >> 0 0 11 1/ 1 0 0 8 1/ 1 >> 0 0 7 1/ 1 0 0 8 1/ 1 >> 0 0 11 1/ 1 0 0 6 1/ 1 >> 0 0 8 1/ 1 0 0 7 1/ 1 >> 0 0 10 1/ 1 0 0 8 1/ 1 >> 0 0 1077 ms/cmd avg 0 0 1382 ms/cmd avg >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 0 0 7 1/ 1 0 0 6 1/ 1 >> 0 0 9 1/ 1 0 0 4 1/ 1 >> 0 0 6 1/ 1 0 0 7 1/ 1 >> 0 0 9 1/ 1 0 0 7 1/ 1 >> 0 0 8 1/ 1 0 0 7 1/ 1 >> 0 0 10 1/ 1 0 0 5 1/ 1 >> 0 0 9 1/ 1 0 0 8 1/ 1 >> 0 0 7 1/ 1 0 0 6 1/ 1 >> 0 0 9 1/ 1 0 0 6 1/ 1 >> 0 0 7 1/ 1 0 0 8 1/ 1 >> 0 0 1246 ms/cmd avg 0 0 1524 ms/cmd avg >> ... >> ... >> ... >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 0 0 8 1/ 1 0 0 5 1/ 1 >> 0 0 6 1/ 1 0 0 4 1/ 1 >> 0 0 6 1/ 1 0 0 4 1/ 1 >> 0 0 7 1/ 1 0 0 4 1/ 1 >> 0 0 5 1/ 1 0 0 5 1/ 1 >> 0 0 5 1/ 1 0 0 4 1/ 1 >> 0 0 6 1/ 1 0 0 4 1/ 1 >> 0 0 7 1/ 1 0 0 5 1/ 1 >> 0 0 6 1/ 1 0 0 4 1/ 1 >> 0 0 7 1/ 1 0 0 5 1/ 1 >> 0 0 1596 ms/cmd avg 0 0 2308 ms/cmd avg >> Logi Sele Appe Logi Sele Appe >> 100% 100% 100% 100% 100% 100% >> 0 0 7 1/ 1 0 0 4 1/ 1 >> 0 0 1583 ms/cmd avg 0 0 4 1/ 1 >> 0 0 2331 ms/cmd avg >> Totals: >> Logi Sele Appe Totals: >> 100% 100% 100% Logi Sele Appe >> 1 1 1978 100% 100% 100% >> 1 1 1145 >> >> >> >> >> >> >> > >