Hi there.
% zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 5.41T 670G 4.75T - 13% 12% 1.00x ONLINE -
% uname -a
FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r322984 [...] amd64
I'm running multiple jails on ZFS, using ezjail to manage them,
including a websever and a mailserver. The mailserver is using a MySQL
database, otherwise depending on dovecot and postfix. Very low volume,
just a few polls / logins per minute.
I am experiencing very high loads as per gstat:
dT: 1.021s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
4 181 0 0 0.0 169 873 20.1 98.2| ada0
2 111 0 0 0.0 100 540 7.3 90.6| ada1
0 88 0 0 0.0 76 458 1.4 43.3| ada2
0 0 0 0 0.0 0 0 0.0 0.0| ada0p1
3 150 0 0 0.0 150 603 20.2 95.1| ada0p2
1 31 0 0 0.0 20 270 19.2 117.0| ada0p3
0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot0
0 0 0 0 0.0 0 0 0.0 0.0| ada1p1
1 85 0 0 0.0 85 341 8.4 68.9| ada1p2
1 25 0 0 0.0 15 200 0.9 75.0| ada1p3
0 0 0 0 0.0 0 0 0.0 0.0| ada2p1
0 62 0 0 0.0 62 251 1.6 9.9| ada2p2
0 26 0 0 0.0 15 208 0.5 42.0| ada2p3
0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot1
0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot2
These loads lead to the system suffering from very much delayed
responses to even the basic task of echoing characters entered on the
console, consequently rendering the services offered unusable to the
users because of the delays.
Restarting the jails (or even the whole machine at that) ends me up at
exactly the same situation.
I do have lab machines for running load scenarios, so if anyone feels
compelled to lend a hand, please do.
MfG CoCo
Jails probably aren't the source of your problem. You need to find out what process or processes are responsible for all this activity. Since the write bandwidth is fairly low, you might have a process that's sync(2)ing or fsync(2)ing. too often. "gstat -o" will show if that's the case. You can also try running "top -mio" to see which processes are doing the most I/O. -Alan On Sun, Aug 12, 2018 at 12:50 PM, Marco Steinbach < coco at executive-computing.de> wrote:> Hi there. > > % zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > zroot 5.41T 670G 4.75T - 13% 12% 1.00x ONLINE - > > % uname -a > FreeBSD XXX 11.1-STABLE FreeBSD 11.1-STABLE #0 r322984 [...] amd64 > > > I'm running multiple jails on ZFS, using ezjail to manage them, > including a websever and a mailserver. The mailserver is using a MySQL > database, otherwise depending on dovecot and postfix. Very low volume, > just a few polls / logins per minute. > > I am experiencing very high loads as per gstat: > > dT: 1.021s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 4 181 0 0 0.0 169 873 20.1 98.2| ada0 > 2 111 0 0 0.0 100 540 7.3 90.6| ada1 > 0 88 0 0 0.0 76 458 1.4 43.3| ada2 > 0 0 0 0 0.0 0 0 0.0 0.0| ada0p1 > 3 150 0 0 0.0 150 603 20.2 95.1| ada0p2 > 1 31 0 0 0.0 20 270 19.2 117.0| ada0p3 > 0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot0 > 0 0 0 0 0.0 0 0 0.0 0.0| ada1p1 > 1 85 0 0 0.0 85 341 8.4 68.9| ada1p2 > 1 25 0 0 0.0 15 200 0.9 75.0| ada1p3 > 0 0 0 0 0.0 0 0 0.0 0.0| ada2p1 > 0 62 0 0 0.0 62 251 1.6 9.9| ada2p2 > 0 26 0 0 0.0 15 208 0.5 42.0| ada2p3 > 0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot1 > 0 0 0 0 0.0 0 0 0.0 0.0| gpt/gptboot2 > > > These loads lead to the system suffering from very much delayed > responses to even the basic task of echoing characters entered on the > console, consequently rendering the services offered unusable to the > users because of the delays. > > Restarting the jails (or even the whole machine at that) ends me up at > exactly the same situation. > > I do have lab machines for running load scenarios, so if anyone feels > compelled to lend a hand, please do. > > MfG CoCo > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >
On 8/12/2018 2:50 PM, Marco Steinbach wrote:> > These loads lead to the system suffering from very much delayed > responses to even the basic task of echoing characters entered on the > console, consequently rendering the services offered unusable to the > users because of the delays.Do you have a LOT of files and or metadata ? Have a look at the cache stats to see if you are perhaps grinding away on big directory lookups? Install sysutils/zfs-stats and post zfs-stats -a Also, does top -mio -I shed any light as to whats taking up the disk io ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 x203 Sentex Communications, mike at sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada
On Sun, 12 Aug 2018 20:50:47 +0200 Marco Steinbach <coco at executive-computing.de> wrote:> Hi there. >[...] Thanks for all the answers and suggestions -- one of the MySQL databases contained a defective table, repairing it brought the storage load down to normal levels. I've had defective tables on UFS in the past, and can't remember them bogging down the system as they do, when running on ZFS. I am aware of the blocksize issue, though. Again, thanks for all the suggestions. MfG CoCo