On Tue, 2015-12-08 at 16:31 -0600, Dustin Wenz wrote:> I suspect this is a zfs bug that is triggered by the access patterns > in the periodic scripts. There is significant load on the system when > the scheduled processes start, because all jails execute the same > scripts at the same time. > > I've been able to alleviate this problem by disabling the security > scans within the jails, but leave it enabled on the root host.To avoid the problem of jails all starting things at the same time, use the cron(8) flags -j and -J to set a 'jitter' which will cause cron to sleep for a random period of specified duration (60 sec max). Cron flags can be set using the rc.conf variable 'cron_flags'.
On 09/12/15 01:04, Michael B. Eichorn wrote:> On Tue, 2015-12-08 at 16:31 -0600, Dustin Wenz wrote: >> I suspect this is a zfs bug that is triggered by the access patterns >> in the periodic scripts. There is significant load on the system when >> the scheduled processes start, because all jails execute the same >> scripts at the same time. >> >> I've been able to alleviate this problem by disabling the security >> scans within the jails, but leave it enabled on the root host. > > To avoid the problem of jails all starting things at the same time, use > the cron(8) flags -j and -J to set a 'jitter' which will cause cron to > sleep for a random period of specified duration (60 sec max). Cron > flags can be set using the rc.conf variable 'cron_flags'.While jitter would reduce the resource contention a thundering herd of cronjobs shouldn't cause the kernel to divide by zero. Spreading the load by introducing jitter to cronjobs might hide the problem, but it still needs further analysis. @Dustin Wenz: Can you reproduce the problem and file a PR to track this?
Michael B. Eichorn wrote:> On Tue, 2015-12-08 at 16:31 -0600, Dustin Wenz wrote: > >> I suspect this is a zfs bug that is triggered by the access patterns >> in the periodic scripts. There is significant load on the system when >> the scheduled processes start, because all jails execute the same >> scripts at the same time. >> >> I've been able to alleviate this problem by disabling the security >> scans within the jails, but leave it enabled on the root host. >> > > To avoid the problem of jails all starting things at the same time, use > the cron(8) flags -j and -J to set a 'jitter' which will cause cron to > sleep for a random period of specified duration (60 sec max). Cron > flags can be set using the rc.conf variable 'cron_flags'. > _______________________________________________ >No that will just hide it (if successful at all) and it won't work in all cases. ... i386 is even worse for similar (not the same) instability triggered by the same scripts ... because zfs should not be used with the stock i386 kernel (which means if you're using it the whole patching process with freebsd-update won't work or will 'undo' your kernel config.) Personally I think zfs should be optional only for 'advanced' users and come with a whole host of warnings about what it is not suitable for.... however, it seems to be treated as a magic bullet for data corruption issues yet all I have seen is an ever growing list of where it causes problems.. when did UFS become an unreliable FS that is susceptible to chronic data corruption? -- Michelle Sullivan http://www.mhix.org/