Hi all It seems zfs scrub is taking a big bit out of I/O when running. During a scrub, sync I/O, such as NFS and iSCSI is mostly useless. Attaching an SLOG and some L2ARC helps this, but still, the problem remains in that the scrub is given full priority. Is this problem known to the developers? Will it be addressed? Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On 14/06/2010 22:12, Roy Sigurd Karlsbakk wrote:> Hi all > > It seems zfs scrub is taking a big bit out of I/O when running. During a scrub, sync I/O, such as NFS and iSCSI is mostly useless. Attaching an SLOG and some L2ARC helps this, but still, the problem remains in that the scrub is given full priority. > > Is this problem known to the developers? Will it be addressed? > >http://sparcv9.blogspot.com/2010/06/slower-zfs-scrubsresilver-on-way.html http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6494473 -- Robert Milkowski http://milek.blogspot.com
On Jun 14, 2010, at 2:12 PM, Roy Sigurd Karlsbakk wrote:> Hi all > > It seems zfs scrub is taking a big bit out of I/O when running. During a scrub, sync I/O, such as NFS and iSCSI is mostly useless. Attaching an SLOG and some L2ARC helps this, but still, the problem remains in that the scrub is given full priority.Scrub always runs at the lowest priority. However, priority scheduling only works before the I/Os enter the disk queue. If you are running Solaris 10 or older releases with HDD JBODs, then the default zfs_vdev_max_pending is 35. This means that your slow disk will have 35 I/Os queued to it before priority scheduling makes any difference. Since it is a slow disk, that could mean 250 to 1500 ms before the high priority I/O reaches the disk.> Is this problem known to the developers? Will it be addressed?In later OpenSolaris releases, the zfs_vdev_max_pending defaults to 10 which helps. You can tune it lower as described in the Evil Tuning Guide. Also, as Robert pointed out, CR 6494473 offers a more resource management friendly way to limit scrub traffic (b143). Everyone can buy George a beer for implementing this change :-) Of course, this could mean that on a busy system a scrub that formerly took a week might now take a month. And the fix does not directly address the tuning of the queue depth issue with HDDs. TANSTAAFL. -- richard -- Richard Elling richard at nexenta.com +1-760-896-4422 ZFS and NexentaStor training, Rotterdam, July 13-15, 2010 http://nexenta-rotterdam.eventbrite.com/
Richard Elling wrote:> On Jun 14, 2010, at 2:12 PM, Roy Sigurd Karlsbakk wrote: >> Hi all >> >> It seems zfs scrub is taking a big bit out of I/O when running. During a scrub, sync I/O, such as NFS and iSCSI is mostly useless. Attaching an SLOG and some L2ARC helps this, but still, the problem remains in that the scrub is given full priority. > > Scrub always runs at the lowest priority. However, priority scheduling only > works before the I/Os enter the disk queue. If you are running Solaris 10 or > older releases with HDD JBODs, then the default zfs_vdev_max_pending > is 35. This means that your slow disk will have 35 I/Os queued to it before > priority scheduling makes any difference. Since it is a slow disk, that could > mean 250 to 1500 ms before the high priority I/O reaches the disk. > >> Is this problem known to the developers? Will it be addressed? > > In later OpenSolaris releases, the zfs_vdev_max_pending defaults to 10 > which helps. You can tune it lower as described in the Evil Tuning Guide. > > Also, as Robert pointed out, CR 6494473 offers a more resource management > friendly way to limit scrub traffic (b143). Everyone can buy George a beer for > implementing this change :-) >I''ll glad accept any beer donations and others on the ZFS team are happy to help consume it. :-) I look forward to hearing people''s experience with the new changes. - George> Of course, this could mean that on a busy system a scrub that formerly took > a week might now take a month. And the fix does not directly address the > tuning of the queue depth issue with HDDs. TANSTAAFL. > -- richard >