Hi, I have a setup with thousands of filesystems, each containing several snapshots. For a good percentage of these filesystems I want to create a snapshots once every hour, for others once every 2 hours and so forth. I built some tools to do this, no problem so far. While examining disk load on the system, I found out that load jumps up whenever the snapshot creation process is running. Delving a bit deeper it seems that every snapshot start a new txg. This seems to be quite costly, about 100-150 IOPs. This takes about one second, so I can create one snapshot per second. Creating all necessary snapshot for one hour takes about 45 minutes. During this time the disks are at 70% utilization and txgs are back-to-back. So I need to optimize this. Looking at the code it seems that recursive snapshots are being collected into a single txg. So my aim is to collect all necessary snapshots in a single txg, too. Using libzfs or ioctl I haven''t found any way to do this. I cannot just use recursive snapshots, because not all filesystems need to be snapshotted. Same with snapshot deletion. My idea is to write a small kernel module that roughly duplicates the code of zfs_ioc_snapshot, but instead of being fully recursive gets passed a list of filesystems to snapshot. My questions: - is there any easier way to bring down disk load and accelerate snapshot creation? - are there any arguments, why my approach isn''t feasible? Thanks for any hits. -Arne
Arne Jansen wrote:> Hi, > > I have a setup with thousands of filesystems, each containing several > snapshots. For a good percentage of these filesystems I want to create > a snapshots once every hour, for others once every 2 hours and so forth. > I built some tools to do this, no problem so far. > > While examining disk load on the system, I found out that load jumps > up whenever the snapshot creation process is running. Delving a bit > deeper it seems that every snapshot start a new txg. This seems to be > quite costly, about 100-150 IOPs. This takes about one second, so I can > create one snapshot per second. Creating all necessary snapshot for one > hour takes about 45 minutes. During this time the disks are at 70% > utilization and txgs are back-to-back. So I need to optimize this. > > Looking at the code it seems that recursive snapshots are being > collected into a single txg. So my aim is to collect all necessary > snapshots in a single txg, too. Using libzfs or ioctl I haven''t found > any way to do this. I cannot just use recursive snapshots, because > not all filesystems need to be snapshotted. Same with snapshot deletion. > > My idea is to write a small kernel module that roughly duplicates the > code of zfs_ioc_snapshot, but instead of being fully recursive gets > passed a list of filesystems to snapshot. > > My questions: > - is there any easier way to bring down disk load and accelerate > snapshot creation? > - are there any arguments, why my approach isn''t feasible? >Just as a followup, the module works smoothly for creations. If I take 100 snapshots at a time, the gain is roughly a factor of 100 :) Because there are no recursive deletions in kernel space (recursive deletions are handled by libzfs), I haven''t found an easy way to do likewise for delete. Maybe someday when I get a better grip on the source... --Arne
Very interesting. This could be useful for a number of us. Would you be willing to share your work? -- This message posted from opensolaris.org
thomas wrote:> Very interesting. This could be useful for a number of us. Would you be willing to share your work?No problem. I''ll contact you off-list.
Hello there, I think you should share it with the list, if you can, seems like an interesting work. ZFS has some issues with snapshots and spa_sync performance for snapshots deletion. Thanks Leal [ http://www.eall.com.br/blog ] -- This message posted from opensolaris.org
Marcelo Leal wrote:> Hello there, > I think you should share it with the list, if you can, seems like an interesting work. ZFS has some issues with snapshots and spa_sync performance for snapshots deletion.I''m a bit reluctant to post it to the list where it can still be found years from now. Because the module is not compiled directly into ZFS but is a separate module that makes heavy use of internal structures of ZFS, it is designed for a specific version of ZFS (Solaris U8). It might still load without problems for years, but already in the next Solaris version it might wreak havoc because of a changed kernel structure. A much better way would be to have a similar operation integrated into the official source tree. I could try to build a patch if it has a chance of getting accepted. Until then, I have no problem with sharing it off-list. --Arne> > Thanks > > Leal > [ http://www.eall.com.br/blog ]