I apologize if this has been addressed countless times, but I have searched & searched and have not found the answer. I''m rather new to ZFS and have learned a lot about it so far. At least one thing confuses me, however. I''ve noticed that writes to the boot disk in OpenSolaris (i.e. pool rpool) are significantly slower than a write on the same system to an identical drive with a pool on an identically configured slice. If the two drives are configured in every way identically, even inside VirtualBox using virtual drives, the non-boot disk always outperforms the boot disk by up to 4-5x, depending on the computer. My newest computer shows the smallest performance difference (less than 30%), I suppose because it is the fastest system with the fastest drives. I did notice that the apparently better performing pools seem to be taking advantage of the ZFS cache, because you can see the activity light for long after the OS reports it''s finished. I''m testing with files far larger than the disk''s own cache. I see the same consistently reported (not real) performance of ~140MB/s even on USB powered external drives. Quite an impossible number for USB! Is it the overhead of being the boot drive with root''s ZFS pool? Does a ZFS root not use ZFS''s caching? Since this is easily reproduced in VirtualBox (with lower numbers), it can''t be the drive''s own cache that aids this. It would be nice to have better performance on the default rpool when you have only two disks and want a mirror, but the OpenSolaris installer doesn''t give you any options. I still like OpenSolaris anyway! Thanks, and I hope I made sense here. -- This message posted from opensolaris.org
I''ve done some more research, but would still greatly appreciate someone helping me understand this. It seems that writes to only the home directory of the person logged in to the console suffers from degraded performance. If I write to a subdirectory beneath my home, or to any other directory on the system, performance is great. But if I have a session on the console, no matter where else I test from (Gnome or a remote shell), writes ONLY to my home suffer. If I log out of the console and then SSH in from another system, writes to the home directory no longer suffer from degraded performance. This has proven true on every OpenSolaris system I''ve tried--all of which are using ZFS. So what is it about logging into the console that slows write performance to ONLY the top level home directory of the username on the same console? -- This message posted from opensolaris.org
William Bauer wrote:> I''ve done some more research, but would still greatly appreciate someone helping me understand this. > > It seems that writes to only the home directory of the person logged in to the console suffers from degraded performance. If I write to a subdirectory beneath my home, or to any other directory on the system, performance is great. But if I have a session on the console, no matter where else I test from (Gnome or a remote shell), writes ONLY to my home suffer. If I log out of the console and then SSH in from another system, writes to the home directory no longer suffer from degraded performance. > > This has proven true on every OpenSolaris system I''ve tried--all of which are using ZFS. So what is it about logging into the console that slows write performance to ONLY the top level home directory of the username on the same consoleMaybe something to do with autofs? What happens if you move your home dir? Ethan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081026/1976c014/attachment.html>
If that were the case, why would it matter if I was logged into the console, and why would subdirectories of my home exhibit better write performance than the top level home directory? A write to /export/home/username is slower than to /export/home/username/blah, but ONLY if that user is logged into the console. I''m not saying it''s not an odd autofs interaction, because I''ve seen things just as strange. It just seems like the console login is the tripwire for this performance issue, and autofs would come to play at every login and not just a login to the console. -- This message posted from opensolaris.org
On Sun, 26 Oct 2008, William Bauer wrote:> This has proven true on every OpenSolaris system I''ve tried--all of > which are using ZFS. So what is it about logging into the console > that slows write performance to ONLY the top level home directory of > the username on the same console?Recently OpenSolaris included a ''tracker'' daemon and supportive framework which responds to any filesystem change in the user''s home directory and opens the changed file so that it can be indexed. There were probably already Gnome features which depend on this file change detection (e.g. so that an icon may be generated). Perhaps this explains the problem? Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
> If that were the case, why would it matter if I was logged into the console, > and why would subdirectories of my home exhibit better write performance > than the top level home directory? A write to /export/home/username is > slower than to /export/home/username/blah, but ONLY if that user is logged > into the console.This smells of name resolution delays somewhere. Do you have a shell prompt that gets some host name or user name from name services? Is your /home directory owned by a non-existing user or group? Do you accidentally have something enabled in /etc/nsswitch.conf that does not exist (ldap, nis, nis+)? Maybe the first miss gets cached and all other misses get resolved from the cache? Or your nscd is disabled/confused/broken disabled altogether? You have an interesting problem there... Good luck -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Sun Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 45 Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt
> This smells of name resolution delays somewhere. Do > you have > a shell prompt that gets some host name or user name > from > name services? Is your /home directory owned by a > non-existing > user or group? Do you accidentally have something > enabled > in /etc/nsswitch.conf that does not exist (ldap, nis, > nis+)? > Maybe the first miss gets cached and all other misses > get resolved > from the cache? > > Or your nscd is disabled/confused/broken disabled > altogether? >I think you''ve misunderstood something here, perhaps in the way I''ve tried to explain it. I''m having trouble figuring out what that might be. Perhaps your response was to the wrong discussion? -- This message posted from opensolaris.org
This sounds plausible I suppose.... Being unfamiliar with this tracker daemon, I can blindly accept it as a maybe! -- This message posted from opensolaris.org
For clarity, here''s how you can reproduce what I''m asking
about:
This is for local file systems on build 86 and not about NFS or
any remote mounts. You can repeat these 100 times and always get
the same result, whether you reboot between trials or leave the
system running.
1. Log into the console of your OpenSolaris system.
2. Open a terminal in your home directory.
3. Create a file as in the command below and note the write
speed it returns. I know this method is not a true test of write
performance, but I''m interested in relative and not absolute numbers.
This method seems quite consistent in the results it returns.
dd if=/dev/zero of=outfile bs=1024 count=1000000
4. Enter a subdirectory of your home and do the same as above.
5. Log out of the console and perform the same tests via a remote
login using SSH or your favorite method. See that now there is no
difference in performance.
I''ve found the difference between steps 3 and 4 to be typically 25%.
This happens even with the most vanilla new install. On VirtualBox,
the differences can be more like 5x (i.e. 500%!).
Note that if you do step 5 when still logged into the console,
you get the same results as in step 3-4.
That''s it. Maybe not hugely important, but I''m trying to
understand
why this happens.
--
This message posted from opensolaris.org
I cannot recreate this on b101. There is no significant difference between the two on my system. -- richard William Bauer wrote:> For clarity, here''s how you can reproduce what I''m asking about: > > This is for local file systems on build 86 and not about NFS or > any remote mounts. You can repeat these 100 times and always get > the same result, whether you reboot between trials or leave the > system running. > > 1. Log into the console of your OpenSolaris system. > > 2. Open a terminal in your home directory. > > 3. Create a file as in the command below and note the write > speed it returns. I know this method is not a true test of write > performance, but I''m interested in relative and not absolute numbers. > This method seems quite consistent in the results it returns. > > dd if=/dev/zero of=outfile bs=1024 count=1000000 > > 4. Enter a subdirectory of your home and do the same as above. > > 5. Log out of the console and perform the same tests via a remote > login using SSH or your favorite method. See that now there is no > difference in performance. > > I''ve found the difference between steps 3 and 4 to be typically 25%. > This happens even with the most vanilla new install. On VirtualBox, > the differences can be more like 5x (i.e. 500%!). > > Note that if you do step 5 when still logged into the console, > you get the same results as in step 3-4. > > That''s it. Maybe not hugely important, but I''m trying to understand > why this happens. > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
> I cannot recreate this on b101. There is no significant difference between > the two on my system.That''s encouraging...unless no one can reproduce it on 86, then I''m forgetting something. I''ve done this a dozen times on several systems, so maybe ZFS performance has been improved. What numbers did you get (out of curiosity)? -- This message posted from opensolaris.org
Bingo! I just updated a system from 86 to 99 and the "problem" is gone. Even better, it was a VB guest, and the ZFS performance on the guest increased 5x in this test, as I mentioned earlier. Granted, a VB guest may not be the best test and it only applies to top level home directories, but it was consistent and now returns the same higher numbers no matter what. Thank you for testing b101. So how close ARE we to the official November release? Early, mid, or late in the month...! -- This message posted from opensolaris.org
After a zpool upgrade, this simple test''s write speed jumped up yet another 20%. Looks like ZFS is getting better. As one would hope & expect. -- This message posted from opensolaris.org
> > This smells of name resolution delays somewhere.[...]> I think you''ve misunderstood something here, perhaps in the way I''ve tried > to explain it.No, I was just offering a hunch. Writing files into a directory checks access permissions for that directory, and that involves name services. It is unlikely but not impossible that your top-level directory triggers some name service delay. I thought it worth checking.> I''m having trouble figuring out what that might be. Perhaps > your response was to the wrong discussion?:-) Regards -- Volker -- ------------------------------------------------------------------------ Volker A. Brandt Consulting and Support for Sun Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim Email: vab at bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgr??e: 45 Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt
Casper.Dik at Sun.COM
2008-Oct-27 09:07 UTC
[zfs-discuss] ZFS write performance on boot disk
>> > This smells of name resolution delays somewhere. > >[...] >> I think you''ve misunderstood something here, perhaps in the way I''ve tried >> to explain it. > >No, I was just offering a hunch. Writing files into a directory checks >access permissions for that directory, and that involves name services. >It is unlikely but not impossible that your top-level directory triggers >some name service delay. I thought it worth checking.Access permissions checks do not use name service. Checks uses the permission bits in the inode and the uid of the process and the directory. At that point in the kernel, no-one knows your name, only your uid). (Not even, I think, if you''re using CIFS with Windows UUID) Casper