zfs discuss - Apr 2007 - zfs performance on fuse (Linux) compared to other fs

Hi,

ok I know zfs-fuse is still incomplete and performance has not been considered,
but still, before I''m going to use it for my /home I wanted a rough
estimate.

Another benchmark already asserted that zfs by itself, on Solaris, is a very
fast beast
(http://cmynhier.blogspot.com/2006/05/zfs-benchmarking.html), faster than ext3
on
Linux and also faster than UFS.  Interestingly UFS seems to be faster than ext3,
I didn''t know that.

So, as a "feeling" benchmark I copied my home over to zfs-fuse, fired
up Eclipse, and did
a build of a large Java project.  It felt /pretty/ slow, man.

Next I ran a find(1) for a particular file, twice.  Obviously the second run
would be
from cache, so I expected fast response.  Well, not quite so fast with fuse:

1.
real    2m6.785s
user    0m0.968s
sys     0m2.308s

2.
real    0m46.920s
user    0m0.844s
sys     0m1.916s

So the second run is not actually fast.  Still, I observed that the disk was
quiet during
the second run, and top(1) showed the CPU maxed out by the zfs-fuse daemon.

For comparison here is the same find(1) on another copy of my /home on ext3:

1. 
real    1m14.494s
user    0m1.488s
sys     0m4.928s

2.
real    0m1.124s
user    0m0.556s
sys     0m0.496s

So, at this point in time that seems pretty discouraging for an everyday user,
on Linux.
Hmmm, maybe I should try switch to OpenSolaris...  but then how do I satisfy the
need to occasionally run Windows in VMWare...

Maybe, since this was on a scratch USB disk, I download a FreeBSD snapshot and
see
how that compares?  Depends on how much time I get and what the wheather is
like :)

I''d be interested in others'' thoughts, esp. why the daemon
eats so much CPU when
it only reads through cache.  For reference, my /home is about 18GB with about
300.000 files.

--
Regards,
Georg.
 
 
This message posted from opensolaris.org

I''m potentially stepping in areas I don''t quite know enough
about, but
others can jump in if I speak any mistruths....   :)

More inline...

Georg-W. Koltermann wrote:
> Hi,
>
> ok I know zfs-fuse is still incomplete and performance has not been
considered,
> but still, before I''m going to use it for my /home I wanted a
rough estimate.
>
> Another benchmark already asserted that zfs by itself, on Solaris, is a
very fast beast
> (http://cmynhier.blogspot.com/2006/05/zfs-benchmarking.html), faster than
ext3 on
> Linux and also faster than UFS.  Interestingly UFS seems to be faster than
ext3,
> I didn''t know that.
>
> So, as a "feeling" benchmark I copied my home over to zfs-fuse,
fired up Eclipse, and did
> a build of a large Java project.  It felt /pretty/ slow, man.
>
> Next I ran a find(1) for a particular file, twice.  Obviously the second
run would be
> from cache, so I expected fast response.  Well, not quite so fast with
fuse:
>
> 1.
> real    2m6.785s
> user    0m0.968s
> sys     0m2.308s
>
> 2.
> real    0m46.920s
> user    0m0.844s
> sys     0m1.916s
>
> So the second run is not actually fast.  Still, I observed that the disk
was quiet during
> the second run, and top(1) showed the CPU maxed out by the zfs-fuse daemon.
>
> For comparison here is the same find(1) on another copy of my /home on
ext3:
>
> 1. 
> real    1m14.494s
> user    0m1.488s
> sys     0m4.928s
>
> 2.
> real    0m1.124s
> user    0m0.556s
> sys     0m0.496s
>
> So, at this point in time that seems pretty discouraging for an everyday
user, on Linux.
> Hmmm, maybe I should try switch to OpenSolaris...  but then how do I
satisfy the
> need to occasionally run Windows in VMWare...
>
> Maybe, since this was on a scratch USB disk, I download a FreeBSD snapshot
and see
> how that compares?  Depends on how much time I get and what the wheather is
> like :)
>   
I don''t think FreeBSD will solve the Windows problem any better than 
Solaris will, but in a not to distant build, we should see the putback 
of the Xen work, which I believe would allow you to run Windows on Xen 
with appropriate hardware.  Check out that project for details.
> I''d be interested in others'' thoughts, esp. why the
daemon eats so much CPU when
> it only reads through cache.  For reference, my /home is about 18GB with
about 300.000 files.
>   
Does ZFS on FUSE have the ARC implemented?  I looked at the page and 
blogs, but am not sure.  On OpenSolaris at least, if there''s memory 
pressure it gives up caching to the OS.  If ZFS on FUSE hasn''t 
implemented (or has somehow disabled) the ARC, you''d be running 
cacheless. I suspect it''s implemented, but don''t know if it
has any
built in limits.

The other thing is that the ZFS ARC is, as I understand it, scan 
resistant so you''d expect a find to take about the same amount of time 
each time.  While this isn''t good for frequent finds, it''s
very good to
not kick files out of the cache your applications are using.  I''d 
recommend reading the comments in the code (arc.c I think) for info on this.

With the Eclipse project, I''d expect somewhat better results, but it 
really depends on the workload.  Perhaps some emperical data comparing 
the two would be useful.

Hope that helps,

- Matt

> So, at this point in time that seems pretty discouraging for an everyday
user, on Linux.
nobody told, that zfs-fuse is ready for an everyday user at it`s current state !
;)

although it runs pretty stable for now, there still remain major issues and
especially, it`s not yet being optimized for performance.
you should give it some more time, perhaps performance will get better one day
if there is some work put into this.
 
 
This message posted from opensolaris.org

> > So, at this point in time that seems pretty
> discouraging for an everyday user, on Linux.
> 
> nobody told, that zfs-fuse is ready for an everyday
> user at it`s current state ! ;)
That''s what I found out, wanted to share and get other''s
opinion on.

I did not complain.  I thought it might work, it might not, so I tried.

BTW last night I tried ZFS on FreeBSD 7.  I got a panic when trying to make it
import my existing pool at first.  Then I tried again another way and did get it
to
recognize it. My simple, non-representative performance measurement was even
slower than zfs-fuse (something like 4-5 minutes for the find, no apparent
caching
effect), and I had many USB read errors along the way as well.  It looks like
FBSD 7 with ZFS is even more immature than zfs-fuse at this time. 
That''s ok,
it is a CVS snapshot of FreeBSD "CURRENT" after all.

I have been a long time happy FreeBSD user before I switched to Ubuntu, I am
definitely not complaining about this.  Sometimes things work, sometimes not.

I guess I will still checkout OpenSolaris once my download finishes. 
I''m in
experimentation mood :)
> although it runs pretty stable for now, there still
> remain major issues and especially, it`s not yet
> being optimized for performance.
> you should give it some more time, perhaps
> performance will get better one day if there is some
> work put into this.
I think that will happen.  There''s much talking about it already.

My perspective currently is to find something for my own everyday use. 
That''s
for today, not forever. I may change again later. As I said,
I am somewhat in experimentation mood.

Thanks for your comments.
 
 
This message posted from opensolaris.org

On Mon, Apr 23, 2007 at 11:42:41PM -0700, Georg-W. Koltermann
wrote:
> > > So, at this point in time that seems pretty
> > discouraging for an everyday user, on Linux.
> > 
> > nobody told, that zfs-fuse is ready for an everyday
> > user at it`s current state ! ;)
> 
> That''s what I found out, wanted to share and get other''s
opinion on.
> 
> I did not complain.  I thought it might work, it might not, so I tried.
> 
> BTW last night I tried ZFS on FreeBSD 7.  I got a panic when trying to make
it
> import my existing pool at first.  [...]
Can I see the panic message and backtrace?
> [...] Then I tried again another way and did get it to 
> recognize it. My simple, non-representative performance measurement was
even
> slower than zfs-fuse (something like 4-5 minutes for the find, no apparent
caching
> effect), and I had many USB read errors along the way as well.  It looks
like
> FBSD 7 with ZFS is even more immature than zfs-fuse at this time. 
That''s ok,
> it is a CVS snapshot of FreeBSD "CURRENT" after all.
First of all CURRENT snapshot comes with a kernel, which contains some
heavely debugging options turned on by default. Turning off WITNESS
should make the ZFS works few times faster. find was the only test you
tried? Currently I''m using ported DNLC namecache, but I''ve a
working
code already that uses FreeBSD''s namecache and it performs much better
for such a test.

There were few nits after the import, which are all (or most of them)
fixed at this point, but I''ve a huge number of reports from the users
that ZFS works very stable on FreeBSD. If you could reproduce the panic
and send me info I''d be grateful.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070424/04924850/attachment.bin>

On Mon, Apr 23, 2007 at 03:08:21PM -0700, roland wrote:
> > So, at this point in time that seems pretty discouraging for an
everyday user, on Linux.
> 
> nobody told, that zfs-fuse is ready for an everyday user at it`s current
state ! ;)
Also keep in mind that FUSE has the disadvantage of data needing to
pass the userland/kernel barrier typically at least twice, so there
will be some slowdown there most likely.

That doesn''t mean ZFS on FUSE can''t get faster, just that I
doubt it
will ever be as fast as "native" ZFS on Solaris and FreeBSD.

I think someone needs to port ZFS to NetBSD, they are in dired need of
a better filesystem.  ;)

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

Brian Hechinger wrote:
> I think someone needs to port ZFS to NetBSD, they are in dired need of
> a better filesystem.  ;)
Already being done:

http://code.google.com/soc/netbsd/about.html

Isn''t the Google SoC great? :)

On 4/24/07, Brian Hechinger <wonko at 4amlunch.net>
wrote:
> Also keep in mind that FUSE has the disadvantage of data needing to
> pass the userland/kernel barrier typically at least twice, so there
> will be some slowdown there most likely.
I think it depends on how the Linux kernel handles the interface...

Block size is always (at least for the ones that I know) a multiple of
page size. So instead of copying data in and out of the kernel, the
kernel can map the memory into the user-space FS daemon.

I am not sure how FUSE handles it as I couldn''t find any tech. doc on
FUSE''s homepage, but there are a lot of papers written in the past 20
years on efficient micro-kernel design...

Here''s a Minix paper with an interesting section: Disk I/O Test Results
http://www.minix3.org/doc/reliable-os.pdf

Rayson




>
> That doesn''t mean ZFS on FUSE can''t get faster, just that
I doubt it
> will ever be as fast as "native" ZFS on Solaris and FreeBSD.
>
> I think someone needs to port ZFS to NetBSD, they are in dired need of
> a better filesystem.  ;)
>
> -brian
> --
> "Perl can be fast and elegant as much as J2EE can be fast and elegant.
> In the hands of a skilled artisan, it can and does happen; it''s
just
> that most of the shit out there is built by people who''d be better
> suited to making sure that my burger is cooked thoroughly."  --
Jonathan Patschke
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

NetBSD could reuse Pawel''s work on FreeBSD but it sits on top of GEOM,
something NetBSD does not have.  It greatly helped the porting work as Pawel is
already a GEOM wizard (gmirror, gjournal and all that).
 
 
This message posted from opensolaris.org

Hello-

I was planning on sending out a more formal sort of introduction in a
few weeks, but.. hey- it came up.

I will be porting ZFS to NetBSD this summer.  Some info on this project
can be found at:
	http://www.olix0r.net/bitbucket/index.cgi/netbsd/zfs
and 
	http://netbsd-soc.sf.net

My proposal is up there now, and I will update it as the project goes
on.  (It doesn''t actually start until June.)

On 2007-04-25 09:03 -0500, Ollivier Robert wrote:
> NetBSD could reuse Pawel''s work on FreeBSD
We will.
> It greatly helped the porting work as Pawel is already a GEOM wizard
> (gmirror, gjournal and all that).
I don''t plan on finishing most of the work in ~10 days like Pawel did
;)
Luckily, I have all summer, and I''m not planning on getting ZPL ported
properly until sometime after that.

I''ve listed some references in the proposal, and the friendly people at
tech-kern at netbsd.org have pointed me to several NetBSD-specific
documents; but, does anyone have other suggested reading?  I''m trying
to
fill my head with all of it now, so that when the project starts I can
get right into the code.

Thanks,
  - Oliver

On Apr 25, 2007, at 9:16 AM, Oliver Gould wrote:
> Hello-
>
> I was planning on sending out a more formal sort of introduction in a
> few weeks, but.. hey- it came up.
>
> I will be porting ZFS to NetBSD this summer.  Some info on this  
> project
> can be found at:
> 	http://www.olix0r.net/bitbucket/index.cgi/netbsd/zfs
> and
> 	http://netbsd-soc.sf.net
>
Cool!  I''ve added you to the porting page:
http://opensolaris.org/os/community/zfs/porting/

We look forward to updates on your progress,
eric

> few weeks, but.. hey- it came up.
Hello!
 
> We will.
Good.
 
> I don''t plan on finishing most of the work in ~10
> days like Pawel did ;)
I''d really really advise you to try to port GEOM which is an awesome
way
of dealing with storage.  You could do w/o of course but it will be
more complicated as Pawel''s work is completely built on GEOM.
 
 
This message posted from opensolaris.org

Not much bliss to report with OpenSolaris.

Getting there was an adventure.  I had to find out how to boot from a changed
root
device (after reconnecting my primary disk, I had disconnected that during the
Solaris install so I wouldn''t accidentally overwrite it).  I had to
find a way of transferring
my /home: ext3 wasn''t recognized, so I copied it to zfs on Linux -- now
that was on a
logical partition, and Solaris doesn''t seem to recognize those; ok,
redid the copy to a
primary partition -- now Solaris complains about not being able to read ZFS
version 6;
ok, redid the copy a third time, this time initing the zfs on the Solaris side
then writing
to it from Linux -- finally it worked.

The times, though, are a "little" discouraging:

1.
real    4m26.736s
user    0m0.509s
sys     0m8.572s

2.
real    4m27.082s
user    0m0.515s
sys     0m8.609s

So zfs-fuse actually outperforms zfs on Solaris by a factor of 2! Now
I''m impatiently
waiting until it gets optimized :)

Disclaimer:

I admit doing a find is a silly FS benchmark.  But I didn''t want to
benchmark,
I wanted to find the best solution for my workload, more or less random access
to
a bunch of small files.  I admit doing a build in Eclipse would have been a
better
sample than find(1), but installing Eclipse on OpenSolaris would even require
that
I build Eclipse from source.  That was more time than I was willing to invest.
 
 
This message posted from opensolaris.org