How badly would a dual-core 1.6 GHz Atom with 4 GBytes RAM be underpowered for serving 4-6 SATA drives? What kind of transfer speed (GBit Ethernet, Intel NICs) can I expect with raidz2 or raidz3? Thanks. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Roy Sigurd Karlsbakk
2010-Jul-21 14:56 UTC
[zfs-discuss] CPU requirements for zfs performance
----- Original Message -----> How badly would a dual-core 1.6 GHz Atom with 4 GBytes RAM > be underpowered for serving 4-6 SATA drives? What kind of > transfer speed (GBit Ethernet, Intel NICs) can I expect with > raidz2 or raidz3?It''ll probably be ok. If you use lzjb compresion, it''ll probably suffice as well. Give it gzip-9 compression, and you might have a cpu bottleneck, but then, for most use, that config will probably do. What sort of traffic do you expect? Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On Wed, Jul 21, 2010 at 04:56:26PM +0200, Roy Sigurd Karlsbakk wrote:> It''ll probably be ok. If you use lzjb compresion, it''ll probably suffice as well. Give it gzip-9 compression, and you might have a cpu bottleneck, but then, for most use, that config will probably do. What sort of traffic do you expect?Thanks for the thumbs-up. Just local GBit LAN. If it does ~20-40 MByte/s it should be quite enough. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 If you plan on using it as a storage server for multimedia data (movies), don''t even bother considering compression, as most media files already come heavily compressed. Dedup might still come in handy, though. - -- Saso On 07/21/2010 05:03 PM, Eugen Leitl wrote:> On Wed, Jul 21, 2010 at 04:56:26PM +0200, Roy Sigurd Karlsbakk wrote: > >> It''ll probably be ok. If you use lzjb compresion, it''ll probably suffice as well. Give it gzip-9 compression, and you might have a cpu bottleneck, but then, for most use, that config will probably do. What sort of traffic do you expect? > > Thanks for the thumbs-up. Just local GBit LAN. If it does > ~20-40 MByte/s it should be quite enough. >-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxHDlYACgkQRO8UcfzpOHChLgCgpwAwsJnmoiDAQ3DCY7YJQpgl +ysAoLelbOxnjq4ONU3Hgf+VD1cG7c0H =k96G -----END PGP SIGNATURE-----
On 21/07/2010 16:12, Saso Kiselkov wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > If you plan on using it as a storage server for multimedia data > (movies), don''t even bother considering compression, as most media files > already come heavily compressed. Dedup might still come in handy, though.Media files may not dedup that well either and more importantly that hardware isn''t likely to be sufficient to get good dedup performance since it is fairly low on DRAM and has no SSD for the L2ARC to keep the DDT cached at least in L2ARC. -- Darren J Moffat
Edward Ned Harvey
2010-Jul-21 15:20 UTC
[zfs-discuss] CPU requirements for zfs performance
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Saso Kiselkov > > If you plan on using it as a storage server for multimedia data > (movies), don''t even bother considering compression, as most media > files > already come heavily compressed. Dedup might still come in handy, > though.If you''re storing movies, I agree compression is a waste. But I think dedup will also be a waste, unless you have multiple copies of the same movie on your disk for some reason.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I do encounter situations when I (or somebody from my family) accidentally create multiple copies of photo albums. :-) - -- Saso On 07/21/2010 05:20 PM, Edward Ned Harvey wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Saso Kiselkov >> >> If you plan on using it as a storage server for multimedia data >> (movies), don''t even bother considering compression, as most media >> files >> already come heavily compressed. Dedup might still come in handy, >> though. > > If you''re storing movies, I agree compression is a waste. But I think dedup > will also be a waste, unless you have multiple copies of the same movie on > your disk for some reason. >-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxHEJIACgkQRO8UcfzpOHBbmQCgqYrov99f1WgtELe0I2pkt44v j0gAoNONmMj6C4fI8l00amJZnhG9rgJz =eWQ9 -----END PGP SIGNATURE-----
On Wed, 2010-07-21 at 17:12 +0200, Saso Kiselkov wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > If you plan on using it as a storage server for multimedia data > (movies), don''t even bother considering compression, as most media files > already come heavily compressed. Dedup might still come in handy, though.Dedup with uncompressible or pre-compressed data is unlikely to be useful, for much the same reason that compression isn''t going to help. You don''t have repeated data. There might be some commonality in headers (but even that that is dubious), but the main data will be different for each of these files. Dedup only wins when you have significantly similar data in different files. Even a one byte difference in a compressed stream will ruin the chance of dedup giving any gains. - Garrett> > - -- > Saso > > On 07/21/2010 05:03 PM, Eugen Leitl wrote: > > On Wed, Jul 21, 2010 at 04:56:26PM +0200, Roy Sigurd Karlsbakk wrote: > > > >> It''ll probably be ok. If you use lzjb compresion, it''ll probably suffice as well. Give it gzip-9 compression, and you might have a cpu bottleneck, but then, for most use, that config will probably do. What sort of traffic do you expect? > > > > Thanks for the thumbs-up. Just local GBit LAN. If it does > > ~20-40 MByte/s it should be quite enough. > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkxHDlYACgkQRO8UcfzpOHChLgCgpwAwsJnmoiDAQ3DCY7YJQpgl > +ysAoLelbOxnjq4ONU3Hgf+VD1cG7c0H > =k96G > -----END PGP SIGNATURE----- > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On 7/21/2010 7:47 AM, Eugen Leitl wrote:> How badly would a dual-core 1.6 GHz Atom with 4 GBytes RAM > be underpowered for serving 4-6 SATA drives? What kind of > transfer speed (GBit Ethernet, Intel NICs) can I expect with > raidz2 or raidz3? > > Thanks. > >For light usage (I''m assuming you want to use this kind of thing for at home), it should be fine. You should have no problems serving via CIFS or NFS or even iSCSI. RAM is a bit light, but if you are not doing anything else which requires much RAM, you should be OK. Without Dedup or Compression turned on, ZFS isn''t that much of a CPU pig. Particularly for serving small numbers of disks. RaidZ[123] will consume slightly more CPU load that mirroring, but not so much as I''d notice in a config such as yours. Remember with RaidZ[123], you only get the IOPS equivalent of a single drive (about 100/s for typical Sata drives), and random I/O performance (throughput) of about the same (i.e. as one disk). However, you should get streaming read/write speeds close to that of the number of data disks (i.e. N-1 for RaidZ1, N-2 for RaidZ2, etc.). So, for things like being a home media server, you''ll easily keep up with a Gbit Ethernet. For doing things like compiling over NFS/CIFS, the disks are going to be your bottleneck. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
Roy Sigurd Karlsbakk
2010-Jul-22 09:23 UTC
[zfs-discuss] CPU requirements for zfs performance
----- Original Message -----> I do encounter situations when I (or somebody from my family) > accidentally create multiple copies of photo albums. :-)I wouldn''t recommend using dedup on this system. Dedup requires lots of RAM or L2ARC, and I don''t think it is suitable for your needs. You may want to "svn co http://svn.karlsbakk.net/svn/roy/deduba" and test that script. It''s a script that looks through a directory and, using MD5 and SHA256, finds identical files. It''s somehow unfinished, but it works. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I didn''t mean to imply that I use it for my media storage, just that I occasionally encounter situations when it could be useful. BR, - -- Saso On 07/22/2010 11:23 AM, Roy Sigurd Karlsbakk wrote:> ----- Original Message ----- >> I do encounter situations when I (or somebody from my family) >> accidentally create multiple copies of photo albums. :-) > > I wouldn''t recommend using dedup on this system. Dedup requires lots of RAM or L2ARC, and I don''t think it is suitable for your needs. You may want to "svn co http://svn.karlsbakk.net/svn/roy/deduba" and test that script. It''s a script that looks through a directory and, using MD5 and SHA256, finds identical files. It''s somehow unfinished, but it works. > > Vennlige hilsener / Best regards > > roy > -- > Roy Sigurd Karlsbakk > (+47) 97542685 > roy at karlsbakk.net > http://blogg.karlsbakk.net/ > -- > I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxIDoIACgkQRO8UcfzpOHD9HACghZz27u6JvhJuLBPrSFJCicrX U00AnA4eDGnK9MGLI07pI3KtABlFKARm =Gn/l -----END PGP SIGNATURE-----