Hello all. I`m new to btrfs and do some testing now. What we have: SL6 - kernel 2.6.32-131.2.1.el6.x86_64 mdadm RAID5 with 8 HDD - 27T partition. Mount options is "noatime,noacl,compress-force" I use scribe daemon to copy log files from 200 hosts to that partition for stress testing. But I found what compression ratio is really small. Partition is full of regular log files - plain text. Most of them a big ones(10-20Gb) and they grow in realtime. du -hs /logs/ 1.1T /logs/ And compressed size is(df -h): 908G Regular gzip do a significant more ratio. There is a problem here? File size? Realtime grow of a files? -- Best regards, Proskurin Kirill -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi, On Mon, Jun 20, 2011 at 03:29:45PM +0400, Proskurin Kirill wrote:> What we have: > SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > mdadm RAID5 with 8 HDD - 27T partition.btw .32 is very old> Mount options is "noatime,noacl,compress-force" > I use scribe daemon to copy log files from 200 hosts to that > partition for stress testing. > > But I found what compression ratio is really small.It is and it will be due to nature of compression method used and the constraints given: * compression has to be fast, only real-time methods can be considered which have natural speed/ratio tradeoff * you''re using zlib (lzo was added recently), which is not that fast and does not have a great compression ratio * there is maximum length of compressed data per round, it''s hardcoded to 128KB by now * zlib works in streaming mode, which compresses whole 128KB with dictionary reused * lzo (as implemented in btrfs) however does not reuse dictionary and compresses each page separately; resulting size is even bigger than it would be when full 128KB were compressed> Partition is full of regular log files - plain text. Most of them a > big ones(10-20Gb) and they grow in realtime.Realtime growth should not be a problem. The situation can be slightly improved if we use a method capable of streamed compression (ie. dictionary reuse). This is naturally supported by eg. quicklz and snappy. There are a few other realtime methods (fastlz, lzrw, lzjb) which may be extended to support streaming compression, but with careful evaluation of the result. david -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hallo, Proskurin, Du meintest am 20.06.11:> I`m new to btrfs and do some testing now.> What we have: > SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > mdadm RAID5 with 8 HDD - 27T partition.You should take a newer kernel. On my system I needed 2.6.38.5 and newer for btrfs; older kernels lead to unreadable partitions. Viele Gruesse! Helmut -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 06/20/2011 06:34 PM, Helmut Hullen wrote:> Hallo, Proskurin, > > Du meintest am 20.06.11: > >> I`m new to btrfs and do some testing now. > >> What we have: >> SL6 - kernel 2.6.32-131.2.1.el6.x86_64 >> mdadm RAID5 with 8 HDD - 27T partition. > > You should take a newer kernel. On my system I needed 2.6.38.5 and newer > for btrfs; older kernels lead to unreadable partitions.What kernel version is most recommended at this time? -- Best regards, Proskurin Kirill -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jun 20, 2011 at 07:17:22PM +0400, Proskurin Kirill wrote:> On 06/20/2011 06:34 PM, Helmut Hullen wrote: > >Du meintest am 20.06.11: > >>What we have: > >>SL6 - kernel 2.6.32-131.2.1.el6.x86_64 > >>mdadm RAID5 with 8 HDD - 27T partition. > > > >You should take a newer kernel. On my system I needed 2.6.38.5 and newer > >for btrfs; older kernels lead to unreadable partitions. > > What kernel version is most recommended at this time?It''s always "the latest version you can manage to run". So, currently, 2.6.39 or the 3.0-rc series. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk == PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- O tempura! O moresushi! ---
On Mon, Jun 20, 2011 at 10:22 AM, Hugo Mills <hugo@carfax.org.uk> wrote:> On Mon, Jun 20, 2011 at 07:17:22PM +0400, Proskurin Kirill wrote: >> On 06/20/2011 06:34 PM, Helmut Hullen wrote: >> >Du meintest am 20.06.11: >> >>What we have: >> >>SL6 - kernel 2.6.32-131.2.1.el6.x86_64 >> >>mdadm RAID5 with 8 HDD - 27T partition. >> > >> >You should take a newer kernel. On my system I needed 2.6.38.5 and newer >> >for btrfs; older kernels lead to unreadable partitions. >> >> What kernel version is most recommended at this time? > > It''s always "the latest version you can manage to run". So, > currently, 2.6.39 or the 3.0-rc series. > > Hugo. > have to audit the code yourself > -- > === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ==> PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk > --- O tempura! O moresushi! --- > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iD8DBQFN/2W9IKyzvlFcI40RAgmuAKCEbTdS/ZjWzNMX3D5t8zNKbULJQACgyl6X > e2SIUGnkDu1BFagOL2e5ymM> =SO2f > -----END PGP SIGNATURE----- > >Actually, the btrfs in the newest RHEL 2.6.32 kernels (and by extension, SL kernels) has been heavily backported, so it is fairly current, although probably not up to 2.6.39 level yet. It isn''t the most current, but it is more current than most people realize, and should be fairly stable. I think you''ll want the RHEL 6.1 (or SL 6.1) 2.6.32 kernel to get the most recent Btrfs backporting. I haven''t seen any detailed information posted on the extent of the backporting. So, for right now, you''re kind-of on your own to ascertain if the btrfs in your RHEL kernel is suitable for your needs. I believe the original poster is getting confused by results of the ''df'' command. Running ''df'' will include a large amount of reserved space for metadata. The Btrfs Wiki has some insights on how to evaluate the space consumed by a Btrfs volume. But, in general, running ''df'' will not give you a complete picture of disk utilization with Btrfs. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html