Happy new year to all! I experience an interesting Problem with perl (both 5.0 and 5.8.0) and Compress::Zlib::memGunzip on FreeBSD 4.9-STABLE (Oct. 28th 2003): Uncompressing a larger ascii-file (200-300kB compressed, 20-40MB uncompressed) in memory takes a ___long____ time (the larger the uncompressed file, the more time it takes per extracted MB): uncompressed size 5 MB => ca. 25 seconds ( 5 seconds/MB) 40 MB => ca. 1200 seconds (30 seconds/MB) gunzip only needs slightly more than 1 second for the 40MB file. I don't have this problem with SuSE Linux 8.1 where extraction with memGunzip only needs about 1.5 seconds and with gunzip around 0.5 seconds (Linux is on a faster system). I am not sure if this problem still exists with CURRENT. Searching on the internet gave the impression that this might be malloc/FreeBSD related. Any chance of this being resolved? Regards, Holger Kipp Example perl program: --- 8< ------------------------------------------------------- use Compress::Zlib; use Time::HiRes qw(gettimeofday tv_interval) ; my $start = [ gettimeofday ] ; my $file = shift ; open FILE, "<$file" || die "Cannot open file $file: $!\n" ; undef $/ ; my $in = <FILE>; close FILE ; my $out = Compress::Zlib::memGunzip($in); die "undefined\n" if ! defined $out ; open FILE, ">$file.tmp" || die "Cannot open file $file: $!\n" ; print FILE $out; my $elapsed = tv_interval ( $start, [gettimeofday]); print "elapsed time is $elapsed\n" ; --- 8< -------------------------------------------------------
On Tue, 06 Jan 2004 11:29:05 +0000 Holger Kipp <Holger.Kipp@alogis.com> wrote:> Searching on the internet gave the impression that this might be > malloc/FreeBSD related. Any chance of this being resolved?Use perls own malloc implementation (make -DWITHOUT_PERL_MALLOC for an up-to-date perl5.8 port, if you use make -DWITH_THREADS perls own malloc implementation gets disabled by the FreeBSD port)? AFAIK perl 5.0 in the 4.x basesystem doesn't use its own malloc implementation. I've tried it on -current with perl 5.6.1: - I've aborted your testcode after 5 minutes without perl malloc - with perl malloc it needs ~9 seconds Bye, Alexander. -- I will be available to get hired in April 2004. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7
Alexander Leidinger (Alexander@Leidinger.net) wrote:> >On Tue, 06 Jan 2004 11:29:05 +0000 >Holger Kipp <Holger.Kipp@alogis.com> wrote: > >> Searching on the internet gave the impression that this might be >> malloc/FreeBSD related. Any chance of this being resolved? > >Use perls own malloc implementation (make -DWITHOUT_PERL_MALLOC for an >up-to-date perl5.8 port, if you use make -DWITH_THREADS perls own malloc >implementation gets disabled by the FreeBSD port)? AFAIK perl 5.0 in the >4.x basesystem doesn't use its own malloc implementation.Upgraded Perl 5.8.0 to 5.8.2 which has these switches set already, so I guess the problem is well-known. Kris Kennaway wrote: "For any algorithm one can come up with a workload that makes it perform badly. The trick is making it perform well for common workloads, which FreeBSD's malloc does." (http://lists.freebsd.org/pipermail/freebsd-hackers/2003-September/003059.html) Even so, if Perl malloc and Linux malloc are O(n) with a small constant and FreeBSD malloc behaves like O(n*n) or worse with a large constant in this easy example (of a real-world applikation), this behaviour should imho be investigated.... See also http://www.dent.med.uni-muenchen.de/~wmglo/malloc-slides.html Regards, Holger
Holger Kipp <Holger.Kipp@alogis.com> writes:> Searching on the internet gave the impression that this might be > malloc/FreeBSD related. Any chance of this being resolved?This is basically a bug in Perl (poor choice of algorithm for growing strings). Perl's malloc() implementation knows about and compensates for this bug. FreeBSD's malloc() implementation does not. DES -- Dag-Erling Sm?rgrav - des@des.no
Kris Kennaway (kris@obsecurity.org) wrote:> >> This is basically a bug in Perl (poor choice of algorithm for growing >> strings). Perl's malloc() implementation knows about and compensates >> for this bug. FreeBSD's malloc() implementation does not.>I heard someone say it was fixed in perl 5.<mumble> (8?)As I now know, at least the perl(5.8.2) port is build with perl malloc as default. Sorry for starting this thread - but I learned a lot about malloc this way ;-) and with perl v5.8.2 performance is very good again! Thanx for all who replied! Regards, Holger