Hi there, I am running a fresh install of FreeBSD-6.1-Stable as a guest OS in VMWare 1.0.1 with 1 GB of RAM. Whenever I try to grep a large text file (400 MB+), grep terminates with "grep: memory exhausted" I have tried piping grep (cat "file" | grep "search term") I have tried it with -line-buffered ulimit -a show: core file size (blocks, -c) unlimited data seg size (kbytes, -d) 524288 file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 11095 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 65536 cpu time (seconds, -t) unlimited max user processes (-u) 5547 virtual memory (kbytes, -v) unlimited Any help? Thanks, Sean *********************************************************************** This e-mail message is privileged, confidential and subject to copyright. Any unauthorized use or disclosure is prohibited. Le contenu du pr'esent courriel est privil'egi'e, confidentiel et soumis `a des droits d'auteur. Il est interdit de l'utiliser ou de le divulguer sans autorisation. ***********************************************************************
On Wed, Nov 29, 2006 at 03:10:33PM -0500, van Osnabrugge, Sean wrote:> Hi there, > > > > I am running a fresh install of FreeBSD-6.1-Stable as a guest OS in > VMWare 1.0.1 with 1 GB of RAM. > > > > Whenever I try to grep a large text file (400 MB+), grep terminates with > "grep: memory exhausted" > > > > I have tried piping grep (cat "file" | grep "search term") > > I have tried it with -line-buffered > > > > ulimit -a show: > > core file size (blocks, -c) unlimited > > data seg size (kbytes, -d) 524288Try increasing this. I think grep mmaps the file, so the large file could be exceeding your limit. Kris -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20061130/8fc5d2aa/attachment.pgp
On Wed, Nov 29, 2006 at 11:41:23PM -0500, Kris Kennaway wrote:> Try increasing this. I think grep mmaps the file, so the large file > could be exceeding your limit.According to the manpage, grep uses read(2) unless you specify --mmap which then (obviously) uses mmap(2). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On Wed, 29 Nov 2006, Kris Kennaway wrote:> On Wed, Nov 29, 2006 at 03:10:33PM -0500, van Osnabrugge, Sean wrote: >> >> I am running a fresh install of FreeBSD-6.1-Stable as a guest OS in VMWare >> 1.0.1 with 1 GB of RAM. >> >> Whenever I try to grep a large text file (400 MB+), grep terminates with >> "grep: memory exhausted" >> >> I have tried piping grep (cat "file" | grep "search term") >> >> I have tried it with -line-buffered >> >> ulimit -a show: >> >> core file size (blocks, -c) unlimited >> >> data seg size (kbytes, -d) 524288 > > Try increasing this. I think grep mmaps the file, so the large file could > be exceeding your limit.I regularly grep multi-hundred-gigabyte files without a problem on boxes with very little memory, so simple file size is unlikely the cause. However, it would be interesting to know how many lines "wc -l" thinks the text file has, and what the length of the longest line is. Robert N M Watson Computer Laboratory University of Cambridge