Hey All I have a dual Xeon box used by our students for data crunching. It has 4GB of RAM. After initial installation everything went well until someone found they couldn;t allocate more than 512MB of RAM per process. After some poking around I found some things things to adjustin the kernel conf file: options MAXDSIZ="(2000*1024*1024)" options MAXSSIZ="(128*1024*1024)" options DFLDSIZ="(512*1024*1024)" Now here's the problem, consistent with three versions of 4-STABLE over the last couple of weeks and finally yesterday. Everything works perfectly as abve except that limit reports datasize 2048000 kbytes - which kind of makes sense given the figures above. However, the guys want to be able to use all the RAM, not just 2GB If I increase MAXDSIZ to 2048 then limit reports datasiz unlimited but mysql will not run, unfortunatly I've lost the log file from then but it was something like a pthreads error 36 - mysql would just start up and shut down. Other than that though the machine ran fine. If MAXDSIZ is set to 4096 then the kernel will not even load, if it's set to 4095 it gets part way through the rc scripts and then crashes. 3500 gets you almost but not quite booted - all kinds of apps failing unable to allocate memory. Any help would be much appreciated - it's a production box and I don't have anothe 4GB box to play with. Cheers, Irvine Short Sys Admin SANBI, University of the Western Cape, South Africa http://www.sanbi.ac.za tel: +27-21-959 3645 cel: +27-82-494 3828
> Now here's the problem, consistent with three versions of 4-STABLE over > the last couple of weeks and finally yesterday. > > Everything works perfectly as abve except that limit reports datasize > 2048000 kbytes - which kind of makes sense given the figures above. > > However, the guys want to be able to use all the RAM, not just 2GBSorry, due to design issues, it isn't possible to have virtual sizes larger than about 3GB on FreeBSD. This is because the kernel is mapped in the upper part of the virtual address space. Of course you can use all of your 4GB of RAM - just not all of it at the same time in one process. -DG David G. Lawrence Download Technologies, Inc. - http://www.downloadtech.com - (866) 399 8500 TeraSolutions, Inc. - http://www.terasolutions.com - (888) 346 7175 The FreeBSD Project - http://www.freebsd.org Pave the road of life with opportunities.
OK, more info... David Lawrence said:> Sorry, due to design issues, it isn't possible to have virtual sizes >larger than about 3GB on FreeBSD. This is because the kernel is mapped in >the upper part of the virtual address space. Of course you can use all of >your 4GB of RAM - just not all of it at the same time in one process.OK, fair enough. Is this going to be any different in FreeBSD with PAE (Intel's scheme for 32bit stuff using > 4GB RAM)? Should I try 5.1? Or isn't 4.9 going to have PAE support anyway? Given what David says though, why do I have a problem with MySQL getting thread errors with MAXDSIZ 2048 or greater? Why does tcsh's "limit" report datasize unlimited when MAXDSIZ is over 2048? -- Irvine On Sat, 13 Sep 2003, Irvine Short wrote:> After initial installation everything went well until someone found they > couldn;t allocate more than 512MB of RAM per process. After some poking > around I found some things things to adjustin the kernel conf file: > > options MAXDSIZ="(2000*1024*1024)" > options MAXSSIZ="(128*1024*1024)" > options DFLDSIZ="(512*1024*1024)" > > Now here's the problem, consistent with three versions of 4-STABLE over > the last couple of weeks and finally yesterday. > > Everything works perfectly as abve except that limit reports datasize > 2048000 kbytes - which kind of makes sense given the figures above. > > However, the guys want to be able to use all the RAM, not just 2GB > > If I increase MAXDSIZ to 2048 then limit reports datasiz unlimited but > mysql will not run, unfortunatly I've lost the log file from then but it > was something like a pthreads error 36 - mysql would just start up and > shut down. > > Other than that though the machine ran fine. > > If MAXDSIZ is set to 4096 then the kernel will not even load, if it's set > to 4095 it gets part way through the rc scripts and then crashes. > > 3500 gets you almost but not quite booted - all kinds of apps failing > unable to allocate memory. > > Any help would be much appreciated - it's a production box and I don't > have anothe 4GB box to play with. > > Cheers, > > Irvine Short > > Sys Admin > SANBI, University of the Western Cape, South Africa > http://www.sanbi.ac.za > tel: +27-21-959 3645 > cel: +27-82-494 3828 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >Irvine Short Sys Admin SANBI, University of the Western Cape, South Africa http://www.sanbi.ac.za tel: +27-21-959 3645 cel: +27-82-494 3828
> However we have a situation where if I set MAXDSIZ to 2048 or above then > things break, so FreeBSD right now has an effectivce limit of 2GB per > process. > > Is this to be considered a bug or a feature?It orta work above 2Gb, and has been known to work to about 3Gb. The fact that tcsh is reporting "unlimited" is probably a bug in tcsh (signed vs unsigned). But you can't put a limit on how much above 2Gb will definitely work, because the kernel's demand for VM space is variable, depending on hardware config, job mix & current activity. (The Kernel uses VM space for various data structures, but the ones that matter most in large RAM machines are the processes VM page tables etc.) So a given MAXDSIZE might work most of the time but fail in interesting ways when things get hot. And failing due to out of VM space is not handled very gracefully at all. If you're lucky you'll get a panic.....> Then, with the PAE support, lets say I have 8GB of RAM. Will those same > design limitations restrict me to 3GB of RAM, or 2GB, or will I be able to > have a single process addressing 4GB all by itself?No, the architectural limit is that the kernel and the user program share the one 32-bit address space, and the kernel typically needs up to 1Gb of it, so no user process can be bigger than this 3 Gb (or so) limit. So PAE will not help at all here (and in fact may make it worse as PAE data structures may be large than normal ones.) But with PAE you could have 2 or 3 of these large processes all RAM-resident at once, if you are willing to buy that much RAM. Fixing this is basically not going to happen; 64bit CPUs will be standard way before a project of that magnitude would be finished. Greg, not really a kernel hacker.
On Sat, Sep 13, 2003 at 12:39:40PM +0200, Irvine Short wrote:> Given what David says though, why do I have a problem with MySQL getting > thread errors with MAXDSIZ 2048 or greater?The larger you set MAXDSIZ, the less VM is available for mmap(). Quoted from http://www.freebsd.org/cgi/mid.cgi?id=200212242312.gBONCjqD044717@apollo.backplane.com yup, pthread_create in FreeBSD is implemented by mmap(). No, I do not know how to fix it. Leave the MAXDSIZ untouched is the best strategy. Clive