Hi, I know that is not recommended by Sun to use ZFS on 32 bits machines but, what are really the consequences of doing this ? I have an old Bipro Xeon server (6 GB ram , 6 disks), and I would like to do a raidz with 4 disks with Solaris 10 update 4. Thanks, Ben
Ben wrote:> Hi, > > I know that is not recommended by Sun > to use ZFS on 32 bits machines but, > what are really the consequences of doing this ?Depends on what kind of performance you need.> I have an old Bipro Xeon server (6 GB ram , 6 disks), > and I would like to do a raidz with 4 disks with Solaris 10 update 4.The issue with 32 bit kernels is the amount of virtual address space available and the impact this has on allowing the ARC to be as big as needed. I run ZFS on a 32 bit AMD Athlon 900MHz with 640Mb RAM and it works just fine for me - but then it for me it is just an NFS and web (and webdav) server for home (serving iTunes, iPhoto locally and my website to the internet). -- Darren J Moffat
>Ben wrote: >> Hi, >> >> I know that is not recommended by Sun >> to use ZFS on 32 bits machines but, >> what are really the consequences of doing this ? > >Depends on what kind of performance you need. > >> I have an old Bipro Xeon server (6 GB ram , 6 disks), >> and I would like to do a raidz with 4 disks with Solaris 10 update 4. > >The issue with 32 bit kernels is the amount of virtual address space >available and the impact this has on allowing the ARC to be as big as >needed. > >I run ZFS on a 32 bit AMD Athlon 900MHz with 640Mb RAM and it works just >fine for me - but then it for me it is just an NFS and web (and webdav) >server for home (serving iTunes, iPhoto locally and my website to the >internet).I think it''s specfically problematic on 32 bit systems with large amounts of RAM. Then you run out of virtual address space in the kernel quickly; a small amount of RAM (I have one with 512MB) works fine. Casper
On Thu, Mar 06, 2008 at 11:39:25AM +0100, Casper.Dik at Sun.COM wrote:> > I think it''s specfically problematic on 32 bit systems with large amounts > of RAM. Then you run out of virtual address space in the kernel quickly; > a small amount of RAM (I have one with 512MB) works fine.I have a 32-bit machine with 4GB of ram. I''ve been researching this for some time now, but can''t find it anywhere. At some point, someone posted a system config tweak to increase the amount of memory available to the ARC on a 32-bit platform. Who was that, and could you please re-post that tweak? Thanks!! -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
2008/3/6, Brian Hechinger <wonko at 4amlunch.net>:> On Thu, Mar 06, 2008 at 11:39:25AM +0100, Casper.Dik at Sun.COM wrote: > > > > I think it''s specfically problematic on 32 bit systems with large amounts > > of RAM. Then you run out of virtual address space in the kernel quickly; > > a small amount of RAM (I have one with 512MB) works fine. > > > I have a 32-bit machine with 4GB of ram. I''ve been researching this for > some time now, but can''t find it anywhere. At some point, someone posted > a system config tweak to increase the amount of memory available to the > ARC on a 32-bit platform. Who was that, and could you please re-post that > tweak?I don''t know how to change the ARC sise, but use this to increase kernel addres space: eeprom kernelbase=0x50000000 Your user address space will shrink when you do that.
ZFS is not 32-bit safe. There are a number of places in the ZFS code where it is assumed that a 64-bit data object is being read atomically (or set atomically). It simply isn''t true and can lead to weird and bugs. This message posted from opensolaris.org
Brian D. Horn wrote:> ZFS is not 32-bit safe. There are a number of places in the ZFS code where > it is assumed that a 64-bit data object is being read atomically (or set > atomically). It simply isn''t true and can lead to weird and bugs.Bug numbers please. -- Darren J Moffat
> ZFS is not 32-bit safe.while this is kinda true, if the systems has 2G or less of ram it shouldn''t be an issue other than poor performance for lack of ARC. Rob
>ZFS is not 32-bit safe. There are a number of places in the ZFS code where >it is assumed that a 64-bit data object is being read atomically (or set >atomically). It simply isn''t true and can lead to weird and bugs.Where do you get that information? (First I''ve heard of it and I have a hard time believing it) Casper
On Thu, Mar 6, 2008 at 10:22 AM, Brian D. Horn <Brian.Horn at sbcglobal.net> wrote:> ZFS is not 32-bit safe. There are a number of places in the ZFS code where > it is assumed that a 64-bit data object is being read atomically (or set > atomically). It simply isn''t true and can lead to weird and bugs.This is disturbing, especially as I have not seen this documented anywhere. I have a dual P-III 550 Intel system with 1 GB of RAM (Intel L440GX+ motherboard). I am running Solaris 10U4 and am using ZFS (mirrors and stripes only, no RAIDz). While this is ''only'' a home server, I still cannot afford to lose over 500 GB of data. If ZFS isn''t supported under 32 bit systems then I need to start migrating to UFS/SLVM as soon as I can. I specifically went with 10U4 so that I would have a stable, supportable environment. Under what conditions are the 32 bit / 64 bit problems likely to occur ? I have been running this system for 6 months (a migration from OpenSuSE 10.1) without any issues. The NFS server performance is at least an order of magnitude better than the SuSE server was. -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Sound Designer, Noel Coward''s Hay Fever @ Albany Civic Theatre, Feb./Mar. 2008 -> Facilities Coordinator, Albacon 2008
Take a look at CR 6634371. It''s worse than you probably thought. This message posted from opensolaris.org
Brian D. Horn wrote:> Take a look at CR 6634371. It''s worse than you probably thought. >Actually, almost all of the problems noted in that bug are statistics. - Bart -- Bart Smaalders Solaris Kernel Performance barts at cyber.eng.sun.com http://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird."
Paul - Don''t substitute redundancy for backup... if your data is important to you, for the love of steak, make sure you have a backup that would not be destroyed by, say, a lightening strike, fire or stray 747. For what it''s worth, I''m also using ZFS on 32 bit and am yet to experience any sort of issues. An external 500GB disk + external USB enclosure runs for what - $150? That''s what I use anyways. :) Nathan. Paul Kraus wrote:> On Thu, Mar 6, 2008 at 10:22 AM, Brian D. Horn <Brian.Horn at sbcglobal.net> wrote: > >> ZFS is not 32-bit safe. There are a number of places in the ZFS code where >> it is assumed that a 64-bit data object is being read atomically (or set >> atomically). It simply isn''t true and can lead to weird and bugs. > > This is disturbing, especially as I have not seen this > documented anywhere. I have a dual P-III 550 Intel system with 1 GB of > RAM (Intel L440GX+ motherboard). I am running Solaris 10U4 and am > using ZFS (mirrors and stripes only, no RAIDz). While this is ''only'' a > home server, I still cannot afford to lose over 500 GB of data. If ZFS > isn''t supported under 32 bit systems then I need to start migrating to > UFS/SLVM as soon as I can. I specifically went with 10U4 so that I > would have a stable, supportable environment. > > Under what conditions are the 32 bit / 64 bit problems likely > to occur ? I have been running this system for 6 months (a migration > from OpenSuSE 10.1) without any issues. The NFS server performance is > at least an order of magnitude better than the SuSE server was. >
>Brian D. Horn wrote: >> Take a look at CR 6634371. It''s worse than you probably thought. >> > >Actually, almost all of the problems noted in that bug are statistics.But not exactly all and some are used for othe rpurposes. And some of the other values will never exceed 32 bit in 32 bit systems (such as the amount of memory in use by ZFS for various caches) It''s still something which warants investigation, but it should be noted that any use of a value which is generally atoamically updated should consider the value just read at most as advisory as there is no guarantee that it is still the same. If you find yurself in need of doing an atomic read there are some questions you should be asking yourself. Casper
On Thu, Mar 06, 2008 at 02:07:09PM +0100, Mattias Pantzare wrote:> > I don''t know how to change the ARC sise, but use this to increase > kernel addres space: > > eeprom kernelbase=0x50000000Ah ha, that''s what I was thinking about.> Your user address space will shrink when you do that.Yes, but from what I understand, the kernel address space in a 32-bit machine is painfully small for the ARC cache. I guess it''s all a matter of trial and error to find the best spot for my needs. Anybody know offhand what the default value is? -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
On Thu, Mar 06, 2008 at 10:29:46AM -0500, Rob Logan wrote:> > ZFS is not 32-bit safe. > > while this is kinda true, if the systems has 2G or less of ram > it shouldn''t be an issue other than poor performance for lack of > ARC.So what happens if you have a 32-bit machine with 4GB RAM like I do? -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
It isn''t a simple as getting an old stale value. You can get a totally incorrect value. Example: Let us assume a monotonically increased 64-bit values which at the start of this discussion is: 0x00000000ffffffff (32-bits 0, 32-bits 1). The 32-bit kernel goes to read the 64-bit value and does so by first loading the upper 32-bits (all 0s). At this point the thread is preempted or on an MP system another thread on another processor starts to do a 64-bit atomic increment. While that update is happening the first thread cannot read the lower 32-bits. The values after the atomic 64-bit update is now 0x0000000100000000 (lower 32-bits now all 0). Now that the update is complete the first thread reads the lower 32-bits. 0. So the 32-bit kernel has the potential for reading a 64-bits of 0 even though all 0 may NEVER have been an valid 64-bit value and all writes to that 64-bit location NEVER set it to all zero. In short, all those 64-bit locations that treat the value as being modified atomically can never rely on being read correctly, only updated correctly. Fortunately, on every modern 32-bit x86 processor, there is a compare and swap 8 byte instruction which can be used (with the lock prefix) to guarantee correct behavior, but first Solaris has to make sure that all 64-bit reads and writes use that feature and secondly, this isn''t a general solution as other 32-bit processors may have no way to do a 64-bit atomic read/write without additional external locking. This message posted from opensolaris.org
If you look at the contents of the CR it does say that. However there are something like 200 instances and of those perhaps one or two dozen are NOT statistics. A few examples from around the kernel were pointed out. (interrupt handling, NIC driver, ZFS, ...) This message posted from opensolaris.org
On Fri, Mar 7, 2008 at 11:43 AM, Brian D. Horn <Brian.Horn at sbcglobal.net> wrote:> If you look at the contents of the CR it does say that. However there > are something like 200 instances and of those perhaps one or two > dozen are NOT statistics. A few examples from around the kernel > were pointed out. (interrupt handling, NIC driver, ZFS, ...)Please forgive my ignorance, I am an admin and not a developer, but is there any serious significance to this bug on 32 bit systems with less than 2GB memory (and does the total amount of virtual memory factor into this... in other words, does physical + swap have to be less than 2 GB). -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Sound Designer, Noel Coward''s Hay Fever @ Albany Civic Theatre, Feb./Mar. 2008 -> Facilities Coordinator, Albacon 2008
On Mar 6, 2008, at 7:58 AM, Brian D. Horn wrote:> Take a look at CR 6634371. It''s worse than you probably thought.The only place i see ZFS mentioned in that bug report is regarding z_mapcnt. Its being atomically inc/dec in zfs_addmap()/zfs_delmap() - so those are ok. In zfs_frlock(), technically we should be using atomic_add_64(, 0), though locking isn''t even necessary around checking z_mapcnt. Its advisory in this case. The ASSERT in zfs_delmap() is broken. That should be fixed. Not an issue on non-debug kernels (which our customers use). The statement of: "> ZFS is not 32-bit safe. There are a number of places in the ZFS > code where > it is assumed that a 64-bit data object is being read atomically > (or set > atomically). It simply isn''t true and can lead to weird and bugs." is completely off though. If you find other places in ZFS that have the problem mentioned in 6634371, please let us know. eric
eric kustarz wrote:> > On Mar 6, 2008, at 7:58 AM, Brian D. Horn wrote: > >> Take a look at CR 6634371. It''s worse than you probably thought. > > The only place i see ZFS mentioned in that bug report is regarding > z_mapcnt. Its being atomically inc/dec in zfs_addmap()/zfs_delmap() - > so those are ok. > > In zfs_frlock(), technically we should be using atomic_add_64(, 0), > though locking isn''t even necessary around checking z_mapcnt. Its > advisory in this case. > > The ASSERT in zfs_delmap() is broken. That should be fixed. Not an > issue on non-debug kernels (which our customers use). > > The statement of: > " >> ZFS is not 32-bit safe. There are a number of places in the ZFS code >> where >> it is assumed that a 64-bit data object is being read atomically (or set >> atomically). It simply isn''t true and can lead to weird and bugs. > " > > is completely off though. > > If you find other places in ZFS that have the problem mentioned in > 6634371, please let us know.I just started looking around, but I''m not going to track down each an every case, but... arc.c:arc_evict the bare use of arc_mru_ghost->arcs_size and arc_mru_ghost->arcs_lsize[type] appears suspect. arc.c:arc_adjust the bare use of arc_anon->arcs_size and arc_mur->arcs_size appear suspect. arc.c:arc_reclaim_thread arc_mru_ghost->arcs_size and arc_mfu_ghost->arcs_size arc.carc_adapt arc_mru_ghost->arcs_size arc_mfu_ghost->arcs_size arc.c:arcspace_return arc_meta_used zfs_vnops.c:zfs_frlock zp->z_mapcnt appears suspect There are certainly others and some may not have bad effects, but these are 64-bit objects which are being atomically updated, but then later (apparently) not atomically read and are not statistics. I''m suggesting that a through examination be done on the code (zfs and other places in Solaris) to look for all such bare uses of 64-bit non local data. Even the statistics cases should be fixed, but they are critical, just confusing when they are wrong.> > eric