Hi all, I'm struggling with some hosting environments where I am managing a large number of jails (>100) spread over about a dozen servers. I am starting to see disk space as a real problem, especially given that each physical box needs to be autonomous - i.e. I can't rely on any external storage, and I am limited to 1U and 2U servers. The solution, or at least parts of it, would be to have certain parts of the jail filesystems mounted in via nullfs (acceptable solution) or unionfs (ideal solution). However, ever since FreeBSD 4.10 this has been a major problem, as both filesystems started exhibiting major stability and data integrity issues. Before I start playing with this again, I'd like to know if any work has been done on either of these in 5.x. Specifically, I'm currently running 5.3-p6 or newer on all the systems, and as of yesterday I've been using 5.4-prerelease (cvsup) on a couple of test systems. What can I expect to see when trying nullfs and/or unionfs today? Has anything changed? Do I have even a remote chance of making it work - and if it doesn't work, what are my chances of anyone having time or energy to look into it? I'm an admin only, no coder, otherwise I'd be happy to look into it myself. Thanks, /Eirik
On Thu, 2005-05-05 at 14:06 +0200, Eirik =?ISO-8859-1?B?2A==?=verby wrote:> [...] The solution, or at least parts of it, would be to have certain parts of the > jail filesystems mounted in via nullfs (acceptable solution) or unionfs > (ideal solution). However, ever since FreeBSD 4.10 this has been a major > problem, as both filesystems started exhibiting major stability and data > integrity issues. > [...] > What can I expect to see when trying nullfs and/or unionfs today? Has > anything changed?Don't know if anything has changed, but I'm using nullfs to mount the ports directory of the host into jails. No ill effects. Works great, both under 4.10 and 5.3. (Back when I toyed with unionfs, I found that to be a bit unstable. But nullfs appears pretty solid) Regards, Frank -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: This is a digitally signed message part Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20050505/1427b252/attachment.bin
Eirik ? verby <ltning@anduin.net> wrote:> Before I start playing with this again, I'd like to know if any work > has been done on either of these in 5.x. Specifically, I'm currently > running > 5.3-p6 or newer on all the systems, and as of yesterday I've been > using > 5.4-prerelease (cvsup) on a couple of test systems.Hi. I am running on 5.3-STABLE since december 2004. We run 14 jails and use nullfs for some shared parts. Everything is just fine. I tryed unionfs, it worked stable but i didn't do what i wanted it to do or probably i didn't get its usage right :) -- Regards, Artem Kuchin IT Legion Ltd. Moscow, Russia www.itlegion.ru matrix@itlegion.ru +7 095 232-0338
Am Donnerstag, 5. Mai 2005 14:06 schrieb Eirik ?verby:> Hi all, > > I'm struggling with some hosting environments where I am managing a > large number of jails (>100) spread over about a dozen servers. I am > starting to see disk space as a real problem, especially given that each > physical box needs to be autonomous - i.e. I can't rely on any external > storage, and I am limited to 1U and 2U servers. > > The solution, or at least parts of it, would be to have certain parts of > the jail filesystems mounted in via nullfs (acceptable solution) or > unionfs (ideal solution). However, ever since FreeBSD 4.10 this has been > a major problem, as both filesystems started exhibiting major stability > and data integrity issues. > > Before I start playing with this again, I'd like to know if any work has > been done on either of these in 5.x. Specifically, I'm currently running > 5.3-p6 or newer on all the systems, and as of yesterday I've been using > 5.4-prerelease (cvsup) on a couple of test systems. > > What can I expect to see when trying nullfs and/or unionfs today? Has > anything changed? Do I have even a remote chance of making it work - and > if it doesn't work, what are my chances of anyone having time or energy > to look into it? I'm an admin only, no coder, otherwise I'd be happy to > look into it myself.Nullfs is as far as I can tell stable on 5.4 but the performance problem together with jails is not solved in 5.4, only in 6. And like Jeff said, it's not sure that it gets MFCd since lot of VFS changes are requred. -Harry> > Thanks, > /Eirik > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org"-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20050505/416cc0f5/attachment.bin
On Thu, 5 May 2005, Eirik [ISO-8859-1] Øverby wrote:> The solution, or at least parts of it, would be to have certain parts of > the jail filesystems mounted in via nullfs (acceptable solution) or > unionfs (ideal solution). However, ever since FreeBSD 4.10 this has been > a major problem, as both filesystems started exhibiting major stability > and data integrity issues.I'm running 4.11 with ~90 mount/jails running on two of our servers ... haven't noticed any stability problems ... what are you seeing? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Eirik ?verby writes: > [...] > What can I expect to see when trying nullfs and/or unionfs today? Has > anything changed? Do I have even a remote chance of making it work - and if > it doesn't work, what are my chances of anyone having time or energy to look > into it? I'm an admin only, no coder, otherwise I'd be happy to look into it > myself. I'm using unionfs to mount a copy of my ports tree into a jail on a fairly currently patched 5.3 system. It works beautifully except that it sometimes can't be unmounted as the machine shuts down, leading to an fsck. I've been trying to characterize it. Seems like I can mount it, start a jail, stop the jail, and unmount it just fine. However if I do anything in the jail's ports tree, then it won't unmount. Last experiment I did was to log into the jail and do a couple of 'syncs', then log out, shut the jail down and unmount it. That worked that one time. Not enough to file a bug yet, but the anecdote might be useful. g.
On 06-05-05 09:25, "Danny Braniss" <danny@cs.huji.ac.il> wrote:> >> Interesting approach. Is this with 4.x or 5.x? How do you union-mount /etc >> (mount command/fstab entry)? >> > > been doing it since 4.x (i think x < 9)Any idea how unionfs will behave if stacked (more mounts on top of each other)? I was playing with the thought of having a "template" jail directory which I unionmount into my jails, then perhaps use your trick to union-mount a md device into certain points in the jail. Got a gut feeling about that? /Eirik> in initdiskless (5.x) we have: > > if [ -e /conf/union ]; then > kldload unionfs > mount_md 4096 /conf/etc > chmod 755 /conf/etc > mount_unionfs /conf/etc /etc > ls -R /etc > /dev/null > touch /etc/.sentinel > md_created_etc=created > fi > > danny > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >