freebsd stable - Feb 2010 - A more secure approach of jail establishment. It could be included in jail chapter of fbsd handbook

Dear all,

After working with jails for a long time, I ended up with a quite 
different approach (architecture) of setting up jails, that seems more 
secure than the one proposed in the freebsd handbook.

The concept is very similar to that of handbook's chapter 15.6.1, only 
it involves a few more concepts. I'll try to be descriptive and 
understandable.

Instead of having the mroot directory (mounted ro) containing the base 
system, and the s directory containing the different configs of each 
different jail (mounted rw), I introduced one more directory, let's call 
it t. Now ls -la /jails/j/mroot gives:

total 32
drwxr-xr-x  14 root  wheel  -  512 Sep 24 18:16 ./
drwxr-xr-x  10 root  wheel  -  512 Feb 10 13:17 ../
drwxr-xr-x   2 root  wheel  - 1024 Sep 24 17:56 bin/
drwxr-xr-x   7 root  wheel  -  512 Sep 24 17:56 boot/
dr-xr-xr-x   2 root  wheel  -  512 Sep 24 17:56 dev/
lrwxr-xr-x   1 root  wheel  -    5 Sep 24 18:15 etc@ -> s/etc
drwxr-xr-x   3 root  wheel  - 1536 Sep 24 17:56 lib/
drwxr-xr-x   2 root  wheel  -  512 Sep 24 17:57 libexec/
drwxr-xr-x   2 root  wheel  -  512 Sep 24 17:56 media/
dr-xr-xr-x   2 root  wheel  -  512 Sep 24 17:56 proc/
drwxr-xr-x   2 root  wheel  - 2560 Sep 24 17:56 rescue/
lrwxr-xr-x   1 root  wheel  -    6 Sep 24 18:15 root@ -> s/root
drwxr-xr-x   2 root  wheel  -  512 Sep 24 18:14 s/
drwxr-xr-x   2 root  wheel  - 2560 Sep 24 17:57 sbin/
lrwxr-xr-x   1 root  wheel  -   11 Sep 24 17:56 sys@ -> usr/src/sys
drwxr-xr-x   2 root  wheel  -  512 Sep 24 18:14 t/
lrwxr-xr-x   1 root  wheel  -    5 Sep 24 18:15 tmp@ -> t/tmp
drwxr-xr-x  14 root  wheel  -  512 Sep 24 18:50 usr/
lrwxr-xr-x   1 root  wheel  -    5 Sep 24 18:16 var@ -> t/var

And ls -la /jails/j/mroot/usr gives:

drwxr-xr-x  14 root  wheel  -   512 Sep 24 18:50 ./
drwxr-xr-x  14 root  wheel  -   512 Sep 24 18:16 ../
lrwxr-xr-x   1 root  wheel  -    15 Sep 24 18:15 X11R6@ -> ../s/usr/-X11R6
drwxr-xr-x   2 root  wheel  -  7168 Sep 24 17:57 bin/
drwxr-xr-x   2 root  wheel  -   512 Sep 24 17:56 games/
drwxr-xr-x  46 root  wheel  -  5120 Sep 24 17:56 include/
drwxr-xr-x   6 root  wheel  - 11776 Sep 24 17:57 lib/
drwxr-xr-x   3 root  wheel  - 12288 Sep 24 17:57 lib32/
drwxr-xr-x   5 root  wheel  -   512 Sep 24 17:56 libdata/
drwxr-xr-x   5 root  wheel  -  1536 Sep 24 17:57 libexec/
lrwxr-xr-x   1 root  wheel  -    14 Sep 24 18:15 local@ -> ../s/usr-local
drwxr-xr-x   2 root  wheel  -   512 Sep 24 17:56 obj/
drwxr-xr-x  68 root  wheel  -  1536 Sep 24 18:50 ports/
drwxr-xr-x   2 root  wheel  -  5120 Sep 24 17:57 sbin/
drwxr-xr-x  25 root  wheel  -   512 Sep 24 17:56 share/
drwxr-xr-x  22 root  wheel  -  1024 Sep 24 18:14 src/


As we can see, only tmp and var is linked to t, while all other symlinks 
are the same with the configuration proposed in the handbook. The 
difference is the following: In my steup, s is mounted ro and t is 
mounted rw. In this way, jail's /etc is mounted ro, /usr/ is mounted ro, 
which means that /usr/local is mounted ro, which means that /s/portbuild 
is mounted read onlty, which means that /root is mounted read only, and 
so on... So, all ports installed on the system, and deamons running 
through /usr/local are now read only and trojaned versions of binaries 
are much more difficult to be installed if the jail is compromised.

In order to be able to manage those jails, I did the following trick:

# cat /etc/fstab
/dev/da0s1a        /        ufs    rw        1    1
/dev/da0s1b        none        swap    sw        0    0
/dev/acd0        /cdrom        cd9660    ro,noauto    0    0


# Jails
/jails/j/mroot        /jails/j/bind        nullfs    ro    0    0
/jails/j/mroot        /jails/j/heimdal    nullfs    ro    0    0
/jails/j/mroot        /jails/j/ldap        nullfs    ro    0    0

/jails/s/ldap        /jails/j/ldap/s        nullfs    ro    0    0
/jails/s/bind        /jails/j/bind/s        nullfs    ro    0    0
/jails/s/heimdal    /jails/j/heimdal/s    nullfs    ro    0    0

/jails/t/heimdal    /jails/j/heimdal/t    nullfs    rw    0    0
/jails/t/bind        /jails/j/bind/t        nullfs    rw    0    0
/jails/t/ldap        /jails/j/ldap/t        nullfs    rw    0    0

# Manage jails
/jails/j/mroot        /jails/manage/bind    nullfs    rw    0    0
/jails/j/mroot        /jails/manage/heimdal    nullfs    rw    0    0
/jails/j/mroot        /jails/manage/ldap    nullfs    rw    0    0

/jails/s/ldap        /jails/manage/ldap/s    nullfs    rw    0    0
/jails/s/bind        /jails/manage/bind/s    nullfs    rw    0    0
/jails/s/heimdal    /jails/manage/heimdal/s    nullfs    rw    0    0

/jails/t/heimdal    /jails/manage/heimdal/t    nullfs    rw    0    0
/jails/t/bind        /jails/manage/bind/t    nullfs    rw    0    0
/jails/t/ldap        /jails/manage/ldap/t    nullfs    rw    0    0

devfs            /jails/manage/heimdal/dev    devfs    rw    0    0
devfs            /jails/manage/bind/dev        devfs    rw    0    0
devfs            /jails/manage/ldap/dev        devfs    rw    0    0

In my setup I have three jails: ldap, heimdal and bind. mroot is monted 
in each jail's root folder, and the s and t directories of each jail are 
null_mounted on each jail's s and t folder respectively. I introduced 
another folder, /jails/manage which has the analogous usage to /jails/j 
folder, only this folder's subfolders are not used as jails. I chroot to 
each folder when I want to have write access on it's corresponding jail, 
and I can perform all my administrative operations (if you notice, even 
mroot is mounted rw in this chroot, so changes in base disto can be 
performed - eg when installing perl where you are asked to change 
/usr/bin/perl). This is the reason why I mount devfs on these chroots.

As a result, with this setup, only /var and /tmp are mounted rw in each 
jail, and all other filesystems are mounted ro. Management (meaning jobs 
that require write access, not starting and/or stopping services) is 
achieved by chrooting to each jail's corresponding chroot.

I am working with this setup for more than 4 months without having any 
problems; I thought that I could propose it as another paragraph in 
chapter 15.6 of freebsd handbook, as an even more secure jail setup, 
instead of just publishing it to some blog,forum,etc.

Thank you all for your time.

-- 
George Mamalakis

IT Officer
Electrical and Computer Engineer (Aristotle Un. of Thessaloniki),
MSc (Imperial College of London)

Department of Electrical and Computer Engineering
Faculty of Engineering
Aristotle University of Thessaloniki

phone number : +30 (2310) 994379

I see people are still installing a full blown OS inside their jails?
You do know that it is possible to have a jail with a single program
inside and not much else, as if it were chroot()ed?


Cheers,
Igor

On Wed, Feb 10, 2010 at 01:10:32PM +0000, Igor Mozolevsky
wrote:
> I see people are still installing a full blown OS inside their jails?
> You do know that it is possible to have a jail with a single program
> inside and not much else, as if it were chroot()ed?
There are two different kind of purposes for jails:
First one is the isolation of single processes, the other one is
the isolation of environments.
For the first one you are right on the ball on, for the second one
you still need the whole userland.

Edwin
-- 
Edwin Groothuis		Website: http://www.mavetju.org/
edwin@mavetju.org	Weblog:  http://www.mavetju.org/weblog/