Gluster users - Jun 2010 - Notes regarding PXE-booting from GlusterFS

Hi all:

I saw the thread "Netboot / PXE-Boot from glusterfs?" in the online
list archives and decided to subscribe to the mailing-list and share
some notes I have.  Sorry for not being able to thread my reply in.

Anyway, I have recently added experimental support for GlusterFS to
Perceus, which is a provisioning system which supports stateful (with
disk) and stateless (diskless) configuration.  More information can be
found at the project website: http://www.perceus.org.

The way Perceus stateless provisioning works is that your client
PXE-boots and retrieves the Perceus kernel/initramfs, then it
retrieves the OS image via a specific transport (could be NFS, http,
or GlusterFS in this case) and then sets it up in RAM.  The system
then kexec into the new kernel and thus booting the system in RAM.

Perceus has a hybrid approach where the OS can reside in both RAM and
NFS (or GlusterFS).  This is to conserve on the memory consumption of
the OS image.

Some notes regarding the integration with GlusterFS:

- you will need the glusterfs binary, libglusterfs, libdl, libpthread,
libc and all the loadable libraries which your client volume file uses
in the initramfs
- right now I just build all the loadable libraries from auth,
scheduler, transport, xlators source directories and copy them to the
initramfs -- this is definitely bloated and I would like to figure out
a good way to cherry pick the ones I need
- since the Perceus kernel is fairly recent (2.6.32), you just need to
enable FUSE in the kernel and the user-space tools are not necessary
- if you are planning to do any operations on the GlusterFS mount
point which require direct mmap (system operations such as yum
requires it), you will need a recent kernel + new FUSE user tools

I haven't been able to do extensive testing but I know that it builds and
runs.

Traditionally, Perceus retrieves the OS image and "hybridizes" the OS
via NFS server on the master server.  If you have a lot of client
nodes, you will probably run into scalability problems and that's one
of the reasons why I started looking at GlusterFS.

Another approach one could take is just run NFS over GlusterFS, which
negates all the work in getting the libraries into initramfs and such.
 The Gluster folks are working on a native NFS server so it would be
interesting to see how the performance stacks up.

If you have any questions, please let me know -- hope this is informative.

Cheers,

Bernard