zfs discuss - Aug 2009 - Books on File Systems and File System Programming

I saw this question on another mailing list, and I too would like to
know. And I have a couple questions of my own.

== Paraphrased from other list =Does anyone have any recommendations for books
on File Systems and/or
File Systems Programming?
== end =
I have some texts listed below, but are there books/journals/periodicals
that start from the kernel side of open(2), read(2), write(2), etc. and
progress to disk transactions?

With the advent of ZFS and other transaction based files systems it
seems to me that the line between File Systems and Databases are
beginning to blur ( If they haven''t already been doing so for some
time ).  Any pointers the likes of "X from here, Y from there, Z from
over yonder and squished together like Q" are also welcome.

(relevant) Books I have:
Understanding the Linux Kernel ( The chapters about ext2 and VFS )
Systems programming in the UNIX envirionment
File Structures: An OO approach using C++
Database System concepts (More about SQL and how to implement Joins )

Thanks in advance.

-- 
Louis-Fr?d?ric Feuillette <jebnor at gmail.com>

Going back a fair way the basics of FS design are in books like the  
following

Chapter 7 and 8 of The Design and Implementation of the 4.4BSD  
Operating System, Leffler, McKusick, Karels and Quarterman covers the  
FFS

Chapter 6 of The Magic Garden Explained, Goodheart and Cox addresses  
SYSV FS

Newer systems implement things differently but they try not to break  
the interfaces that these systems established too much.

Maurice Castro

On 14/08/2009, at 9:02 AM, Louis-Fr?d?ric Feuillette wrote:
> I saw this question on another mailing list, and I too would like to
> know. And I have a couple questions of my own.
>
> == Paraphrased from other list => Does anyone have any recommendations
for books on File Systems and/or
> File Systems Programming?
> == end =>

Louis-Fr?d?ric Feuillette <jebnor at gmail.com> wrote:
> I saw this question on another mailing list, and I too would like to
> know. And I have a couple questions of my own.
>
> == Paraphrased from other list => Does anyone have any recommendations
for books on File Systems and/or
> File Systems Programming?
> == end =
Are you interested in how to write a filesystem or in how to write the 
filesystem/kernel interface part?

If it is the latter, you will not get help from Linux related books.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Fri, 2009-08-14 at 12:34 +0200, Joerg Schilling
wrote:
> Louis-Fr?d?ric Feuillette <jebnor at gmail.com> wrote:
> 
> > I saw this question on another mailing list, and I too would like to
> > know. And I have a couple questions of my own.
> >
> > == Paraphrased from other list => > Does anyone have any
recommendations for books on File Systems and/or
> > File Systems Programming?
> > == end => 
> Are you interested in how to write a filesystem or in how to write the 
> filesystem/kernel interface part?
I am primarily interested in the theory of how to write a filesystem.
The kernel interface comes later when I dive into a OS specific details.

-- 
Louis-Fr?d?ric Feuillette <jebnor at gmail.com>

I did see this, Thanks.

On Fri, 2009-08-14 at 10:51 -0400, Christine Tran wrote:
> 
> 
> 2009/8/14 Louis-Fr?d?ric Feuillette <jebnor at gmail.com>
>         
>         
>         I am primarily interested in the theory of how to write a
>         filesystem.
>         The kernel interface comes later when I dive into a OS
>         specific details.
> 
> Have you seen this?  
> 
> http://www.letterp.com/~dbg/practical-file-system-design.pdf
> 
> I found this an excellent read.  The author begins by explaining
> what''s expected from an FS, he explains the design choices, some
> trade-offs, how the design interfaces with the actually hardware.  No
> specific OS detail, no API, no performance number.  Very solid
> fundamentals.
-- 
Louis-Fr?d?ric Feuillette <jebnor at gmail.com>

On Thu, Aug 13, 2009 at 05:02:46PM -0600, Louis-Fr?d?ric Feuillette
wrote:
> I saw this question on another mailing list, and I too would like to
> know. And I have a couple questions of my own.
> 
> == Paraphrased from other list => Does anyone have any recommendations
for books on File Systems and/or
> File Systems Programming?
> == end =
Going back ten years, but still a good tutorial:

   "Practical File System Design with the Be File System"
   by Dominic Giampaolo

   http://www.nobius.org/~dbg/practical-file-system-design.pdf

On 14-Aug-09, at 11:14 AM, Peter Schow wrote:
> On Thu, Aug 13, 2009 at 05:02:46PM -0600, Louis-Fr?d?ric Feuillette  
> wrote:
>> I saw this question on another mailing list, and I too would like to
>> know. And I have a couple questions of my own.
>>
>> == Paraphrased from other list =>> Does anyone have any
recommendations for books on File Systems and/or
>> File Systems Programming?
>> == end =>
> Going back ten years, but still a good tutorial:
>
>    "Practical File System Design with the Be File System"
>    by Dominic Giampaolo
>
>    http://www.nobius.org/~dbg/practical-file-system-design.pdf

Great cite (that I have not read) because Giampaolo is a noted expert  
on the second part of Louis-Frederic''s question, how filesystems are  
merging with databases. Namesys'' papers relating to Reiser4 are also  
worth reading in this respect.

--Toby
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Aug 14, 2009, at 11:14 AM, Peter Schow wrote:
> On Thu, Aug 13, 2009 at 05:02:46PM -0600, Louis-Fr?d?ric Feuillette  
> wrote:
>> I saw this question on another mailing list, and I too would like to
>> know. And I have a couple questions of my own.
>>
>> == Paraphrased from other list =>> Does anyone have any
recommendations for books on File Systems and/or
>> File Systems Programming?
>> == end =>
> Going back ten years, but still a good tutorial:
>
>   "Practical File System Design with the Be File System"
>   by Dominic Giampaolo
>
>   http://www.nobius.org/~dbg/practical-file-system-design.pdf
I think he''s still at apple now working on spotlight .. his fs-kit is  
good study too:
http://www.nobius.org/~dbg/fs-kit-0.4.tgz

for understanding the vnode/vfs interface - you might want to take a  
look at:
- Solaris Internals (2nd edition) - chapter 14
- Zadok''s FiST paper:
http://www.fsl.cs.sunysb.edu/docs/zadok-thesis-proposal/

UFS:
- Solaris Internals (2nd edition) - chapter 15
HFS+:
- Amit Singh''s Mac OS X Internals chapter 11 (see http://osxbook.com/)

then opensolaris src of course for:
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/
http://opensolaris.org/os/community/zfs/source/
http://opensolaris.org/os/project/samqfs/sourcecode/
http://opensolaris.org/os/project/ext3/

There aren''t many good books on file system design.

The "VAX/VMS Internals and Data Structures" book by Goldenberg covers
a fair amount of the RMS file system design along with its rationale. There is
also a "VMS File System Internals" book which I haven''t yet
read. Apple''s early Inside Macintosh volumes include a good description
of MFS and HFS (which became HFS+) and the rationale behind HFS.

There are many good academic papers. I would suggest starting with
McKusick''s exposition on UFS and Selzer et al''s work on
log-structured file systems, if you''re interested in traditional
UNIX-style (stream-of-bytes) file systems. If you''re willing to go a
little deeper, IBM has published a variety of papers related to file systems
through the years, starting in the 1960s or so.

The database literature is a rich source of ideas for file systems. In
particular, you should understand a bit about journaling.
-- 
This message posted from opensolaris.org

On Fri, Aug 14, 2009 at 12:34:50PM +0200, Joerg Schilling
wrote:
> Louis-Fr?d?ric Feuillette <jebnor at gmail.com> wrote:
> 
> > I saw this question on another mailing list, and I too would like to
> > know. And I have a couple questions of my own.
> >
> > == Paraphrased from other list => > Does anyone have any
recommendations for books on File Systems and/or
> > File Systems Programming?
> > == end =
"Solaris Internals" has a chapter on Filesystems which talks about the
VNODE/VFS
layer and how they interface with different filesystems. You might find this
useful as well.

Thanks and regards,
Sanjeev.
> 
> Are you interested in how to write a filesystem or in how to write the 
> filesystem/kernel interface part?
> 
> If it is the latter, you will not get help from Linux related books.
> 
> J?rg
> 
> -- 
>  EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353
Berlin
>        js at cs.tu-berlin.de                (uni)  
>        joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
>  URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
----------------
Sanjeev Bagewadi
Solaris RPE 
Bangalore, India