Xen users - Nov 2005 - cache for partition based VBD?

For partition backed VBDs, is there still an effective cache in dom0?  The Linux
buffer cache will affect file based VBDs, but does the partition backed VBD get
around it? 

I think that the Linux architecture puts the buffer cache in front of all block
device access, not at the VFS layer.  So does that answer the question?  The
dom0 disk backend driver sits "above" the buffer cache?  (sorry for my
crude
understanding)

Thanks,
Tim 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Tim Freeman
> Sent: 29 November 2005 17:12
> To: xen-users@lists.xensource.com
> Cc: xen-devel@lists.xensource.com
> Subject: [Xen-users] cache for partition based VBD?
> 
> For partition backed VBDs, is there still an effective cache 
> in dom0?  The Linux buffer cache will affect file based VBDs, 
> but does the partition backed VBD get around it? 
Let''s first start with the facts:  don''t know the answer to
your
question 

Now for some speculation: 
The block device cache would be efficient in the sense that it''s useful
for every type of file-system, and it''s not going to have to be clever
about what sort of accesses are frequent, not so frequent etc. This
gives benefit to the "cache at the device level".

On the other hand, a file-system level cache would also make sense.
Caching directory structurs or file-allocation-table [not necessarily in
the Microsoft sense] would obviously help a whole lot more than caching
some block from the middle of a file that isn''t ever going to be read
again. This motivates having a targetted caching mechanism in the
file-system (VFS) layer. 

A combination would of course make even more sense - cache important
data structures in the File-system layer, and cache blocks that have
frequent/recent accesses in the block-device. 
> 
> I think that the Linux architecture puts the buffer cache in 
> front of all block device access, not at the VFS layer.  So 
> does that answer the question?  The dom0 disk backend driver 
> sits "above" the buffer cache?  (sorry for my crude
> understanding)
I would say that this is correct - the Dom0 disk driver would have a
cache in itself, and the DomU would also have a caching layer. But I''m
not SURE about this at all. 

If someone can make further comments, it would be great. 


--
Mats
> 
> Thanks,
> Tim 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for responding,

On Tue, 29 Nov 2005 18:56:32 +0100
"Petersson, Mats" <mats.petersson@amd.com> wrote:
> > -----Original Message-----
> > From: xen-users-bounces@lists.xensource.com 
> > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> > Tim Freeman
> > Sent: 29 November 2005 17:12
> > To: xen-users@lists.xensource.com
> > Cc: xen-devel@lists.xensource.com
> > Subject: [Xen-users] cache for partition based VBD?
> > 
> > For partition backed VBDs, is there still an effective cache 
> > in dom0?  The Linux buffer cache will affect file based VBDs, 
> > but does the partition backed VBD get around it? 
> 
> Let''s first start with the facts:  don''t know the answer
to your
> question 
> 
> Now for some speculation: 
> The block device cache would be efficient in the sense that it''s
useful
> for every type of file-system, and it''s not going to have to be
clever
> about what sort of accesses are frequent, not so frequent etc. This
> gives benefit to the "cache at the device level".
I''m looking into this a little, reading chapter 15 of Robert
Love''s ''Linux
Kernel Development,'' here is a good bit: 

"Individual disk blocks can also tie into the page cache, by way of block
I/O
buffers [...] this caching is often referred to as the "buffer cache,"
although
in reality it is not a separate cache and is part of the page cache."

(the difference here is that the page is larger than the disk block and this I/O
buffer actually describes the mapping of block onto a page... also, he notes
back in Linux 2.2 they used to be separate caches)

He explans there is a temporal locality algorithm *by page* not with any
preference for filesystem structures.
> On the other hand, a file-system level cache would also make sense.
> Caching directory structurs or file-allocation-table [not necessarily in
> the Microsoft sense] would obviously help a whole lot more than caching
> some block from the middle of a file that isn''t ever going to be
read
> again. This motivates having a targetted caching mechanism in the
> file-system (VFS) layer. 
This also happening somewhat, found this statement:

"The Linux VFS layer attempts to cache inodes in memory; this cache is
maintained by a linked list of dentry structures. Each dentry contains a
reference to the inode, and the name of the file. "

http://www.kevinboone.com/linux_kernel_file_1.html

Writing+filesystem-cache seems more complicated, but I''m not worried
about this
right now, this talk seems to have a lot of good info on that for ext3:

http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html

I am continuing to look for Xen specific device driver + buffer cache (w/
partition based VBD) evidence besides our experiment.  We''re going to
run a new
test w/ this setup as well, running sar or vmstat in dom0, but I''d
still like to
have code/documentation/xen-developer evidence if possible.  Also looking for
ways of clearing the buffer cache entirely (not just flushing writes). 

Thanks,
Tim



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> For partition backed VBDs, is there still an effective cache in dom0?  The
Linux
> buffer cache will affect file based VBDs, but does the partition backed VBD
get
> around it?
Filesystem caching shouldn''t happen in dom0 for the domU''s and
I doubt you''d want
it to.  Caching *will* happen at the domU level and that''s where you
want it --
close to your applications.

Besides, if you can run a dom0 in like 128MB of memory to devote as much as
possible to your domU''s, how much caching could you do?

John



-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 29 Nov 2005 12:39:25 -0600
Tim Freeman <tfreeman@mcs.anl.gov> wrote:
[...]
> 
> > On the other hand, a file-system level cache would also make sense.
> > Caching directory structurs or file-allocation-table [not necessarily
in
> > the Microsoft sense] would obviously help a whole lot more than
caching
> > some block from the middle of a file that isn''t ever going to
be read
> > again. This motivates having a targetted caching mechanism in the
> > file-system (VFS) layer. 
> 
> This also happening somewhat, 
Here is a good link, too:
http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=278

Tim 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 29 Nov 2005 18:56:32 +0100
"Petersson, Mats" <mats.petersson@amd.com> wrote:
[...]
> > I think that the Linux architecture puts the buffer cache in 
> > front of all block device access, not at the VFS layer.  So 
> > does that answer the question?  The dom0 disk backend driver 
> > sits "above" the buffer cache?  (sorry for my crude
> > understanding)
> 
> I would say that this is correct - the Dom0 disk driver would have a
> cache in itself, and the DomU would also have a caching layer. But
I''m
> not SURE about this at all. 
> 
> If someone can make further comments, it would be great. 
I found this post that directly addresses the point:

http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01167.html

(now the experimental data makes less sense to me)

Tim 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 29 Nov 2005 17:44:34 -0600
Tim Freeman <tfreeman@mcs.anl.gov> wrote:
> On Tue, 29 Nov 2005 18:56:32 +0100
> "Petersson, Mats" <mats.petersson@amd.com> wrote:
> [...]
> > > I think that the Linux architecture puts the buffer cache in 
> > > front of all block device access, not at the VFS layer.  So 
> > > does that answer the question?  The dom0 disk backend driver 
> > > sits "above" the buffer cache?  (sorry for my crude
> > > understanding)
> > 
> > I would say that this is correct - the Dom0 disk driver would have a
> > cache in itself, and the DomU would also have a caching layer. But
I''m
> > not SURE about this at all. 
> > 
> > If someone can make further comments, it would be great. 
> 
> I found this post that directly addresses the point:
> 
> http://lists.xensource.com/archives/html/xen-devel/2005-05/msg01167.html
> 
> (now the experimental data makes less sense to me)
> 
Regarding that thread (and one of my own questions), I did find this patch for
clearing the page cache entirely (not just flushing pending writes)...

http://lwn.net/Articles/123343/

Tim 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> The block device cache would be efficient in the sense that it''s
useful
> for every type of file-system, and it''s not going to have to be
clever
> about what sort of accesses are frequent, not so frequent etc. This
> gives benefit to the "cache at the device level".
>
> On the other hand, a file-system level cache would also make sense.
> Caching directory structurs or file-allocation-table [not necessarily in
> the Microsoft sense] would obviously help a whole lot more than caching
> some block from the middle of a file that isn''t ever going to be
read
> again. This motivates having a targetted caching mechanism in the
> file-system (VFS) layer.
Linux does caching at the level of blocks on a device and of contiguous pages 
within a file by stuffing them both in the page cache.  Disk-based 
filesystems are able to use a cached interface to block storage (when 
appropriate) to access their on disk data structures.  Filesystems use the 
block-level interface to the page cache to read their own metadata but store 
file contents into the page cache in a file-oriented manner.

The VFS manipulates and caches various meta-stuff like inode data generically 
(although it has to get this from the filesystem, which in term retrieves 
that data from the block layer).

The filesystem is also free to build its own data structures that cache useful 
information.  For instance, filesystems may choose to store metadata in 
privately allocated structures or (sometimes) in parts of the page cache.  
There are private pointers in the file and inode structures to facilitate 
this.
> A combination would of course make even more sense - cache important
> data structures in the File-system layer, and cache blocks that have
> frequent/recent accesses in the block-device.
Yup.
> > I think that the Linux architecture puts the buffer cache in
> > front of all block device access, not at the VFS layer.  So
> > does that answer the question?  The dom0 disk backend driver
> > sits "above" the buffer cache?  (sorry for my crude
> > understanding)
>
> I would say that this is correct - the Dom0 disk driver would have a
> cache in itself, and the DomU would also have a caching layer. But
I''m
> not SURE about this at all.
>
> If someone can make further comments, it would be great.
I''m pretty sure our blkback driver uses a non-caching interface to
storage: it
requests stuff directly from the block layer, rather than from the 
block-level interface to the page cache.  The only way caching gets to happen 
in dom0 is if you''re using loop devices in which case direct block
access is
looped back into the VFS (and therefore the page cache inevitable gets 
involved).

Sounds sane?,
Mark

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> For partition backed VBDs, is there still an effective cache in dom0?  The
> Linux buffer cache will affect file based VBDs, but does the partition
> backed VBD get around it?
AFAIK the blkback driver uses uncached interfaces to storage.  Caching only 
happens if you''re using loop devices in dom0, because that indirects
through
the filesystem, which will being use the page cache.

You don''t want caching in dom0, really: the domU will manage its own
caches
(and do so more effectively than dom0 could!) anyhow, so there''s no
point
duplicating the effort.
> I think that the Linux architecture puts the buffer cache in front of all
> block device access, not at the VFS layer.  So does that answer the
> question?  The dom0 disk backend driver sits "above" the buffer
cache?
> (sorry for my crude understanding)
AFAIK, it sits *alongside* the page cache layer, as another client of the 
block system proper.

Cheers,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users