hi, I have a domU that dumps some data (a checkpoint) to a virtual device. Afterwards, I want to create a new domU based on that data, but because right now my domain bootstrapper does not have a disk driver, I first build a ramdisk which I give as an argument to my bootstrapper. This all works fine, but I have the problem that I have to reboot dom0 between each run, because apparently dom0 Linux does not notice that the contents on disk have changed, and keep giving me stale data from its buffer cache when I create the ramdisk. I have tested that this is actually what happens by grepping for a string in the block device (/dev/hda5), and the value of the string changes between reboots. Does anyone know of an ioctl or similar that I could use to make Linux forget about (not flush) buffer cache contents, or how to disable it? I have tried reading from the block dev with O_DIRECT but that does not help. Thanks, Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Does anyone know of an ioctl or similar that I could use to > make Linux forget about (not flush) buffer cache contents, or > how to disable it? > > I have tried reading from the block dev with O_DIRECT but > that does not help.Have you tried using raw devices? See "man raw". I''ve never used them, but I''d be interested to know whether this works. There''s an argument that we should be using them by default, particularly for people using using cluster filesystems -- caching stuff in dom0 could be very bad. Perhaps some of the ocfs2 guys that hang out on the list could comment? Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28 May 2005, at 19:49, Ian Pratt wrote:> Have you tried using raw devices? See "man raw". I''ve never used them, > but I''d be interested to know whether this works. There''s an argument > that we should be using them by default, particularly for people using > using cluster filesystems -- caching stuff in dom0 could be very bad.This isn''t a problem for domU virtual blkdevs: our backend driver doesn''t hook into the page cache, so unless the block device it sends requests to loops back to the page cache then we will not be affected by stale buffer cache contents. Jacob has given a domU direct access to a partition as a virtual blkdev. It has modified it, and the updates did go straight to disc. But if dom0 tries to read partition contents itself then it sees stale page cache data (because the backend completely circumvented the page cache). I''m surprised that O_DIRECT didn''t fix the problem -- I suppose it''s being ''too clever'' in this case and checking for potentially dirty pages in the cache before going direct to disc. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:> Have you tried using raw devices? See "man raw". I''ve never used them, > but I''d be interested to know whether this works. There''s an argument > that we should be using them by default, particularly for people using > using cluster filesystems -- caching stuff in dom0 could be very bad.It is my impression that in recent 2.6 kernels raw devices and O_DIRECT are the same thing. Actually, I was forced to use O_DIRECT inside my domU for the self-checkpointing, as otherwise the buffer cache would inflate the working set and create additional downtime. O_DIRECT works really well, and minus a ton of bugs I can now self-checkpoint to disk as well as self-migrate via the network. O_DIRECT works for writes. However, it seems it still tries to be a little too clever when reading from a blockdev. This is not a huge prob for me, just an annoyance that I have to reboot all the time to flush the cache, but I guess that for a Xen system in general it would be nice to have consistency of block device contents between domUs and dom0. thanks, Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel