search for: dontneed

...gt; Maybe a filter can handle alignment? > > > implementation, but my idea was that it might be an alternate > > implementation of cache=none. But if we thought we might use O_DIRECT > > as a separate mode, then maybe we should rename cache=none. > > cache=advise? cache=dontneed? I can't think of a good name! > > > > Yes, don't call it none if you use the cache. > > How about advise=? > > I would keep cache semantics similar to qemu. qemu uses cache=none as a synonym for O_DIRECT, but AFAIK it has nothing that tries to use posix_fadvise...

...home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern. It looks like a patch[1] was written (although never merged, it seems) incorporating fadvise support, but I found its implementation rather odd, using mincore() and FADV_DONTNEED to kick out only regions brought in by rsync. It seemed to me the simpler and more appropriate solution would be to simply flag every touched file with FADV_NOREUSE and let the kernel manage automatically expelling used pages. After looking deeper into the kernel implementation[2] of fadvise() the...

...at writethrough is the same as cache=none. In > qemu it only applies to writes, and says nothing about reads AFAIK. > Where do you get the information that writethrough (in qemu, or > anywhere else) marks pages as not needed if they are only read and > never written? qemu does not use DONTNEED, but otherwise current cache=none is what qemu and LIO call writethrough. Using a cache and evicting the pages does not make much sense, since when reading the pages are not in the cache. So why do you use the cache? I think we are mixing different things in the current code. Flushing after every...

...ng > O_DIRECT. I'm not sure if or even how we could ever do a robust O_DIRECT implementation, but my idea was that it might be an alternate implementation of cache=none. But if we thought we might use O_DIRECT as a separate mode, then maybe we should rename cache=none. cache=advise? cache=dontneed? I can't think of a good name! > >@@ -355,6 +428,17 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, > > { > > struct handle *h = handle; > >+#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_DONTNEED) > >+ uint32...

...ndle alignment? > > > > > implementation, but my idea was that it might be an alternate > > > implementation of cache=none. But if we thought we might use O_DIRECT > > > as a separate mode, then maybe we should rename cache=none. > > > cache=advise? cache=dontneed? I can't think of a good name! > > > > > > > Yes, don't call it none if you use the cache. > > > > How about advise=? > > > > I would keep cache semantics similar to qemu. > > qemu uses cache=none as a synonym for O_DIRECT, but AFAIK it ha...

...> We tried to use fadvice but it did not help. The only way to avoid such issues > is with O_SYNC or O_DIRECT. O_SYNC is much slower but this is the path > we took for now in this flow. I'm interested in more background about this, because while it is true that O_DIRECT and POSIX_FADV_DONTNEED are not exactly equivalent, I think I've shown here that DONTNEED can be used to avoid polluting the page cache. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you e...

On Sun, Jul 30, 2017 at 05:59:17AM +0000, Wang, Wei W wrote: > On Sunday, July 30, 2017 12:23 PM, Michael S. Tsirkin wrote: > > On Sat, Jul 29, 2017 at 08:47:08PM +0800, Wei Wang wrote: > > > On 07/29/2017 07:08 AM, Michael S. Tsirkin wrote: > > > > On Thu, Jul 27, 2017 at 10:50:11AM +0800, Wei Wang wrote: > > > > > > > > OK I thought this

On Sun, Jul 30, 2017 at 05:59:17AM +0000, Wang, Wei W wrote: > On Sunday, July 30, 2017 12:23 PM, Michael S. Tsirkin wrote: > > On Sat, Jul 29, 2017 at 08:47:08PM +0800, Wei Wang wrote: > > > On 07/29/2017 07:08 AM, Michael S. Tsirkin wrote: > > > > On Thu, Jul 27, 2017 at 10:50:11AM +0800, Wei Wang wrote: > > > > > > > > OK I thought this

...*handle, void *buf, uint32_t count, uint64_t offset, #ifdef HAVE_POSIX_FADVISE /* On Linux this will evict the pages we just read from the page cache. */ - if (cache_mode == cache_none) + if (cache_mode == cache_writethrough) posix_fadvise (h->fd, orig_offset, orig_count, POSIX_FADV_DONTNEED); #endif @@ -441,11 +441,11 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, uint32_t orig_count = count; uint64_t orig_offset = offset; - /* If cache=none we want to force pages we have just written to the - * file to be flushed to disk so we can immedi...

...ce for this, posix_fadvise, has existed for some time, but there has been no useable implementation in any of the major operating systems. Attempts have been made in the past[1] to use the fadvise interface, but kernel limitations have made this quite messy. In particular, the kernel supports FADV_DONTNEED as a single-shot hint; i.e. if the page is clean when the hint is given it will be freed, otherwise the hint is ignored. For this reason it is necessary to fdatasync() against dirtied pages before giving the hint. This, however, requires that rsync do some accounting, calling fdatasync() and fadvis...

...The only way to avoid such issues > > > is with O_SYNC or O_DIRECT. O_SYNC is much slower but this is the path > > > we took for now in this flow. > > > > I'm interested in more background about this, because while it is true > > that O_DIRECT and POSIX_FADV_DONTNEED are not exactly equivalent, I > > think I've shown here that DONTNEED can be used to avoid polluting the > > page cache. > > This fixes the minor issue of polluting the page cache, but it does not help > to avoid stale data in the cache, or unconfrolled flushes. > >...

...r and require aligned requests. Maybe a filter can handle alignment? implementation, but my idea was that it might be an alternate > implementation of cache=none. But if we thought we might use O_DIRECT > as a separate mode, then maybe we should rename cache=none. > cache=advise? cache=dontneed? I can't think of a good name! > Yes, don't call it none if you use the cache. How about advise=? I would keep cache semantics similar to qemu. > > >@@ -355,6 +428,17 @@ file_pwrite (void *handle, const void *buf, > uint32_t count, uint64_t offset, > > > { &g...

On Thu, Jul 27, 2017 at 10:50:11AM +0800, Wei Wang wrote: > > > > OK I thought this over. While we might need these new APIs in > > > > the future, I think that at the moment, there's a way to implement > > > > this feature that is significantly simpler. Just add each s/g > > > > as a separate input buffer. > > > > > > Should

On Thu, Jul 27, 2017 at 10:50:11AM +0800, Wei Wang wrote: > > > > OK I thought this over. While we might need these new APIs in > > > > the future, I think that at the moment, there's a way to implement > > > > this feature that is significantly simpler. Just add each s/g > > > > as a separate input buffer. > > > > > > Should

...nt deflate) > { > #if defined(__linux__) > if (!qemu_balloon_is_inhibited() && (!kvm_enabled() || > kvm_has_sync_mmu())) { > qemu_madvise(addr, TARGET_PAGE_SIZE, > deflate ? QEMU_MADV_WILLNEED : QEMU_MADV_DONTNEED); > } > #endif > } > > The virtio-balloon code only does stuff through ballon_page, > and an madvise DONTNEED should fail if you try and do it on > a size smaller than the host page size. So does ballooning work on > Power/ARM? > > Am I misunderstanding this? I...

...> > > > Best, > > Wei > > That's a hypervisor implementation detail. From guest point of view, > discarding contents can not be distinguished from writing old contents. > Besides, ignoring the free page tricks, consider regular ballooning. We map page with DONTNEED then back with WILLNEED. Result is getting a zero page. So at least one of deflate/inflate should be input. I'd say both for symmetry. -- MST

...vice but it did not help. The only way to avoid such issues > > is with O_SYNC or O_DIRECT. O_SYNC is much slower but this is the path > > we took for now in this flow. > > I'm interested in more background about this, because while it is true > that O_DIRECT and POSIX_FADV_DONTNEED are not exactly equivalent, I > think I've shown here that DONTNEED can be used to avoid polluting the > page cache. This fixes the minor issue of polluting the page cache, but it does not help to avoid stale data in the cache, or unconfrolled flushes. The bug I mentioned is: https://bu...

...nt deflate) > { > #if defined(__linux__) > if (!qemu_balloon_is_inhibited() && (!kvm_enabled() || > kvm_has_sync_mmu())) { > qemu_madvise(addr, TARGET_PAGE_SIZE, > deflate ? QEMU_MADV_WILLNEED : QEMU_MADV_DONTNEED); > } > #endif > } > > The virtio-balloon code only does stuff through ballon_page, > and an madvise DONTNEED should fail if you try and do it on > a size smaller than the host page size. So does ballooning work on > Power/ARM? > > Am I misunderstanding this? I...

...> Attached is a patch to unstable that stops save/restore/dump from > hosing > Dom0 when dealing with large domains - I''m actually resubmitting the > dump patch I previously submitted in addition as it hasn''t been > incorporated yet; this is based on using fadvise64(DONTNEED) to throw > the page cache away once it has been written to disk -- with this in > place, memory usage does go up somewhat but then immediately drops > again > when the action is done and this change, in conjunction with setting > the > vm.dirty_ratio sysctl parameter seems to giv...

...evice to send commands can be an inbuf, I think. > >> I think it may only >> need to read out the info(base,size). > And then do what? For the free pages, the info will be used to clear the corresponding "1" in the dirty bitmap. For balloon pages, they will be made DONTNEED and given to other host processes to use (the device won't write them, so no need to set "write" when virtqueue_map_desc() in the device). > >> I think it should be like this: >> the cmd hdr buffer: input, because the hypervisor would write it to >> send a cmd...