search for: madvise

Hi, I'm using py-rrdtool 0.2.1 with rrdtool 1.3.0 under 7.0-STABLE, and there's a couple of things about this new version of rrdtool that hurt performance under FreeBSD, but apparently help on whatever they tested on. For every update, the database file is opened, mapped into memory, madvise() is called, contents are modified, msync() is called, and the file is unmapped and closed: 65074 python 0.000009 CALL open(0x831a1b4,O_RDWR,<unused>0) 65074 python 0.000013 NAMI "rg2.rrd" 65074 python 0.000024 RET open 7 65074 python 0.000007 CALL fstat(0x...

(I think this has been mentioned before.) Do these hurt?: May 2 13:41:30 sysname dovecot: POP3(xxxx): madvise() failed with file /users/xx/xxxx/Maildir/dovecot.index.log: Operation not supported I think not, since I've been seeing them forever. But they do muddy the logs. This is on an old BSD/OS server, where the man page for madvise ends with: BUGS This call is not yet supported. Th...

...e it tends not to clump? > I didn't do a comprehensive test. But I found most of the time spending on allocating the pages and sending the PFNs to guest, I don't know that's the most time consuming operation, allocating the pages or sending the PFNs. > * is it due to the madvise on the host? > If we were using the normal balloon messages, then we > could, during migration, just route those to the migration > code rather than bothering with the madvise. > If they're clumping together we could just turn that into >...

...e it tends not to clump? > I didn't do a comprehensive test. But I found most of the time spending on allocating the pages and sending the PFNs to guest, I don't know that's the most time consuming operation, allocating the pages or sending the PFNs. > * is it due to the madvise on the host? > If we were using the normal balloon messages, then we > could, during migration, just route those to the migration > code rather than bothering with the madvise. > If they're clumping together we could just turn that into >...

...ent input: I\"...,82) = 82 (0x52) getpid() = 3446 (0xd76) __sysctl(0x7fffffffe8d0,0x4,0x0,0x0,0x42034000,0x20) = 0 (0x0) kevent(10,{ 11,EVFILT_READ,EV_DELETE,0x0,0x0,0x0 },1,0x0,0,0x0) = 0 (0x0) close(11) = 0 (0x0) madvise(0x4207e000,0x2000,0x5) = 0 (0x0) madvise(0x42094000,0x3000,0x5) = 0 (0x0) madvise(0x42098000,0x3000,0x5) = 0 (0x0) madvise(0x420a0000,0x3000,0x5) = 0 (0x0) madvise(0x420a6000,0x5000,0x5) = 0 (0x0) madvise(0x42...

...solving my last issue. I got dovecot up and running on one of my machines, and it does exactly what I want. Now I'm trying to install dovecot on an older box (glibc 2.0.7 Linux kernel 2.0.36), and encountering some problems: -- firstly autoconf/automake does not seem to correctly deal with the madvise function on this system. It detects that we don't have madvise, yet it's prototyped in /usr/include/sys/mman.h. This causes the compilation to fail when dovecot tries to redefine this function. -- I attempted to solve this problem by renaming the function dovecot prototypes to dc_madvise a...

> > Could provide more information on how to use virtio-serial to exchange > data? Thread , Wiki or code are all OK. > > I have not find some useful information yet. > > See this commit in the Linux sources: > > 108fc82596e3b66b819df9d28c1ebbc9ab5de14c > > that adds a way to send guest trace data over to the host. I think that's the > most relevant to

> > Could provide more information on how to use virtio-serial to exchange > data? Thread , Wiki or code are all OK. > > I have not find some useful information yet. > > See this commit in the Linux sources: > > 108fc82596e3b66b819df9d28c1ebbc9ab5de14c > > that adds a way to send guest trace data over to the host. I think that's the > most relevant to

On 07/31/2017 02:55 PM, Michal Hocko wrote: > On Mon 31-07-17 12:13:33, Wei Wang wrote: >> Ballooned pages will be marked as MADV_DONTNEED by the hypervisor and >> shouldn't be given to the host ksmd to scan. > Could you point me where this MADV_DONTNEED is done, please? Sure. It's done in the hypervisor when the balloon pages are received. Please see line 40 at

On 07/31/2017 02:55 PM, Michal Hocko wrote: > On Mon 31-07-17 12:13:33, Wei Wang wrote: >> Ballooned pages will be marked as MADV_DONTNEED by the hypervisor and >> shouldn't be given to the host ksmd to scan. > Could you point me where this MADV_DONTNEED is done, please? Sure. It's done in the hypervisor when the balloon pages are received. Please see line 40 at

...0x20ee7fd1,0x21e72000,0x0,0xa7,0x4e,0xc) = 1 (0x1) open("/usr/share/zoneinfo/posixrules",O_RDONLY,056) = 12 (0xc) fstat(12,{ mode=-r--r--r-- ,inode=545,size=3519,blksize=4096 }) = 0 (0x0) read(12,"TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0"...,41448) = 3519 (0xdbf) close(12) = 0 (0x0) madvise(0x21eba000,0xb000,0x5,0x2b9,0x17,0x1) = 0 (0x0) madvise(0x21e72000,0x5000,0x5,0x271,0x17,0x1) = 0 (0x0) madvise(0x21e63000,0xc000,0x5,0x262,0x7fffffffcbf0,0x7fffffffcbf0) = 0 (0x0) stat("/mailstore/domains/test.com/test/mdbox/mailboxes/Spam/dbox-Mails/dovecot.index.log",{ mode=-rw------...

...> > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. > > Do you have some statistics/some rough feeling how many consecutive bits are > usually set in the bitmaps? Is it really just purely random or is there some > granularity that is usually consecutive? > I did so...

...> > One is the change for speeding up the inflating & deflating process, > > the main idea of this optimization is to use bitmap to send the page > > information to host instead of the PFNs, to reduce the overhead of > > virtio data transmission, address translation and madvise(). This can > > help to improve the performance by about 85%. > > Do you have some statistics/some rough feeling how many consecutive bits are > usually set in the bitmaps? Is it really just purely random or is there some > granularity that is usually consecutive? > I did so...

Nothing, the same error Oct 24 10:11:16 server dovecot: imap(email at example.com)<82402><Nhu9MPR4KuaASO/q>: Error: kevent(-1) for notify remove failed: Bad file descriptor Oct 24 10:11:16 server dovecot: imap(email at example.com)<82402><Nhu9MPR4KuaASO/q>: Error: close(-1) for notify remove failed: Bad file descriptor Oct 24 10:11:16 server dovecot: imap-hibernate:

...g > on allocating the pages and sending the PFNs to guest, I don't know that's > the most time consuming operation, allocating the pages or sending the PFNs. It might be a good idea to analyse it a bit more to convince people where the problem is. > > * is it due to the madvise on the host? > > If we were using the normal balloon messages, then we > > could, during migration, just route those to the migration > > code rather than bothering with the madvise. > > If they're clumping together we could just turn...

...change for speeding up the inflating & deflating process, >>>> the main idea of this optimization is to use bitmap to send the page >>>> information to host instead of the PFNs, to reduce the overhead of >>>> virtio data transmission, address translation and madvise(). This can >>>> help to improve the performance by about 85%. >>> >>> Do you have some statistics/some rough feeling how many consecutive >>> bits are >>> usually set in the bitmaps? Is it really just purely random or is >>> there some >&...

...change for speeding up the inflating & deflating process, >>>> the main idea of this optimization is to use bitmap to send the page >>>> information to host instead of the PFNs, to reduce the overhead of >>>> virtio data transmission, address translation and madvise(). This can >>>> help to improve the performance by about 85%. >>> >>> Do you have some statistics/some rough feeling how many consecutive >>> bits are >>> usually set in the bitmaps? Is it really just purely random or is >>> there some >&...

...t;> will stay - whereby the unwritten parts might either be from the old, or >> from the zero page." - a QEMU BUG. > > How is this a bug? This is the behavior you would see with the current > balloon driver. When you put a page into a balloon it has the option > to either madvise it away, defer it, or just skip it because he > balloon is disabled. Is the "zero page" a typo? If it was > uninitialized data that would be a bug, but I don't see how a zero > page or the old data would be a bug. Sorry, I meant if this was the original design idea of hintin...

...erred to the > host one by one. Here is the breakdown of the time in percentage > spent on each step of the balloon inflating process (inflating > 7GB of an 8GB idle guest). > > 1) allocating pages (6.5%) > 2) sending PFNs to host (68.3%) > 3) address translation (6.1%) > 4) madvise (19%) > > It takes about 4126ms for the inflating process to complete. > The above profiling shows that the bottlenecks are stage 2) > and stage 4). > > This patch optimizes step 2) by transferring pages to the host in > sgs. An sg describes a chunk of guest physically continuo...

...erred to the > host one by one. Here is the breakdown of the time in percentage > spent on each step of the balloon inflating process (inflating > 7GB of an 8GB idle guest). > > 1) allocating pages (6.5%) > 2) sending PFNs to host (68.3%) > 3) address translation (6.1%) > 4) madvise (19%) > > It takes about 4126ms for the inflating process to complete. > The above profiling shows that the bottlenecks are stage 2) > and stage 4). > > This patch optimizes step 2) by transferring pages to the host in > sgs. An sg describes a chunk of guest physically continuo...