All- Per this page https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the above example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). Typically in the rsync daemon log we see something like: receiving file list sent 17 bytes received 24 bytes total size 0 but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs APIs like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week. Unfortunately there are no actual API examples online, no recipes on the libguestfs.org site, etc. Does anyone know where there are some actual libguestfs_rsync_out() working examples? Thanks. -Jeff
Richard W.M. Jones
2015-Feb-08 11:05 UTC
Re: [Libguestfs] getting guestfs_rsync_out to work
On Sat, Feb 07, 2015 at 05:19:42PM -0600, Jeff Brower wrote:> All- > > Per this page > > https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ > > We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: > > guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) > > surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the above > example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). > Typically in the rsync daemon log we see something like: > > receiving file list > sent 17 bytes received 24 bytes total size 0 > > but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs APIs > like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week.I think it's really hard to diagnose problems without turning on debugging in libguestfs: http://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-the-api> Unfortunately there are no actual API examples online, no > recipes on the libguestfs.org site, etc. Does anyone know > where there are some actual libguestfs_rsync_out() working examples?The only example I can point to is this test: https://github.com/libguestfs/libguestfs/blob/master/tests/rsync/test-rsync.sh Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Richard->> Per this page >> >> https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ >> >> We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: >> >> guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) >> >> surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the >> above >> example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). >> Typically in the rsync daemon log we see something like: >> >> receiving file list >> sent 17 bytes received 24 bytes total size 0 >> >> but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs >> APIs >> like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week. > > I think it's really hard to diagnose problems without turning on > debugging in libguestfs: > > http://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-the-api > >> Unfortunately there are no actual API examples online, no >> recipes on the libguestfs.org site, etc. Does anyone know >> where there are some actual libguestfs_rsync_out() working examples? > > The only example I can point to is this test: > > https://github.com/libguestfs/libguestfs/blob/master/tests/rsync/test-rsync.shThanks very much. In our example, the libguestfs API syntax required turns out to be: guestfs_rsync_out(g, "/Win7Folder", "rsync://root@10.0.1.90:2999/backup", GUESTFS_RSYNC_OUT_ARCHIVE, 1, -1) This produces a trace result: libguestfs: trace: rsync_out "/Win7Folder" "rsync://root@10.0.1.90:2999/backup" "archive:true" Also, to avoid chgrp and other permission related messages, we had to add: uid = 0 gid = 0 in our rsyncd.conf file (we are running rsync daemaon as root). Now we are testing guestfs_rsync_out() in a loop that runs every 1 sec. We hope to achieve continuous synchronization with a live Win7 guest. As I mentioned before, we're only reading from the Win7 guest, and not a lot of data is involved, maybe a few files up to 50 MByte. These files are not changing size, just contents. -Jeff
Richard->>> Per this page >>> >>> https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/ >>> >>> We are trying to get the libguestfs API guestfs_rsync_out to work. Weve tried combinations similar to: >>> >>> guestfs_rsync_out(g, /HostShared, rsync://root@10.0.1.90:2999/backup/ -av, -1) >>> >>> surrounding it with APIs as in the recipe examples, for example replacing libguestfs_cat(), with no luck. In the >>> above >>> example the "backup" rsync module name matches the script on Richard's wordpress.com blog page (above link). >>> Typically in the rsync daemon log we see something like: >>> >>> receiving file list >>> sent 17 bytes received 24 bytes total size 0 >>> >>> but no data is transferred. We are successful using rsync from the Linux command line, and we can get libguestfs >>> APIs >>> like guestfs_cat() to work fine. But weve had zero luck with rsync_out over the last week. >> >> I think it's really hard to diagnose problems without turning on >> debugging in libguestfs: >> >> http://libguestfs.org/guestfs-faq.1.html#how-do-i-debug-when-using-the-api >> >>> Unfortunately there are no actual API examples online, no >>> recipes on the libguestfs.org site, etc. Does anyone know >>> where there are some actual libguestfs_rsync_out() working examples? >> >> The only example I can point to is this test: >> >> https://github.com/libguestfs/libguestfs/blob/master/tests/rsync/test-rsync.sh > > Thanks very much. In our example, the libguestfs API syntax required turns out to be: > > guestfs_rsync_out(g, "/Win7Folder", "rsync://root@10.0.1.90:2999/backup", GUESTFS_RSYNC_OUT_ARCHIVE, 1, -1) > > This produces a trace result: > > libguestfs: trace: rsync_out "/Win7Folder" "rsync://root@10.0.1.90:2999/backup" "archive:true" > > Also, to avoid chgrp and other permission related messages, we had to add: > > uid = 0 > gid = 0 > > in our rsyncd.conf file (we are running rsync daemaon as root). > > Now we are testing guestfs_rsync_out() in a loop that runs every 1 sec. We hope to achieve continuous synchronization > with a live Win7 guest. As I mentioned before, we're only reading from the Win7 guest, and not a lot of data is > involved, maybe a few files up to 50 MByte. These files are not changing size, just contents.[Update] With continuous loop testing, what we found is that we have to shut down and re-launch the image handle to see changes on the Win7 live guest. Unfortunately image re-launch takes time, 3-5 sec (the image size is 50 GByte). I'm assuming this is because libguestfs makes an internal copy of filesystem, and works from that, and doesn't "refresh" this internal copy until re-launching the image handle. Could we create a second partition on the guest that is much smaller, say 50 MByte, attach the image to that, and thus reduce the shutdown and re-launch time into the msec range? Is there another real-time method, not using rsync? As a note, for Linux guest images, we use virtFs, but it seems that WinX doesn't support the 9p filesystem drivers required for virtFs. Thanks. -Jeff
Richard W.M. Jones
2015-Feb-09 08:59 UTC
Re: [Libguestfs] getting guestfs_rsync_out to work
On Sun, Feb 08, 2015 at 12:11:37PM -0600, Jeff Brower wrote:> With continuous loop testing, what we found is that we have to shut > down and re-launch the image handle to see changes on the Win7 live > guest. Unfortunately image re-launch takes time, 3-5 sec (the image > size is 50 GByte). I'm assuming this is because libguestfs makes an > internal copy of filesystem, and works from that, and doesn't > "refresh" this internal copy until re-launching the image handle.This isn't surprising. Libguestfs doesn't make an internal copy of the whole filesystem, but the libguestfs appliance will have a copy (in its kernel memory) of any parts of the disk that rsync read. See also: http://libguestfs.org/guestfs.3.html#architecture> Could we create a second partition on the guest that is much > smaller, say 50 MByte, attach the image to that, and thus reduce the > shutdown and re-launch time into the msec range? Is there another > real-time method, not using rsync?The time to relaunch the appliance doesn't depend on the size of the disk. You could try hotplugging but it may not be much faster: http://libguestfs.org/guestfs.3.html#hotplugging I think what you really need to do is to install a backup agent in the Windows guest and use a regular network backup. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v