Lee Garrett
2023-Sep-22 14:47 UTC
[Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot
On 22.09.23 14:54, Richard W.M. Jones wrote:> On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>> On 21.09.23 19:43, Richard W.M. Jones wrote: >>>> So this is probably another instance or variation of the timezone >>>> formatting problem (of schtasks). Which version of virt-v2v is this? >>>> I want to check that you have a version with all the latest patches in >>>> this area. >>> >>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>> doesn't have any distro-specific patches. >>> >>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>> would have a patches/series file in this case) >> >> The timezone fixes are: >> >> commit 597d177567234c3a539098c423649781424eeb6f >> Author: Laszlo Ersek <lersek at redhat.com> >> Date: Tue Mar 8 15:30:51 2022 +0100 >> >> convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell >> >> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >> Author: Richard W.M. Jones <rjones at redhat.com> >> Date: Fri Nov 12 08:47:55 2021 +0000 >> >> convert/convert_windows.ml: Handle date formats with dots instead of / >> >> They are all included in >= 2.0 >> >> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >> somewhere when refactoring this code later. >> >> Lee: Do you have a theory about exactly what is wrong with the >> schtasks date? Like what was it supposed to be, assuming it was 120 >> seconds in the future from boot time, versus what it was set to: >> >>> Firstboot-qemu-ga 9/21/2023 4:04:00 PM Ready >> >> Could a date or time field have not been swapped or been corrupted >> in some predictable way? > > Or in even simpler terms, what is the time (and timezone) that > this ^^^ machine was booted?I believe I have it figured out. The guest local time is currently 7:08 AM (a few minutes after firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays it as "UTC-08:00"). This is the timezone that the guest comes configured with at first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 hours in the future. So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. From what I can see, the host machine time zone is irrelevant (UTC+2). I don't know where the timezone mixup comes from, though. Running `(get-date)` in the powershell at this point correctly returns the local time (7:08 AM). I guess during injection the time is in UTC, and schtasks.exe has no awareness of timezones?> > Rich. > >> The code we run is here: >> >> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml#L571 >> >> Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >> somehow related to having a single digit month field? >> >> Rich. >> >> -- >> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones >> Read my programming and virtualization blog: http://rwmj.wordpress.com >> libguestfs lets you edit virtual machines. Supports shell scripting, >> bindings from many languages. http://libguestfs.org >
Lee Garrett
2023-Sep-22 15:27 UTC
[Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot
On 22.09.23 16:47, Lee Garrett wrote:> On 22.09.23 14:54, Richard W.M. Jones wrote: >> On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >>> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>>> On 21.09.23 19:43, Richard W.M. Jones wrote: >>>>> So this is probably another instance or variation of the timezone >>>>> formatting problem (of schtasks).? Which version of virt-v2v is this? >>>>> I want to check that you have a version with all the latest patches in >>>>> this area. >>>> >>>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>>> doesn't have any distro-specific patches. >>>> >>>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>>> would have a patches/series file in this case) >>> >>> The timezone fixes are: >>> >>> commit 597d177567234c3a539098c423649781424eeb6f >>> Author: Laszlo Ersek <lersek at redhat.com> >>> Date:?? Tue Mar 8 15:30:51 2022 +0100 >>> >>> ???? convert_windows: rewrite "configure_qemu_ga" script purely in PowerShell >>> >>> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >>> Author: Richard W.M. Jones <rjones at redhat.com> >>> Date:?? Fri Nov 12 08:47:55 2021 +0000 >>> >>> ???? convert/convert_windows.ml: Handle date formats with dots instead of / >>> >>> They are all included in >= 2.0 >>> >>> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >>> somewhere when refactoring this code later. >>> >>> Lee: Do you have a theory about exactly what is wrong with the >>> schtasks date?? Like what was it supposed to be, assuming it was 120 >>> seconds in the future from boot time, versus what it was set to: >>> >>>> Firstboot-qemu-ga??????????????????????? 9/21/2023 4:04:00 PM?? Ready >>> >>> Could a date or time field have not been swapped or been corrupted >>> in some predictable way? >> >> Or in even simpler terms, what is the time (and timezone) that >> this ^^^ machine was booted? > > I believe I have it figured out. > The guest local time is currently 7:08 AM (a few minutes after > firstboot/provisioning), pacific daylight time (UTC-7, though Windows displays > it as "UTC-08:00"). This is the timezone that the guest comes configured with at > first boot. The task is scheduled for 2:01 PM, meaning it's scheduled to run ~7 > hours in the future. > > So it seems like the task was meant to be scheduled for 2:01 PM UTC (= 7:01 AM > PDT), but for some reason was scheduled for 2:01 PM *local time*. > > From what I can see, the host machine time zone is irrelevant (UTC+2). > > I don't know where the timezone mixup comes from, though. Running `(get-date)` > in the powershell at this point correctly returns the local time (7:08 AM). I > guess during injection the time is in UTC, and schtasks.exe has no awareness of > timezones?I digged a bit into the XML description and noticed this: <clock offset="utc"/> To my knowledge Windows runs the BIOS clock in local time. So This should probably be <clock offset="localtime"> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> <timer name="hypervclock" present="yes"/> </clock> as this is what libvirt uses in the preset for Windows 11 hosts. I manually changed this before first boot, however it didn't solve the issue. It only increased the offset to +9 hours in the future (7 hours difference to UTC, +2 hours of local timezone). How does the firstboot injection exactly happen? Is the guest actually booted for it?> >> >> Rich. >> >>> The code we run is here: >>> >>> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml#L571 >>> >>> Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >>> somehow related to having a single digit month field? >>> >>> Rich. >>> >>> -- >>> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones >>> Read my programming and virtualization blog: http://rwmj.wordpress.com >>> libguestfs lets you edit virtual machines.? Supports shell scripting, >>> bindings from many languages.? http://libguestfs.org >> >
Laszlo Ersek
2023-Sep-23 17:37 UTC
[Libguestfs] Fwd: virt-v2v creating image that does not install guest agent on first boot
On 9/22/23 16:47, Lee Garrett wrote:> On 22.09.23 14:54, Richard W.M. Jones wrote: >> On Fri, Sep 22, 2023 at 11:40:03AM +0100, Richard W.M. Jones wrote: >>> On Thu, Sep 21, 2023 at 07:47:52PM +0200, Lee Garrett wrote: >>>> On 21.09.23 19:43, Richard W.M. Jones wrote: >>>>> So this is probably another instance or variation of the timezone >>>>> formatting problem (of schtasks).? Which version of virt-v2v is this? >>>>> I want to check that you have a version with all the latest patches in >>>>> this area. >>>> >>>> It's 2.2.0-1 from Debian (12) bookworm. I've verified that it >>>> doesn't have any distro-specific patches. >>>> >>>> (https://salsa.debian.org/libvirt-team/virt-v2v/-/tree/debian/master/debian >>>> would have a patches/series file in this case) >>> >>> The timezone fixes are: >>> >>> commit 597d177567234c3a539098c423649781424eeb6f >>> Author: Laszlo Ersek <lersek at redhat.com> >>> Date:?? Tue Mar 8 15:30:51 2022 +0100 >>> >>> ???? convert_windows: rewrite "configure_qemu_ga" script purely in >>> PowerShell >>> >>> commit d9dc6c42ae64ba92993dbd9477f003ba73fcfa2f >>> Author: Richard W.M. Jones <rjones at redhat.com> >>> Date:?? Fri Nov 12 08:47:55 2021 +0000 >>> >>> ???? convert/convert_windows.ml: Handle date formats with dots >>> instead of / >>> >>> They are all included in >= 2.0 >>> >>> I wonder if 597d177567 has a subtle flaw, or if we introduced a bug >>> somewhere when refactoring this code later. >>> >>> Lee: Do you have a theory about exactly what is wrong with the >>> schtasks date?? Like what was it supposed to be, assuming it was 120 >>> seconds in the future from boot time, versus what it was set to: >>> >>>> Firstboot-qemu-ga??????????????????????? 9/21/2023 4:04:00 PM?? Ready >>> >>> Could a date or time field have not been swapped or been corrupted >>> in some predictable way? >> >> Or in even simpler terms, what is the time (and timezone) that >> this ^^^ machine was booted? > > I believe I have it figured out. > The guest local time is currently 7:08 AM (a few minutes after > firstboot/provisioning), pacific daylight time (UTC-7, though Windows > displays it as "UTC-08:00"). This is the timezone that the guest comes > configured with at first boot. The task is scheduled for 2:01 PM, > meaning it's scheduled to run ~7 hours in the future. > > So it seems like the task was meant to be scheduled for 2:01 PM UTC (> 7:01 AM PDT), but for some reason was scheduled for 2:01 PM *local time*. > > From what I can see, the host machine time zone is irrelevant (UTC+2). > > I don't know where the timezone mixup comes from, though. Running > `(get-date)` in the powershell at this point correctly returns the local > time (7:08 AM). I guess during injection the time is in UTC, and > schtasks.exe has no awareness of timezones?Right, I think there is a timezone disagreement between how we format the timestamp and how schtasks.exe takes it. What matters here is the /ST (start time) flag. Today we have (in the common submodule): add "$d = (get-date).AddSeconds(120)"; add "$dtfinfo = [System.Globalization.DateTimeFormatInfo]::CurrentInfo"; add "$sdp = $dtfinfo.ShortDatePattern"; add "$sdp = $sdp -replace 'y+', 'yyyy'"; add "$sdp = $sdp -replace 'M+', 'MM'"; add "$sdp = $sdp -replace 'd+', 'dd'"; add "schtasks.exe /Create /SC ONCE `"; add " /ST $d.ToString('HH:mm') /SD $d.ToString($sdp) `"; add " /RU SYSTEM /TN Firstboot-qemu-ga `"; add (sprintf " /TR \"C:\\%s /forcerestart /qn /l+*vx C:\\%s.log\"" msi_path msi_path); Note that for the /ST option's argument, we only perform the following steps: $d = (get-date).AddSeconds(120) /ST $d.ToString('HH:mm') This actually goes back to commit dc66e78fa37d ("windows: delay installation of qemu-ga MSI", 2020-03-10). The timestamp massaging we've since done only targeted the /SD (start date) option, not the start time (/ST) one! So the problem may be that (get-date).AddSeconds(120).ToString('HH:mm') formats the hour:minute timestamp in UTC (why though?), but the /ST option takes hour:minute in local time. Interestingly, DateTime objects seem to have a "Kind" property, which may be Utc, Local, or Unspec. https://learn.microsoft.com/en-us/dotnet/api/system.datetime.kind It seems to be used when converting from UTC to local or vice versa, and it probably influences how ToString() behaves too. I thought "get-date" returned a Local one, and /ST took a local one as well, but perhaps "get-date" returns a UTC timestamp in this case (when run from the script)? Laszlo> >> >> Rich. >> >>> The code we run is here: >>> >>> https://github.com/libguestfs/libguestfs-common/blob/e70d89a58dae068be2e19c7c21558707261af96a/mlcustomize/inject_virtio_win.ml#L571 >>> >>> Ming: this could be a bug affecting PST (UTC-8) guests, perhaps >>> somehow related to having a single digit month field? >>> >>> Rich. >>> >>> --? >>> Richard Jones, Virtualization Group, Red Hat >>> http://people.redhat.com/~rjones >>> Read my programming and virtualization blog: http://rwmj.wordpress.com >>> libguestfs lets you edit virtual machines.? Supports shell scripting, >>> bindings from many languages.? http://libguestfs.org >> >