The unstable tree now contains support for dynamic suspend and resume of guest OS instances. Here''s how it can currently be used (it''s not currently hooked up to higher-level user tools such as xenctl): tools/internal/xi_save_linux <domain number> <state file to write> ...this will stop the given domain, save its state, and resume execution of the domain. tools/internal/xi_restore_linux <state file> ...this will create a new domain, filling it with the state saved in <state file>. The created domain identifier will be printed to stdout. The domain must then be resumed with ''xi_start <domain id>''. Note that only CPU state is saved and restored --- there is currently no support for saving and restoring IP address information, or block-device access permissions. It is planned that this will be implemented by higher-level tools in the near future. For now, you will need to use tools such as ''xi_vifinit'' to specify this information for a restored guest OS (just as you do if you build a guest using ''xi_build'' instead of using ''xenctl''). -- Keir ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
First, I noted that xen_nat_enable was *not* built along with the other tools in xeno-clone/install/bin. Is this still needed (per the README.CD instructions, for a NAT-based virtual host, rather than IP-based)? I copied & ran the xen_nat_enable from the CD, and immediately was unable to access my machine to/from the network (I had already run "ifconfig eth0:0 169.254.1.0 up"). What I found was that the SuSEfirewall default configuration did not get along well with whatever changes to iptables were made by xen_nat_enable. My solution, which needs to be tuned later, was to edit /etc/sysconfig/SuSEfirewall2 to greatly loosen the firewall. I then restarted it: /etc/rc.d/SuSEfirewall2_init restart /etc/rc.d/SuSEfirewall2_setup restart /etc/rc.d/SuSEfirewall2_final restart The changes I made (again, these are certainly TOO MANY changes, but as you''ll see in my next note there are still problems with network access to the virtual systems): 127c127 < FW_DEV_INT="eth0:0" ---> FW_DEV_INT=""164c164 < FW_ROUTE="yes" ---> FW_ROUTE="no"179c179 < FW_MASQUERADE="yes" ---> FW_MASQUERADE="no"201c201 < FW_MASQ_NETS="169.254.1.0" ---> FW_MASQ_NETS=""217c217 < FW_PROTECT_FROM_INTERNAL="no" ---> FW_PROTECT_FROM_INTERNAL="yes"254c254 < FW_SERVICES_EXT_TCP="2200:2300 2049 http ssh rsync ftp smtp" ---> FW_SERVICES_EXT_TCP="2049 http ssh"Of course, your firewall configuration might be different. -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> First, I noted that xen_nat_enable was *not* built along with the > other tools in xeno-clone/install/bin. Is this still needed (per the > README.CD instructions, for a NAT-based virtual host, rather than > IP-based)?It''s a script rather than a binary. The current ''loop through domain0'' approach to NAT is not the long term solution (we''re adding NAT to Xen). I''m afraid I''m not entirely surprised that xen_nat_enable doesn''t play well with your firewall. Are you short of IP addresses? I''d certainly recommend using one IP per guest for the moment unless you really have to use NAT. Of course, you don''t need to use NAT if you only want to do inter-guest communication (you can use the 169.254.1.X addresses directly).> I copied & ran the xen_nat_enable from the CD, and immediately was > unable to access my machine to/from the network (I had already run > "ifconfig eth0:0 169.254.1.0 up"). > > What I found was that the SuSEfirewall default configuration did not > get along well with whatever changes to iptables were made by > xen_nat_enable. My solution, which needs to be tuned later, was to > edit /etc/sysconfig/SuSEfirewall2 to greatly loosen the firewall. I > then restarted it:Another thing to watch out for is that some distributions ''helpfully'' create random link-local 169.254.x.x addresses for all interfaces automatically. This doesn''t play well with our use of link-local addresses. e.g. you have to nail this in RH9 with ZEROCONF=NO in ifcfg-eth0 Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Nov 07, 2003 at 10:03:41PM +0000, Ian Pratt wrote:> > First, I noted that xen_nat_enable was *not* built along with the > > other tools in xeno-clone/install/bin. Is this still needed (per the > > README.CD instructions, for a NAT-based virtual host, rather than > > IP-based)? > > It''s a script rather than a binary.Yes....I was just worried about versioning, as we''ve been warned with the xi_ programs.> The current ''loop through domain0'' approach to NAT is not the > long term solution (we''re adding NAT to Xen). > > I''m afraid I''m not entirely surprised that xen_nat_enable doesn''t > play well with your firewall.I''ll do a little more diagnosis in the future. What I think happened, though, is that the NAT''s nat* rules somehow discarded the filter* rules. I was also getting some complaints about mangle* needing to load the iptables module, which was not found (this was when I was trying to re-add my default rules). iptables is a big pain, no matter what. But adding nat* rules (especially when there were none in the first place) seems like it should work.> Are you short of IP addresses? I''d certainly recommend using one > IP per guest for the moment unless you really have to use NAT. Of > course, you don''t need to use NAT if you only want to do > inter-guest communication (you can use the 169.254.1.X addresses > directly).1) Yes, I''ll have IP addresses to play with, but don''t yet as of today. 2) Hmmm -- this does not work. Any quick guess what to try fixing? $ xenctl domain list id: 0 (Domain-0) processor: 0 has cpu: true state: 0 active mcu advance: 10 total pages: 192000 id: 2 (XenoLinux) processor: 0 has cpu: false state: 1 stopped mcu advance: 10 total pages: 24576 $ ifconfig eth0:0 eth0:0 Link encap:Ethernet HWaddr 00:B0:D0:DF:FA:ED inet addr:169.254.1.0 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 $ ping -c1 169.254.1.0 PING 169.254.1.0 (169.254.1.0) from 169.254.1.0 : 56(84) bytes of data. 64 bytes from 169.254.1.0: icmp_seq=1 ttl=64 time=0.083 ms --- 169.254.1.0 ping statistics --- 1 packets transmitted, 1 received, 0% loss, time 0ms rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms $ ping -c1 169.254.1.1 PING 169.254.1.1 (169.254.1.1) from 169.254.1.0 : 56(84) bytes of data. [times out] --- 169.254.1.1 ping statistics --- 1 packets transmitted, 0 received, 100% loss, time 0ms I saw nat* rules for this, also for port 2201. But now I re-ran xen_nat_enable and locked myself out again. I''ll reboot, and look some more, but meanwhile maybe you can tell me: - Should 169.254.1.1 be ping-able from 169.254.1.0 ? - Do I "ssh -p2201 root@169.254.1.1" or "ssh -p2201 root@169.254.1.0" or "ssh -p2201 root@localhost" or the native IP address (137. ...) Finally, and this concludes today''s confusion: I seem unable to get any sort of console output after the kernel boots. I redirected with "console=xencons0", but even after NAT, don''t get anything. /dev/tty0 didn''t seem any different. I''d *really* like to redirect to a file, or to the physical console (tty0). I suspect this is another firewall issue with console messages via UDP & NAT, but if there is a workaround to get it sent to a file I''d greatly prefer it. Ok, one more random question: I noted that the new domain wanted an initrd.gz, but I did not get a new one in the xeno-clone tree. "mkinitrd -k image.gz -i initrd.gz" failed ("couldn''t find modules"). I copied the initrd from the CD and it seems to work, but it might not forever.> > I copied & ran the xen_nat_enable from the CD, and immediately was > > unable to access my machine to/from the network (I had already run > > "ifconfig eth0:0 169.254.1.0 up"). > > > > What I found was that the SuSEfirewall default configuration did not > > get along well with whatever changes to iptables were made by > > xen_nat_enable. My solution, which needs to be tuned later, was to > > edit /etc/sysconfig/SuSEfirewall2 to greatly loosen the firewall. I > > then restarted it: > > Another thing to watch out for is that some distributions > ''helpfully'' create random link-local 169.254.x.x addresses for > all interfaces automatically. This doesn''t play well with our use > of link-local addresses. e.g. you have to nail this in RH9 with ZEROCONF=NO > in ifcfg-eth0I''m using SuSE, which doesn''t seem to do this. However, the SuSE setup for iptables is *really* much more difficult than for RedHat. They obfuscate it in multiple layers of scripts and stuff... -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > I''m afraid I''m not entirely surprised that xen_nat_enable doesn''t > > play well with your firewall. > > I''ll do a little more diagnosis in the future. What I think > happened, though, is that the NAT''s nat* rules somehow discarded > the filter* rules. I was also getting some complaints about > mangle* needing to load the iptables module, which was not found > (this was when I was trying to re-add my default rules).I fear the xen_nat_enable script basically does a ''flush all rules'' to start with. Someone who understands iptables better should be able to fix this...> 2) Hmmm -- this does not work. Any quick guess what to try fixing?> $ xenctl domain list > id: 0 (Domain-0) > processor: 0 > has cpu: true > state: 0 active > mcu advance: 10 > total pages: 192000 > id: 2 (XenoLinux) > processor: 0 > has cpu: false > state: 1 stopped > mcu advance: 10 > total pages: 24576Did you start a domain 1 that then exited? The IP address of you''re currently running domain (id: 2) should be 169.254.1.2 "state: 1 stopped" doesn''t look good, though. Have you actually "xenctl domain start"''ed the domain? Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Nov 07, 2003 at 10:53:59PM +0000, Ian Pratt wrote:> > > > I''m afraid I''m not entirely surprised that xen_nat_enable doesn''t > > > play well with your firewall. > > > > I''ll do a little more diagnosis in the future. What I think > > happened, though, is that the NAT''s nat* rules somehow discarded > > the filter* rules. I was also getting some complaints about > > mangle* needing to load the iptables module, which was not found > > (this was when I was trying to re-add my default rules). > > I fear the xen_nat_enable script basically does a ''flush all > rules'' to start with. Someone who understands iptables better > should be able to fix this...Aha....easy to do. I just commented out the lines that flush the existing filter rules in xen_nat_enable: # run_iptables -t filter -F # run_iptables -t filter -X I can now run xen_nat_enable and it leaves my existing filter rules in place. The existing filter rules are extremely permissive.> > 2) Hmmm -- this does not work. Any quick guess what to try fixing? > > > $ xenctl domain list > > id: 0 (Domain-0) > > processor: 0 > > has cpu: true > > state: 0 active > > mcu advance: 10 > > total pages: 192000 > > id: 2 (XenoLinux) > > processor: 0 > > has cpu: false > > state: 1 stopped > > mcu advance: 10 > > total pages: 24576 > > Did you start a domain 1 that then exited?Yes, I had domain 1 that I stopped then killed. After starting domain 2, I still can''t connect. Details below.> The IP address of you''re currently running domain (id: 2) should > be 169.254.1.2 > > "state: 1 stopped" doesn''t look good, though. Have you actually > "xenctl domain start"''ed the domain?$ xenctl script -f/etc/xen-mydom (the default script) $ xenctl domain start -n2 $ xenctl domain list id: 0 (Domain-0) processor: 0 has cpu: true state: 0 active mcu advance: 10 total pages: 192000 id: 2 (XenoLinux) processor: 0 has cpu: false state: 0 active mcu advance: 10 total pages: 24576 $ ifconfig eth0:0 eth0:0 Link encap:Ethernet HWaddr 00:B0:D0:DF:FA:ED inet addr:169.254.1.0 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 (I''ll use raw telnet to better diagnose the failures): The system I''m using is 137.229.71.6, statically assigned. works: telnet 169.254.1.0 22 times out: telnet 169.254.1.2 22 connection refused: telnet 169.254.1.0 2202 connection refused: telnet 137.229.71.6 2202 It looks to me like either the built-in firewall is blocking incoming access at 169.254.1.2 (the virtual domain), or the virtual domain is simply unable to access the network connection. As I mentioned in my other message, it would be great to be able to see console messages, but they are either being firewalled or otherwise redirected. -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> # run_iptables -t filter -F > # run_iptables -t filter -X > > I can now run xen_nat_enable and it leaves my existing filter > rules in place. The existing filter rules are extremely > permissive.It''s arguable that these 2 lines are a bug in the script...> $ xenctl script -f/etc/xen-mydom (the default script) > $ xenctl domain start -n2The /etc/xen-mydom should automatically start the domain.> As I mentioned in my other message, it would be great to be able to > see console messages, but they are either being firewalled or > otherwise redirected.Have you been using xen_read_console? You should be able to watch the other domain booting, and check that it comes up OK. Please can you send me the output from running xenctl, and the console message from the booting domain. Thanks, Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Nov 08, 2003 at 12:36:25AM +0000, Ian Pratt wrote:> > # run_iptables -t filter -F > > # run_iptables -t filter -X > > > > I can now run xen_nat_enable and it leaves my existing filter > > rules in place. The existing filter rules are extremely > > permissive. > > It''s arguable that these 2 lines are a bug in the script...:-) Meanwhile, I have completely disabled the firewall (iptables still works, but allows everything). This hasn''t changed behaviour from my previous message, though.> > $ xenctl script -f/etc/xen-mydom (the default script) > > $ xenctl domain start -n2 > > The /etc/xen-mydom should automatically start the domain.It doesn''t. (You saw my prior "xenctl domain list" output, which said it was stopped.)> > As I mentioned in my other message, it would be great to be able to > > see console messages, but they are either being firewalled or > > otherwise redirected. > > Have you been using xen_read_console? You should be able to > watch the other domain booting, and check that it comes up OK.I run it (in the background) but never see anything. Even when I reboot, I don''t get shutdown messages (they don''t appear on the physical console).> Please can you send me the output from running xenctl, and the > console message from the booting domain.Yep. Maybe the output from the "xenctl script..." startup is informative. This is with the default /etc/xen-mynewdom, containing: -- domain new physical grant -pcdrom_link domain start -- Script started on Fri Nov 7 15:53:22 2003 peabody(root) ~ [2] > xenctl script -f/etc/xen-mynewdom Domain defaults: name XenoLinux size 98304 vifs 1 domainImage /boot/xenolinux.gz domainInitRD /boot/initrd.gz rootDevice /dev/ram0 rootArgs rw usrDevice null NWIP 169.254.1.0+ NWGW 169.254.1.0 NWMask 255.255.0.0 MaxDomainNumber 1000 NWNFSServer 169.254.1.0 NWNFSRoot null XIToolsDir /usr/local/bin/ args init=/linuxrc 4 DOMID=+ Domain created with arguments: /usr/local/bin/xi_create 98304 XenoLinux Domain built with arguments: /usr/local/bin/xi_build 3 /tmp/xen-image-40068.tmp 1 initrd=/tmp/xen-initrd-40069.tmp ip=169.254.1.3:169.254.1.0:169.254.1.0:255.255.0.0::eth0:off init=/linuxrc 4 DOMID=3 root=/dev/ram0 rw VIF 0 initialized with arguments: /usr/local/bin/xi_vifinit 3 0 169.254.1.3 warning: state file not found [/var/lib/xen/vdstate.xml] Partition cdrom_link (resolved to cdrom_link) does not exist. peabody(root) ~ [3] > xenctl domain list id: 0 (Domain-0) processor: 0 has cpu: true state: 0 active mcu advance: 10 total pages: 192000 id: 1 (XenoLinux) processor: 1 has cpu: false state: 1 stopped mcu advance: 10 total pages: 24576 id: 2 (XenoLinux) processor: 0 has cpu: false state: 1 stopped mcu advance: 10 total pages: 24576 id: 3 (XenoLinux) processor: 1 has cpu: false state: 1 stopped mcu advance: 10 total pages: 24576 peabody(root) ~ [4] > xenctl domain start -n3 Started domain 3 peabody(root) ~ [5] > ifconfig -a eth0 Link encap:Ethernet HWaddr 00:B0:D0:DF:FA:ED inet addr:137.229.71.6 Bcast:137.229.71.15 Mask:255.255.255.240 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:86 errors:0 dropped:0 overruns:0 frame:0 TX packets:51 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:8575 (8.3 Kb) TX bytes:3063 (2.9 Kb) eth0:0 Link encap:Ethernet HWaddr 00:B0:D0:DF:FA:ED inet addr:169.254.1.0 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:78 errors:0 dropped:0 overruns:0 frame:0 TX packets:78 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5267 (5.1 Kb) TX bytes:5267 (5.1 Kb) peabody(root) ~ [6] > telnet 169.254.1.3 22 Trying 169.254.1.3... telnet: connect to address 169.254.1.3: Connection refused peabody(root) ~ [7] > telnet 169.254.1.3 22 Trying 169.254.1.0... telnet: connect to address 169.254.1.0: Connection refused peabody(root) ~ [8] > telnet 169.254.1.0 2203 Trying 169.254.1.1... telnet: connect to address 169.254.1.1: No route to host peabody(root) ~ [9] > telnet 169.254.1.1 2203 Trying 169.254.1.3... telnet: connect to address 169.254.1.3: Connection refused peabody(root) ~ [10] > telnet 169.254.1.3 22 Trying 169.254.1.3... telnet: connect to address 169.254.1.3: Connection refused Script done on Fri Nov 7 15:54:43 2003 ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > > $ xenctl script -f/etc/xen-mydom (the default script) > > > $ xenctl domain start -n2 > > > > The /etc/xen-mydom should automatically start the domain. > > It doesn''t. (You saw my prior "xenctl domain list" output, which said > it was stopped.)This is really odd. The last line of the script contains a "domain start". Is there a missing LF or something?> > > As I mentioned in my other message, it would be great to be able to > > > see console messages, but they are either being firewalled or > > > otherwise redirected. > > > > Have you been using xen_read_console? You should be able to > > watch the other domain booting, and check that it comes up OK. > > I run it (in the background) but never see anything. Even > when I reboot, I don''t get shutdown messages (they don''t > appear on the physical console).Very odd. Any chance you can get a serial line on the system? The other domain''s boot messages should also come out on serial.> > Please can you send me the output from running xenctl, and the > > console message from the booting domain. > > Yep. Maybe the output from the "xenctl script..." startup is > informative. This is with the default /etc/xen-mynewdom, containing:I take it that you''re wanting to boot with the initrd copied off the CD, and use the CD for the new domain''s /usr ?> peabody(root) ~ [6] > telnet 169.254.1.3 22 > Trying 169.254.1.3... > telnet: connect to address 169.254.1.3: Connection refusedAFAIK, Our CD doesn''t run a telnetd by default. There should be an sshd, but I think your problem lies elsewhere... Connection refused is a slightly odd message. If the domain was totally dead, I''d expect the telnet to hang. What happens if you run tcpdump in domain0. Do you see any packets arriving at 169.254.1.0 ? Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
I hope folks on the xen-devel list don''t mind all these messages. I think it''s been fascinating to see answers to all these questions, and hope you agree. My view is that the mailing list archive is, currently, a strong supplement to the documentation :-) More: On Sat, Nov 08, 2003 at 01:22:03AM +0000, Ian Pratt wrote:> > > > > $ xenctl script -f/etc/xen-mydom (the default script) > > > > $ xenctl domain start -n2 > > > > > > The /etc/xen-mydom should automatically start the domain. > > > > It doesn''t. (You saw my prior "xenctl domain list" output, which said > > it was stopped.) > > This is really odd. The last line of the script contains a > "domain start". Is there a missing LF or something?No, the file looks OK, but the answer below about /usr probably explains this.> > > > As I mentioned in my other message, it would be great to be able to > > > > see console messages, but they are either being firewalled or > > > > otherwise redirected. > > > > > > Have you been using xen_read_console? You should be able to > > > watch the other domain booting, and check that it comes up OK. > > > > I run it (in the background) but never see anything. Even > > when I reboot, I don''t get shutdown messages (they don''t > > appear on the physical console). > > Very odd. Any chance you can get a serial line on the system? > The other domain''s boot messages should also come out on serial.Yes, I brought in a null modem. I''ll try this.> > > Please can you send me the output from running xenctl, and the > > > console message from the booting domain. > > > > Yep. Maybe the output from the "xenctl script..." startup is > > informative. This is with the default /etc/xen-mynewdom, containing: > > I take it that you''re wanting to boot with the initrd copied > off the CD, and use the CD for the new domain''s /usr ?Huh? No, that''s the first I heard about that. I''m using the standard /usr This could explain a lot. How am I supposed to make the CD''s /usr available to the domains? All I did was copy the xeno-clone/install/bin/ programs to /usr/local/bin , and the xen_nat_enable from the CD to /usr/local/bin> > peabody(root) ~ [6] > telnet 169.254.1.3 22 > > Trying 169.254.1.3... > > telnet: connect to address 169.254.1.3: Connection refused > > AFAIK, Our CD doesn''t run a telnetd by default. There should be > an sshd, but I think your problem lies elsewhere...sshd listens on port 22. By "telnet HOSTNAME 22" I''m trying to connect to the ssh port. The advantage of doing it this way is that the client & negotiation don''t matter... just the ability to connect. The NAT rules in iptables redirects port 22 on 169.254.1.3 (in this case) to port 2203 on 169.254.1.0. So, theoretically, "telnet 169.254.1.3 22" is the same as "telnet 169.254.1.0 2203". To actually login, ssh root@169.254.1.3 or ssh -p 2203 root@169.254.1.0 (right?)> Connection refused is a slightly odd message. If the domain was > totally dead, I''d expect the telnet to hang. > > What happens if you run tcpdump in domain0. Do you see any > packets arriving at 169.254.1.0 ?Yes. Here is "grep 169" from a tcpdump log while I tried (from domain0) "telnet 169.254.1.3 22" (yes, the arp reply matches eth0''s MAC): 16:27:44.364911 peabody.arsc.edu.1028 > 137.229.18.15.domain: 49905+ PTR? 3.1.254.169.in-addr.arpa. (42) (DF) 16:27:44.366554 arp who-has 169.254.1.3 tell 169.254.1.0 16:27:44.366633 arp reply 169.254.1.3 is-at 0:b0:d0:df:fa:ed 16:27:44.366644 169.254.1.0.1041 > 169.254.1.3.ssh: S 2092748429:2092748429(0) win 5840 <mss 1460,sackOK,timestamp 283781 0,nop,wscale 0> (DF) [tos 0x10] 16:27:44.366727 169.254.1.3.ssh > 169.254.1.0.1041: R 0:0(0) ack 2092748430 win 0 (DF) [tos 0x10] 16:27:44.367337 peabody.arsc.edu.1028 > 137.229.18.15.domain: 28243+ PTR? 3.1.254.169.in-addr.arpa. (42) (DF) -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > Very odd. Any chance you can get a serial line on the system? > > The other domain''s boot messages should also come out on serial. > > Yes, I brought in a null modem. I''ll try this.This will be very intersting.> > > > Please can you send me the output from running xenctl, and the > > > > console message from the booting domain. > > > > > > Yep. Maybe the output from the "xenctl script..." startup is > > > informative. This is with the default /etc/xen-mynewdom, containing: > > > > I take it that you''re wanting to boot with the initrd copied > > off the CD, and use the CD for the new domain''s /usr ? > > Huh? No, that''s the first I heard about that. > > I''m using the standard /usr > > This could explain a lot. How am I supposed to make > the CD''s /usr available to the domains?The easiest thing to do for testing is to put the CD in the drive. You really need to install other filesystems (on either real partitions or virtual disks) for other domains, or export them from domain 0 via local NFS.> > an sshd, but I think your problem lies elsewhere... > > sshd listens on port 22. By "telnet HOSTNAME 22" I''m trying > to connect to the ssh port. The advantage of doing it this way > is that the client & negotiation don''t matter... just the > ability to connect.I missed the final "22".> The NAT rules in iptables redirects port 22 on 169.254.1.3 > (in this case) to port 2203 on 169.254.1.0. So, theoretically, > "telnet 169.254.1.3 22" is the same as "telnet 169.254.1.0 2203". > To actually login, > ssh root@169.254.1.3 > or ssh -p 2203 root@169.254.1.0I''m still nervous about the NAT/firewall set up. Seeing as you''re only using local networking for this, you shouldn''t need xen_nat_enable at all -- just reboot and bring up eth0:0 by hand. After starting a new domain you should be able to ping and ssh root@169.254.1.X if things are well.> > What happens if you run tcpdump in domain0. Do you see any > > packets arriving at 169.254.1.0 ? > > Yes. Here is "grep 169" from a tcpdump log while I tried (from > domain0) "telnet 169.254.1.3 22" (yes, the arp reply matches > eth0''s MAC):It would be interesting to see if you receive any packets while the domain is booting (console packets). Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Nov 08, 2003 at 01:57:36AM +0000, Ian Pratt wrote:> > > > Very odd. Any chance you can get a serial line on the system? > > > The other domain''s boot messages should also come out on serial. > > > > Yes, I brought in a null modem. I''ll try this. > > This will be very intersting.I''ll forward the console log to you privately (since it''s long and boring) in my next message. Basically, the serial port captured the DOM0 boot messages (which, previously, I had not seen), but didn''t generate anything from DOMID=1 etc. when I started them. The log shows a boot, then I ran "xen_nat_enable" followed by "xenctl script -f/etc/xen-mynewdom" followed by "xenctl domain start -n1" Only the boot generated any messages.> > > > > Please can you send me the output from running xenctl, and the > > > > > console message from the booting domain. > > > > > > > > Yep. Maybe the output from the "xenctl script..." startup is > > > > informative. This is with the default /etc/xen-mynewdom, containing: > > > > > > I take it that you''re wanting to boot with the initrd copied > > > off the CD, and use the CD for the new domain''s /usr ? > > > > Huh? No, that''s the first I heard about that. > > > > I''m using the standard /usr > > > > This could explain a lot. How am I supposed to make > > the CD''s /usr available to the domains? > > The easiest thing to do for testing is to put the CD in the > drive.You mean, it will automatically mount & find the /usr on the drive? OR, I should mount first (where?) OR, I need to boot from the CD (that was last week...this week, we''re trying to get it all installed on the hard drives).> You really need to install other filesystems (on either real > partitions or virtual disks) for other domains, or export them > from domain 0 via local NFS.Actually, this might be easier. Let''s say I allocate a real partition, and configure grub to boot from it (rather than my current /dev/sda2) Should I simply copy (with the same permissions) the entire CD, so that the root on the real partition is the root on the CD...and then, over-write the files in /boot and /bin with those from the new xeno-clone/install/.. tree?> > > an sshd, but I think your problem lies elsewhere... > > > > sshd listens on port 22. By "telnet HOSTNAME 22" I''m trying > > to connect to the ssh port. The advantage of doing it this way > > is that the client & negotiation don''t matter... just the > > ability to connect. > > I missed the final "22". > > > The NAT rules in iptables redirects port 22 on 169.254.1.3 > > (in this case) to port 2203 on 169.254.1.0. So, theoretically, > > "telnet 169.254.1.3 22" is the same as "telnet 169.254.1.0 2203". > > To actually login, > > ssh root@169.254.1.3 > > or ssh -p 2203 root@169.254.1.0 > > I''m still nervous about the NAT/firewall set up. > > Seeing as you''re only using local networking for this, you > shouldn''t need xen_nat_enable at all -- just reboot and bring up > eth0:0 by hand.I tried that...> After starting a new domain you should be able to ping and ssh > root@169.254.1.X if things are well.Things are not well. It''s looking to me like DOMID=1 etc. are not able to access the network, or start sshd, or some other trajedy.> > > What happens if you run tcpdump in domain0. Do you see any > > > packets arriving at 169.254.1.0 ? > > > > Yes. Here is "grep 169" from a tcpdump log while I tried (from > > domain0) "telnet 169.254.1.3 22" (yes, the arp reply matches > > eth0''s MAC): > > It would be interesting to see if you receive any packets while > the domain is booting (console packets).I''ll check this. ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Nov 08 03 at 12:36:25AM +0000, Ian Pratt wrote:> > # run_iptables -t filter -F > > # run_iptables -t filter -X > > > > I can now run xen_nat_enable and it leaves my existing filter > > rules in place. The existing filter rules are extremely > > permissive. > > It''s arguable that these 2 lines are a bug in the script...The flush/clear commands are to make the script idempotent; as long as you only run it once, and the actions it takes don''t interact with your existing firewall, it''ll be fine. You would do better to add the NAT rules into the firewall setup itself, though, as this would get rid of any ordering problems..etc. xen_nat_enable, other than making assumptions for you about IP addresses, doesn''t do anything Xen-specific. Torne who wrote that ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > > Have you been using xen_read_console? You should be able to > > > watch the other domain booting, and check that it comes up OK. > > > > I run it (in the background) but never see anything. Even > > when I reboot, I don''t get shutdown messages (they don''t > > appear on the physical console). > > Very odd. Any chance you can get a serial line on the system? > The other domain''s boot messages should also come out on serial.It sounds to me like a misconfigured domain 0 firewall. Can you send the output from ''iptables -L -v'' and ''iptables -tnat -L -v'' ? If you do that just before and just after booting a new domain then that may allow us to see which rule is dropping the console UDP packets. -- Keir ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Nov 08, 2003 at 04:20:59AM +0000, Torne Wuff wrote:> On Sat, Nov 08 03 at 12:36:25AM +0000, Ian Pratt wrote: > > > # run_iptables -t filter -F > > > # run_iptables -t filter -X > > > > > > I can now run xen_nat_enable and it leaves my existing filter > > > rules in place. The existing filter rules are extremely > > > permissive. > > > > It''s arguable that these 2 lines are a bug in the script... > > The flush/clear commands are to make the script idempotent; as long as > you only run it once, and the actions it takes don''t interact with your > existing firewall, it''ll be fine.Sorry for the late follow-up, Torne. I don''t understand this - reality seems to the be opposite of what you wrote. If you flush (-F) & delete (-X) all the existing filter rules & chain, then it clearly *does* interact with the current firewall. Since the script applies to NAT rules, not filter rules (the *nat section, not the *filter section), I don''t understand why you would touch the filter rules at all. Flushing the existing *nat rules might make sense, but I''m not so certain about that either. It might be there is some variability in how different distributions set up firewall rules, and I''m not an expert in iptables at all, so apologies if this doesn''t make sense.> You would do better to add the NAT rules into the firewall setup itself, > though, as this would get rid of any ordering problems..etc. > xen_nat_enable, other than making assumptions for you about IP > addresses, doesn''t do anything Xen-specific.Yes, this is clearly the way to go for a stable setup. You''d need to have all the firewall rules in place at system boot time, assuming that you''ll also be booting non-0 Xen domains. But for experimentation, people will try to use the existing tools. -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I don''t understand this - reality seems to the be opposite of what you > wrote. If you flush (-F) & delete (-X) all the existing filter rules > & chain, then it clearly *does* interact with the current firewall.I think Richard meant that having the -F and -X made the script idempotent with respect to itself. I think it''s more useful just to remove the two lines -- I''ll check in a ''fix''. BTW: Greg, have you had a chance to modify the xen-mynewdom script as I suggested? Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sat, Nov 08, 2003 at 08:48:59AM +0000, Keir Fraser wrote:> > > > > Have you been using xen_read_console? You should be able to > > > > watch the other domain booting, and check that it comes up OK.> > > I run it (in the background) but never see anything. Even > > > when I reboot, I don''t get shutdown messages (they don''t > > > appear on the physical console). > > > > Very odd. Any chance you can get a serial line on the system? > > The other domain''s boot messages should also come out on serial.They do. But the system unit is in another room, so it''s not too convenient to get these messages. I''d be happiest for them to go to a file!> It sounds to me like a misconfigured domain 0 firewall. Can you send > the output from ''iptables -L -v'' and ''iptables -tnat -L -v'' ? > > If you do that just before and just after booting a new domain then > that may allow us to see which rule is dropping the console UDP packets.I''m finally picking this up again - sorry for not getting right to it. The problem we''re trying to solve is that console messages are going to the serial port, but not the physical console or to the shell via xen_read_console. I experiemented a lot, and this message was 1000 lines longer with output from iptables etc. Bottom line is this now works, though I''m not 100% certain I can replicate all the differences. Basically: 1) Reconfigure the default firewall rules to block nothing and accept everything; 2) Reboot There is still a very desirable feature: I''d *really* like xenconsole messages from all domains to go to a file. The basic setup I have for virtual domains required: 1) ln -s /dev/hdc /dev/cdrom_link (or modify /etc/xen-mynewdom) 2) leave the CD-ROM in the drawer, but don''t boot from it 3) boot to Xen (my new images, discussed earlier) 3a) run "xen_read_console &" as root, to see boot messages 4) start new domains with xenctl Steps 1 and 2 are not clear from the 1.0 README.CD. I now have virtual domains booted and can access them. I will send another note describing what I''d like to do to get these living on the real (non-ram) file system with NFS and shared /usr etc., but will experiment more first. Thanks! -- Greg ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I experiemented a lot, and this message was 1000 lines longer > with output from iptables etc. Bottom line is this now works, > though I''m not 100% certain I can replicate all the differences. > > Basically: > 1) Reconfigure the default firewall rules to block nothing and > accept everything; > 2) RebootGreat. Everything was pointing toward a firewall problem -- if the messages were making it to serial, they really should have been making it to domain 0. What domain0 chooses to do with them is another matter ;-)> There is still a very desirable feature: I''d *really* like > xenconsole messages from all domains to go to a file.There''s plans to change some of the domain console stuff to make it do input as well as output. One option under consideration is to make the console present itself to domain0 using a custom mechanism rather than UDP. This would have the advantage of avoiding dependencies on people''s firewall setups, but I''m not personally keen on introducing another communication mechanism. Besides, it''s only a dependency on the domain 0 firewall configuration -- all other domains can do what they like. As for sending to a file, you can just redirect as per normal. "xen_read_console | tee myconsole" (though this obviously assumes that the 169.254.1.0 alias is in place and your firewall isn''t binning the packets)> The basic setup I have for virtual domains required: > 1) ln -s /dev/hdc /dev/cdrom_link (or modify /etc/xen-mynewdom) > 2) leave the CD-ROM in the drawer, but don''t boot from it > 3) boot to Xen (my new images, discussed earlier) > 3a) run "xen_read_console &" as root, to see boot messages > 4) start new domains with xenctl > > Steps 1 and 2 are not clear from the 1.0 README.CD.Phew. We hadn''t anticipated that anyone would want to use the CD in quite this manner, but we can update the documentation accordingly.> I now have virtual domains booted and can access them. I will send > another note describing what I''d like to do to get these living on the > real (non-ram) file system with NFS and shared /usr etc., but will > experiment more first.NFS to domain0 is the way I have my laptop configured and it works well. Ian ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Mon, Nov 10 03 at 8:32:30PM +0000, Ian Pratt wrote:> > I don''t understand this - reality seems to the be opposite of what you > > wrote. If you flush (-F) & delete (-X) all the existing filter rules > > & chain, then it clearly *does* interact with the current firewall. > > I think Richard meant that having the -F and -X made the script > idempotent with respect to itself.Yes.> I think it''s more useful just to remove the two lines -- I''ll > check in a ''fix''.This will probably break NAT. The NAT script adds rules to the filter table which are appended to the end; these rules are required to allow the traffic to be forwarded. If a firewall script runs first, then they will be added after the firewall''s rules; many firewalls put in a catch-all DROP or REJECT rule as the last entry (so that logging can be done..etc rather than rely on a table policy) so this will break. Also, the line ''-t filter -P FORWARD DROP'' changes the default policy for the FORWARD table, whcih may also interact with a firewall. If the firewall only touches the INPUT table you shouldn''t have a problem. You still want to flush the FORWARD table on running this script, however; Ian: substitute ''-t filter -F'' for ''-t filter -F FORWARD'' and remove ''-t filter -X''. Any firewall which touches the FORWARD table is liable to either break, or break NAT. If you want to be able to use an existing firewall with NAT and be assured of it definatly working, you need to write the NAT rules yourself. If you need documentation on how to do NAT, the NAT HOWTO at www.netfilter.org is very informative and covers how to set up firewall rules to play nicely with NAT. -- Torne Wuff torne@wolfpuppy.org.uk ------------------------------------------------------- This SF.Net email sponsored by: ApacheCon 2003, 16-19 November in Las Vegas. Learn firsthand the latest developments in Apache, PHP, Perl, XML, Java, MySQL, WebDAV, and more! http://www.apachecon.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel