Zhigang Wang
2013-Sep-13 16:04 UTC
Suggestion for merging xl save/restore/migrate/migrate-receive
Hi, As we talked in http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a suggestion: what about merging xl save/restore/migrate/migrate-receive? Here is the description: xl-migrate.rst Also there is a workaround for the xl migration daemon example: xl-migrate-socat.rst Patch for the example: xen-xl-migrate-socat.patch (I don''t have enough knowledge to implement the merge.) Thanks, Zhigang _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
George Dunlap
2013-Sep-16 10:04 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On Fri, Sep 13, 2013 at 5:04 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote:> Hi, > > As we talked in > http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a > suggestion: what about merging xl save/restore/migrate/migrate-receive? > > Here is the description: xl-migrate.rstThanks for bringing this up, but this thing with attaching files that actually contain your proposal isn''t really going to work. Can you please resend this with your proposals in-line, so that 1) the entire discussion can happen in the mail reader, rather than having to switch back and forth between an editor and a reader, 2) people can comment in-line on the proposals? Thanks, -George
Zhigang Wang
2013-Sep-16 15:51 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/16/2013 06:04 AM, George Dunlap wrote:> On Fri, Sep 13, 2013 at 5:04 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote: >> Hi, >> >> As we talked in >> http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a >> suggestion: what about merging xl save/restore/migrate/migrate-receive? >> >> Here is the description: xl-migrate.rst > > Thanks for bringing this up, but this thing with attaching files that > actually contain your proposal isn''t really going to work. Can you > please resend this with your proposals in-line, so that 1) the entire > discussion can happen in the mail reader, rather than having to switch > back and forth between an editor and a reader, 2) people can comment > in-line on the proposals? > > Thanks, > -George >Thanks George for the comment. Here it is: As we talked in http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a suggestion: what about merging xl save/restore/migrate/migrate-receive? Here is the description: xl-migrate.rst Also there is a workaround for the xl migration daemon example: xl-migrate-socat.rst Example patch for the workaround: xen-xl-migrate-socat.patch (I don''t have enough knowledge to implement the merge.) Thanks, Zhigang ---- xl-migrate.rst ---- =========XL Migrate ========= :Date: 2013-09-16 Current Status ============= * xl migrate leverages ssh/sshd. * In order to migrate a VM without user interactive, we have to configure ssh keys for all Servers in a pool. Key management brings complexity. * In certain cases, customers need non-ssl migrate, which greatly improves the migration speed. There''s no way to do it with ssh. * Current xl migrate command is not intuitive, especially the `-s` option:: # xl migrate Usage: xl [-v] migrate [options] <Domain> <host> Save a domain state to restore later. Options: -h Print this help. -C <config> Send <config> instead of config file from creation. -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed to sh. If empty, run <host> instead of ssh <host> xl migrate-receive [-d -e] -e Do not wait in the background (on <host>) for the death of the domain. It''s a little hard to adapt other tools as transport. * We have differnt implementation for `xl save/restore` and `xl migrate/migrate-receive`. Can we merge them? Proposal ======= * Implement dedicated daemons for ssl and non-ssl migration receive (`socat <http://www.dest-unreach.org/socat/>`_ can be used). Example patch for dedicated migrate receive daemon: xen-xl-migrate-socat.patch `socat` will call `xl restore` instead of `xl migrate-receive` after the following change. * Merge `xl migrate/migrate-receive` to `xl save/restore`: - To save a VM:: # xl save [-c] <domain> -f vm.chk Or:: # xl save [-c] <domain> >vm.chk - To restore a VM:: # xl restore -f vm.chk Or:: # cat vm.chk | xl restore - To migrate a VM using ssh/sshd:: # xl save -c <domain> | ssh root@<remote-host> xl restore We can implement a wrapper to make `xl migrate <domain> <remote-host>` to call the above command. - To migrate a VM using dedicated migrate receive daemon:: # xl save -c <domain> | socat - TCP:<remote-host>:8004" Or with SSL:: # xl save -c <domain> | socat - OPENSSL:<remote-host>:8005,verify=0 - Localhost migration:: # xl save <domain> | xl restrore - Localhost live migration:: # xl save -c <domain> | xl restrore Patch: I don''t have the capability and time to implement it yet. ---- xl-migrate-socat.rst ---- =========XL Migrate ========= :Date: 2013-09-16 Current Status ============= * xl migrate leverages ssh/sshd:: xl migrate <domain> <host> * In order to migrate a VM without user interactive, we have to configure ssh keys for all Servers in a pool. Key management with dynamic Server Pools is error prone. * In certain cases, customers need non-ssl migrate, which greatly improves the migration speed. There''s no way to do it with ssh. Proposal ======= * Implement dedicated daemons for ssl and non-ssl migration receive. * `socat <http://www.dest-unreach.org/socat/>`_ can be used. * Patch: xen-xl-migrate-socat.patch Testing ====== * Install the new Xen. * Start the services:: service xl-migrate-received start service xl-ssl-migrate-received start * Start a VM:: xl create vm.cfg * Migrate:: xl migrate -s xl-migrate-command <domid> localhost * SSL migrate:: xl migrate -s xl-ssl-migrate-command <domid> localhost * Speed for PV guest with 1024M memory: - ssh: 45s - socat: 9s - socat ssl: 57s ---- xen-xl-migrate-socat.patch ---- diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile index 47655f6..586728b 100644 --- a/tools/hotplug/Linux/Makefile +++ b/tools/hotplug/Linux/Makefile @@ -55,6 +55,8 @@ endif $(INSTALL_PROG) $(XENCOMMONS_INITD) $(DESTDIR)$(INITD_DIR) $(INSTALL_DATA) $(XENCOMMONS_SYSCONFIG) $(DESTDIR)$(SYSCONFIG_DIR)/xencommons $(INSTALL_PROG) init.d/xen-watchdog $(DESTDIR)$(INITD_DIR) + $(INSTALL_PROG) init.d/xl-migrate-received $(DESTDIR)$(INITD_DIR) + $(INSTALL_PROG) init.d/xl-ssl-migrate-received $(DESTDIR)$(INITD_DIR) .PHONY: install-scripts install-scripts: diff --git a/tools/hotplug/Linux/init.d/xl-migrate-received b/tools/hotplug/Linux/init.d/xl-migrate-received new file mode 100644 index 0000000..d67cf7d --- /dev/null +++ b/tools/hotplug/Linux/init.d/xl-migrate-received @@ -0,0 +1,106 @@ +#!/bin/bash +# +# xl-migrate-received: xl migrate receive daemon +# +# chkconfig: 2345 80 20 +# description: xl migrate receive daemon + +. /etc/rc.d/init.d/functions + +start() { + socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + action $"Starting xl migrate receive daemon: " true + touch /var/lock/subsys/xl-migrate-received + else + action $"Starting xl migrate receive daemon: " false + fi +} + +stop() { + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + action $"Stopping xl migrate receive daemon: " kill -9 $pid + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received +} + +status() { + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + echo $"xl migrate receive daemon is running" + else + echo $"xl migrate receive daemon is stopped" + fi +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + stop + start + ;; + status) + status + ;; + *) + echo $"Usage: $0 {start|stop|restart|status}" + exit 1 +esac +#!/bin/bash +# +# xl-migrate-received: xl migrate receive daemon +# +# chkconfig: 2345 80 20 +# description: xl migrate receive daemon + +. /etc/rc.d/init.d/functions + +start() { + socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + action $"Starting xl migrate receive daemon: " true + touch /var/lock/subsys/xl-migrate-received + else + action $"Starting xl migrate receive daemon: " false + fi +} + +stop() { + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + action $"Stopping xl migrate receive daemon: " kill -9 $pid + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received +} + +status() { + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + echo $"xl migrate receive daemon is running" + else + echo $"xl migrate receive daemon is stopped" + fi +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + stop + start + ;; + status) + status + ;; + *) + echo $"Usage: $0 {start|stop|restart|status}" + exit 1 +esac diff --git a/tools/hotplug/Linux/init.d/xl-ssl-migrate-received b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received new file mode 100644 index 0000000..272ed9c --- /dev/null +++ b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received @@ -0,0 +1,106 @@ +#!/bin/bash +# +# xl-ssl-migrate-received: xl ssl migrate receive daemon +# +# chkconfig: 2345 80 20 +# description: xl ssl migrate receive daemon + +. /etc/rc.d/init.d/functions + +start() { + socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + action $"Starting xl ssl migrate receive daemon: " true + touch /var/lock/subsys/xl-migrate-received + else + action $"Starting xl ssl migrate receive daemon: " false + fi +} + +stop() { + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received +} + +status() { + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + echo $"xl ssl migrate receive daemon is running" + else + echo $"xl ssl migrate receive daemon is stopped" + fi +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + stop + start + ;; + status) + status + ;; + *) + echo $"Usage: $0 {start|stop|restart|status}" + exit 1 +esac +#!/bin/bash +# +# xl-ssl-migrate-received: xl ssl migrate receive daemon +# +# chkconfig: 2345 80 20 +# description: xl ssl migrate receive daemon + +. /etc/rc.d/init.d/functions + +start() { + socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + action $"Starting xl ssl migrate receive daemon: " true + touch /var/lock/subsys/xl-migrate-received + else + action $"Starting xl ssl migrate receive daemon: " false + fi +} + +stop() { + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received +} + +status() { + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') + if [ -n "$pid" ]; then + echo $"xl ssl migrate receive daemon is running" + else + echo $"xl ssl migrate receive daemon is stopped" + fi +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + stop + start + ;; + status) + status + ;; + *) + echo $"Usage: $0 {start|stop|restart|status}" + exit 1 +esac diff --git a/tools/misc/Makefile b/tools/misc/Makefile index 59def7a..88aaf20 100644 --- a/tools/misc/Makefile +++ b/tools/misc/Makefile @@ -16,7 +16,7 @@ TARGETS := $(TARGETS-y) SUBDIRS := $(SUBDIRS-y) -INSTALL_BIN-y := xencons xencov_split +INSTALL_BIN-y := xencons xencov_split xl-migrate-command xl-ssl-migrate-command INSTALL_BIN-$(CONFIG_X86) += xen-detect INSTALL_BIN := $(INSTALL_BIN-y) diff --git a/tools/misc/xl-migrate-command b/tools/misc/xl-migrate-command new file mode 100644 index 0000000..0558ff9 --- /dev/null +++ b/tools/misc/xl-migrate-command @@ -0,0 +1,3 @@ +#!/bin/sh + +socat - TCP:$1:8004 diff --git a/tools/misc/xl-ssl-migrate-command b/tools/misc/xl-ssl-migrate-command new file mode 100644 index 0000000..628c554 --- /dev/null +++ b/tools/misc/xl-ssl-migrate-command @@ -0,0 +1,3 @@ +#!/bin/sh + +socat - OPENSSL:$1:8005,verify=0
George Dunlap
2013-Sep-16 16:05 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 16/09/13 16:51, Zhigang Wang wrote:> On 09/16/2013 06:04 AM, George Dunlap wrote: >> On Fri, Sep 13, 2013 at 5:04 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote: >>> Hi, >>> >>> As we talked in >>> http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a >>> suggestion: what about merging xl save/restore/migrate/migrate-receive? >>> >>> Here is the description: xl-migrate.rst >> Thanks for bringing this up, but this thing with attaching files that >> actually contain your proposal isn''t really going to work. Can you >> please resend this with your proposals in-line, so that 1) the entire >> discussion can happen in the mail reader, rather than having to switch >> back and forth between an editor and a reader, 2) people can comment >> in-line on the proposals? >> >> Thanks, >> -George >> > Thanks George for the comment. Here it is:Thanks.> * Merge `xl migrate/migrate-receive` to `xl save/restore`: > > - To save a VM:: > > # xl save [-c] <domain> -f vm.chk > > Or:: > > # xl save [-c] <domain> >vm.chk > > - To restore a VM:: > > # xl restore -f vm.chk > > Or:: > > # cat vm.chk | xl restore > > - To migrate a VM using ssh/sshd:: > > # xl save -c <domain> | ssh root@<remote-host> xl restoreI don''t necessarily mind *adding* an interface like this, if it can be made to work, but I definitely don''t think that we should be replacing "xl migrate" with this interface; this is too techy, and not really at all like the rest of the xl interface. But what I think would be better is to implement the two other transports you mention -- ssl and no encryption.> > We can implement a wrapper to make `xl migrate <domain> <remote-host>` to > call the above command. > > - To migrate a VM using dedicated migrate receive daemon:: > > # xl save -c <domain> | socat - TCP:<remote-host>:8004" > > Or with SSL:: > > # xl save -c <domain> | socat - OPENSSL:<remote-host>:8005,verify=0 > > - Localhost migration:: > > # xl save <domain> | xl restrore > > - Localhost live migration:: > > # xl save -c <domain> | xl restrore > > Patch: I don''t have the capability and time to implement it yet. > > > ---- xl-migrate-socat.rst ---- > > =========> XL Migrate > =========> > :Date: 2013-09-16 > > Current Status > =============> > * xl migrate leverages ssh/sshd:: > > xl migrate <domain> <host> > > * In order to migrate a VM without user interactive, we have to configure ssh > keys for all Servers in a pool. Key management with dynamic Server Pools is > error prone. > * In certain cases, customers need non-ssl migrate, which greatly improves the > migration speed. There''s no way to do it with ssh.Just to make sure I understand correctly then: you''re throwing authentication out the window, assuming that the host network is entirely trusted -- even when using ssl? -George
George Dunlap
2013-Sep-16 16:07 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 16/09/13 17:05, George Dunlap wrote:>> =========>> XL Migrate >> =========>> >> :Date: 2013-09-16 >> >> Current Status >> =============>> >> * xl migrate leverages ssh/sshd:: >> >> xl migrate <domain> <host> >> >> * In order to migrate a VM without user interactive, we have to >> configure ssh >> keys for all Servers in a pool. Key management with dynamic Server >> Pools is >> error prone. >> * In certain cases, customers need non-ssl migrate, which greatly >> improves the >> migration speed. There''s no way to do it with ssh. > > Just to make sure I understand correctly then: you''re throwing > authentication out the window, assuming that the host network is > entirely trusted -- even when using ssl?FWIW, I think XenServer (via xapi) manage to do the host ssh keys for a pool all right. But having an option to do the migration unencrypted, if you trust your network, seems like it might be a worthwhile option. -George
Ian Jackson
2013-Sep-16 16:20 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl
save/restore/migrate/migrate-receive"):> ---- xl-migrate.rst ----
...> * Current xl migrate command is not intuitive, especially the `-s` option::
>
> # xl migrate
> Usage: xl [-v] migrate [options] <Domain> <host>
>
> Save a domain state to restore later.
>
> Options:
>
> -h Print this help.
> -C <config> Send <config> instead of config file from
creation.
> -s <sshcommand> Use <sshcommand> instead of ssh. String
will be passed
> to sh. If empty, run <host> instead of ssh
<host> xl
> migrate-receive [-d -e]
> -e Do not wait in the background (on <host>) for
the death
> of the domain.
>
> It''s a little hard to adapt other tools as transport.
Perhaps the documentation needs to be improved. But you can just say
xl migrate -s '''' 42 ''nc remotehost 1234''
and in the receiving host''s inetd.conf:
1234 stream tcp nowait root /usr/bin/xl xl migrate-receive
(NB I haven''t tested this). If you want better logging then use a
better superserver than inetd.
> * We have differnt implementation for `xl save/restore` and
> `xl migrate/migrate-receive`. Can we merge them?
I''m afraid not. The migration protocol includes a confirmation that
the receiver is ready, to try to reduce the chance that a failed
migration ends up killing the domain.
> Proposal
> =======>
> * Implement dedicated daemons for ssl and non-ssl migration receive
> (`socat <http://www.dest-unreach.org/socat/>`_ can be used).
>
> Example patch for dedicated migrate receive daemon:
> xen-xl-migrate-socat.patch
I think a one-line change to inetd.conf is probably better. Your
script is very complicated (and still throws away the error messages
from xl migrate-receive rather than logging them).
As for the encrypted version: ssl has pretty awful security
properties, at least by default, which you need to work around. For
example, the default usually involves the X.509 root certificate
oligopoly, and doesn''t provide forward secrecy. If you need
encryption, ssh has a much better security model.
If you don''t need encryption and authentication then default mode of
use for xl is rather heavyweight and you might want to use a simple
unencrypted unauthenticated TCP session as I describe above.
> * In order to migrate a VM without user interactive, we have to configure
ssh
> keys for all Servers in a pool. Key management brings complexity.
Surely your automated server deployment system can manage this ?
Ian.
George Dunlap
2013-Sep-16 16:40 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 16/09/13 17:20, Ian Jackson wrote:> Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >> ---- xl-migrate.rst ---- > ... >> * Current xl migrate command is not intuitive, especially the `-s` option:: >> >> # xl migrate >> Usage: xl [-v] migrate [options] <Domain> <host> >> >> Save a domain state to restore later. >> >> Options: >> >> -h Print this help. >> -C <config> Send <config> instead of config file from creation. >> -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed >> to sh. If empty, run <host> instead of ssh <host> xl >> migrate-receive [-d -e] >> -e Do not wait in the background (on <host>) for the death >> of the domain. >> >> It''s a little hard to adapt other tools as transport. > Perhaps the documentation needs to be improved. But you can just say > xl migrate -s '''' 42 ''nc remotehost 1234''Actually, that''s a pretty bizarre interface -- I don''t think I would have gotten that from the help. If we had a new "transport" option that could take format strings, we could even set the default transport in xl.conf, like so: migrate.default.transport="myssh %h %r" migrate.default.transport="nc %h 1234" migrate.default.transport="socat - OPENSSL:%h:8005,verify=0" (Where %h would be the remote host, and %r the remote command to execute -- i.e., xl recieve.) Or of course: xl migrate -t "nc %h 1234" 42 remotehost Zhigang, would that work better for you guys? -George
Ian Jackson
2013-Sep-16 17:06 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
George Dunlap writes ("Re: [Xen-devel] Suggestion for merging xl
save/restore/migrate/migrate-receive"):> On 16/09/13 17:20, Ian Jackson wrote:
> > Perhaps the documentation needs to be improved. But you can just say
> > xl migrate -s '''' 42 ''nc remotehost
1234''
>
> Actually, that''s a pretty bizarre interface -- I don''t
think I would
> have gotten that from the help.
Yes, sorry. It was intended to be an escape hatch to make everything
possible, not the main interface.
> If we had a new "transport" option that could take format
strings, we
> could even set the default transport in xl.conf, like so:
>
> migrate.default.transport="myssh %h %r"
> migrate.default.transport="nc %h 1234"
> migrate.default.transport="socat - OPENSSL:%h:8005,verify=0"
For the 2nd of these I would be inclined to implement something like
xl migrate -t inet 42 remotehost
with xl making the networking calls itself.
Ian.
Zhigang Wang
2013-Sep-16 17:21 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/16/2013 12:40 PM, George Dunlap wrote:> On 16/09/13 17:20, Ian Jackson wrote: >> Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >>> ---- xl-migrate.rst ---- >> ... >>> * Current xl migrate command is not intuitive, especially the `-s` option:: >>> >>> # xl migrate >>> Usage: xl [-v] migrate [options] <Domain> <host> >>> >>> Save a domain state to restore later. >>> >>> Options: >>> >>> -h Print this help. >>> -C <config> Send <config> instead of config file from creation. >>> -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed >>> to sh. If empty, run <host> instead of ssh <host> xl >>> migrate-receive [-d -e] >>> -e Do not wait in the background (on <host>) for the death >>> of the domain. >>> >>> It''s a little hard to adapt other tools as transport. >> Perhaps the documentation needs to be improved. But you can just say >> xl migrate -s '''' 42 ''nc remotehost 1234'' > > Actually, that''s a pretty bizarre interface -- I don''t think I would > have gotten that from the help. > > If we had a new "transport" option that could take format strings, we > could even set the default transport in xl.conf, like so: > > migrate.default.transport="myssh %h %r" > > migrate.default.transport="nc %h 1234" > > migrate.default.transport="socat - OPENSSL:%h:8005,verify=0" > > (Where %h would be the remote host, and %r the remote command to execute > -- i.e., xl recieve.) > > Or of course: > > xl migrate -t "nc %h 1234" 42 remotehost > > Zhigang, would that work better for you guys?Yes. I think it''s a improvement to the -s option anyways. I was thinking the same thing when I work on the example. With this, the example doesn''t need the customized command any more (xl-migrate-command/xl-ssl-migrate-command). (But haven''t decided how to do the migration yet; the socat just a example) Thanks, Zhigang
Zhigang Wang
2013-Sep-16 17:41 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/16/2013 12:20 PM, Ian Jackson wrote:> Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >> ---- xl-migrate.rst ---- > ... >> * Current xl migrate command is not intuitive, especially the `-s` option:: >> >> # xl migrate >> Usage: xl [-v] migrate [options] <Domain> <host> >> >> Save a domain state to restore later. >> >> Options: >> >> -h Print this help. >> -C <config> Send <config> instead of config file from creation. >> -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed >> to sh. If empty, run <host> instead of ssh <host> xl >> migrate-receive [-d -e] >> -e Do not wait in the background (on <host>) for the death >> of the domain. >> >> It''s a little hard to adapt other tools as transport. > > Perhaps the documentation needs to be improved. But you can just say > xl migrate -s '''' 42 ''nc remotehost 1234'' > and in the receiving host''s inetd.conf: > 1234 stream tcp nowait root /usr/bin/xl xl migrate-receive > (NB I haven''t tested this). If you want better logging then use a > better superserver than inetd. > >> * We have differnt implementation for `xl save/restore` and >> `xl migrate/migrate-receive`. Can we merge them? > > I''m afraid not. The migration protocol includes a confirmation that > the receiver is ready, to try to reduce the chance that a failed > migration ends up killing the domain. > >> Proposal >> =======>> >> * Implement dedicated daemons for ssl and non-ssl migration receive >> (`socat <http://www.dest-unreach.org/socat/>`_ can be used). >> >> Example patch for dedicated migrate receive daemon: >> xen-xl-migrate-socat.patch > > I think a one-line change to inetd.conf is probably better. Your > script is very complicated (and still throws away the error messages > from xl migrate-receive rather than logging them). > > As for the encrypted version: ssl has pretty awful security > properties, at least by default, which you need to work around. For > example, the default usually involves the X.509 root certificate > oligopoly, and doesn''t provide forward secrecy. If you need > encryption, ssh has a much better security model. > > If you don''t need encryption and authentication then default mode of > use for xl is rather heavyweight and you might want to use a simple > unencrypted unauthenticated TCP session as I describe above. > >> * In order to migrate a VM without user interactive, we have to configure ssh >> keys for all Servers in a pool. Key management brings complexity. > > Surely your automated server deployment system can manage this ?Yes, we can. keys are states; we need to make sure they are always sync. Also after this, all Servers in a pool can login to each other. I don''t know whether it''s a security issue for our product. This is something we try to avoid at this time. Thanks, Zhigang
Ian Campbell
2013-Sep-16 20:42 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On Mon, 2013-09-16 at 13:41 -0400, Zhigang Wang wrote:> keys are states; we need to make sure they are always sync. Also after > this, all Servers in a pool can login to each other. I don''t know > whether it''sa security issue for our product.FYI a ssh key can be limited to only allow certain commands to be run, by using the "command=" syntax in authorized_keys. Ian.
Zhigang Wang
2013-Sep-16 20:51 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/16/2013 04:42 PM, Ian Campbell wrote:> On Mon, 2013-09-16 at 13:41 -0400, Zhigang Wang wrote: >> keys are states; we need to make sure they are always sync. Also after >> this, all Servers in a pool can login to each other. I don''t know >> whether it''sa security issue for our product. > > FYI a ssh key can be limited to only allow certain commands to be run, > by using the "command=" syntax in authorized_keys.Thanks Ian. I don''t know this yet. I will try this and consider this. Zhigang
George Dunlap
2013-Sep-17 08:25 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/16/2013 06:41 PM, Zhigang Wang wrote:> On 09/16/2013 12:20 PM, Ian Jackson wrote: >> Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >>> ---- xl-migrate.rst ---- >> ... >>> * Current xl migrate command is not intuitive, especially the `-s` option:: >>> >>> # xl migrate >>> Usage: xl [-v] migrate [options] <Domain> <host> >>> >>> Save a domain state to restore later. >>> >>> Options: >>> >>> -h Print this help. >>> -C <config> Send <config> instead of config file from creation. >>> -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed >>> to sh. If empty, run <host> instead of ssh <host> xl >>> migrate-receive [-d -e] >>> -e Do not wait in the background (on <host>) for the death >>> of the domain. >>> >>> It''s a little hard to adapt other tools as transport. >> >> Perhaps the documentation needs to be improved. But you can just say >> xl migrate -s '''' 42 ''nc remotehost 1234'' >> and in the receiving host''s inetd.conf: >> 1234 stream tcp nowait root /usr/bin/xl xl migrate-receive >> (NB I haven''t tested this). If you want better logging then use a >> better superserver than inetd. >> >>> * We have differnt implementation for `xl save/restore` and >>> `xl migrate/migrate-receive`. Can we merge them? >> >> I''m afraid not. The migration protocol includes a confirmation that >> the receiver is ready, to try to reduce the chance that a failed >> migration ends up killing the domain. >> >>> Proposal >>> =======>>> >>> * Implement dedicated daemons for ssl and non-ssl migration receive >>> (`socat <http://www.dest-unreach.org/socat/>`_ can be used). >>> >>> Example patch for dedicated migrate receive daemon: >>> xen-xl-migrate-socat.patch >> >> I think a one-line change to inetd.conf is probably better. Your >> script is very complicated (and still throws away the error messages >> from xl migrate-receive rather than logging them). >> >> As for the encrypted version: ssl has pretty awful security >> properties, at least by default, which you need to work around. For >> example, the default usually involves the X.509 root certificate >> oligopoly, and doesn''t provide forward secrecy. If you need >> encryption, ssh has a much better security model. >> >> If you don''t need encryption and authentication then default mode of >> use for xl is rather heavyweight and you might want to use a simple >> unencrypted unauthenticated TCP session as I describe above. >> >>> * In order to migrate a VM without user interactive, we have to configure ssh >>> keys for all Servers in a pool. Key management brings complexity. >> >> Surely your automated server deployment system can manage this ? > > Yes, we can. > > keys are states; we need to make sure they are always sync. Also after this, > all Servers in a pool can login to each other. I don''t know whether it''s > a security issue for our product. > > This is something we try to avoid at this time....so instead of allowing anyone on one of the hosts log in, you''re going to allow anyone with access to the network to create a VM without any kind of authentication? From a security perspective, that doesn''t really sound like an improvement... -George
Ian Jackson
2013-Sep-17 09:26 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
George Dunlap writes ("Re: [Xen-devel] Suggestion for merging xl
save/restore/migrate/migrate-receive"):> On 09/16/2013 06:41 PM, Zhigang Wang wrote:
> > ... Also after this, all Servers in a pool can login to each
> > other. I don''t know whether it''s a security issue
for our product.
> >
> > This is something we try to avoid at this time.
>
> ...so instead of allowing anyone on one of the hosts log in,
you''re
> going to allow anyone with access to the network to create a VM without
> any kind of authentication?
>
> From a security perspective, that doesn''t really sound like an
> improvement...
Note that if host B allows incoming migrations from host A, then host
B is trusting host A completely. This is because the migration data
contains not just the guest''s state (which is of course encapsulated
inside the Xen VM security boundary), but also the VM configuration.
The VM configuration specifies the mapping between guest resources and
host resources.
So host B trusts host A to specify the correct set of host B''s own
resources to expose to the guest VM. If host A is malicious it can
send a VM whose configuration specifies (for example) that the whole
of host B''s disk is to be exposed to the guest, along with a guest
which will make whatever malicious changes host A desires.
In summary: accepting incoming migration images is just as dangerous
as allowing root login (from the same source host). So switching the
transport from ssh to unauthenticated ssl makes the security against
malicious migration source hosts strictly worse.
The only way unauthenticated ssl is better than simply unauthenticated
unencrypted TCP is protection against passive eavesdropping. This is
important for much general traffic on the public Internet (see recent
revelations about widespread eavesdropping), but probably not relevant
for the control plane of a VM hosting setup. If your control plane
network has bad people on it, you need authentication as well as
encryption.
So I don''t think we should be adding new code to xl which might
encourage the use of ssl. The proposed format-string based template
would be OK, but I think really that we should have better (more
convenient) support for unencrypted migration.
Things that would be helpful:
* An option to xl migrate which causes xl to make the TCP connection
itself. This is a not-quite-trivial SMOP and the specification
ought to be trivial.
* A separate executable (or perhaps argv[0] mode)
"xl-migrate-receive" so that the hosts.{allow,deny} etc. files
used by tcpd can contain "xl-migrate-receive" and not just
"xl".
The specification for this would need to be discussed, but the
implementation will be trivial.
* A command line option for logging redirection so that /all/ the
error messages from an inetd-launched xl migrate-receive go
somewhere useful. The specification for this would need to be
discussed, but the implementation should be very simple.
* Better documentation, particularly including a recipe for setting
this up, covering: inetd.conf, hosts.{allow,deny}, invocation at
the sending end, security considerations.
Ian.
George Dunlap
2013-Sep-17 10:07 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/17/2013 10:26 AM, Ian Jackson wrote:> George Dunlap writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >> On 09/16/2013 06:41 PM, Zhigang Wang wrote: >>> ... Also after this, all Servers in a pool can login to each >>> other. I don''t know whether it''s a security issue for our product. >>> >>> This is something we try to avoid at this time. >> >> ...so instead of allowing anyone on one of the hosts log in, you''re >> going to allow anyone with access to the network to create a VM without >> any kind of authentication? >> >> From a security perspective, that doesn''t really sound like an >> improvement... > > Note that if host B allows incoming migrations from host A, then host > B is trusting host A completely. This is because the migration data > contains not just the guest''s state (which is of course encapsulated > inside the Xen VM security boundary), but also the VM configuration. > The VM configuration specifies the mapping between guest resources and > host resources. > > So host B trusts host A to specify the correct set of host B''s own > resources to expose to the guest VM. If host A is malicious it can > send a VM whose configuration specifies (for example) that the whole > of host B''s disk is to be exposed to the guest, along with a guest > which will make whatever malicious changes host A desires. > > In summary: accepting incoming migration images is just as dangerous > as allowing root login (from the same source host). So switching the > transport from ssh to unauthenticated ssl makes the security against > malicious migration source hosts strictly worse. > > The only way unauthenticated ssl is better than simply unauthenticated > unencrypted TCP is protection against passive eavesdropping. This is > important for much general traffic on the public Internet (see recent > revelations about widespread eavesdropping), but probably not relevant > for the control plane of a VM hosting setup. If your control plane > network has bad people on it, you need authentication as well as > encryption. > > > So I don''t think we should be adding new code to xl which might > encourage the use of ssl. The proposed format-string based template > would be OK, but I think really that we should have better (more > convenient) support for unencrypted migration. > > Things that would be helpful:And once we get all this sorted out, a blog post and/or wiki page with the issues, the options as they exist in the most recent release, and the options as they will exist in the next release, would be helpful. -George
George Dunlap
2013-Sep-17 10:28 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
create ^ title it xl migrate transport improvements thanks On Mon, Sep 16, 2013 at 4:51 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote:> On 09/16/2013 06:04 AM, George Dunlap wrote: >> On Fri, Sep 13, 2013 at 5:04 PM, Zhigang Wang <zhigang.x.wang@oracle.com> wrote: >>> Hi, >>> >>> As we talked in >>> http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a >>> suggestion: what about merging xl save/restore/migrate/migrate-receive? >>> >>> Here is the description: xl-migrate.rst >> >> Thanks for bringing this up, but this thing with attaching files that >> actually contain your proposal isn''t really going to work. Can you >> please resend this with your proposals in-line, so that 1) the entire >> discussion can happen in the mail reader, rather than having to switch >> back and forth between an editor and a reader, 2) people can comment >> in-line on the proposals? >> >> Thanks, >> -George >> > > Thanks George for the comment. Here it is: > > As we talked in > http://lists.xen.org/archives/html/xen-devel/2013-09/msg00211.html , I have a > suggestion: what about merging xl save/restore/migrate/migrate-receive? > > Here is the description: xl-migrate.rst > > Also there is a workaround for the xl migration daemon example: > xl-migrate-socat.rst > > Example patch for the workaround: xen-xl-migrate-socat.patch > > (I don''t have enough knowledge to implement the merge.) > > Thanks, > > Zhigang > > ---- xl-migrate.rst ---- > > =========> XL Migrate > =========> > :Date: 2013-09-16 > > Current Status > =============> > * xl migrate leverages ssh/sshd. > > * In order to migrate a VM without user interactive, we have to configure ssh > keys for all Servers in a pool. Key management brings complexity. > > * In certain cases, customers need non-ssl migrate, which greatly improves the > migration speed. There''s no way to do it with ssh. > > * Current xl migrate command is not intuitive, especially the `-s` option:: > > # xl migrate > Usage: xl [-v] migrate [options] <Domain> <host> > > Save a domain state to restore later. > > Options: > > -h Print this help. > -C <config> Send <config> instead of config file from creation. > -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed > to sh. If empty, run <host> instead of ssh <host> xl > migrate-receive [-d -e] > -e Do not wait in the background (on <host>) for the death > of the domain. > > It''s a little hard to adapt other tools as transport. > > * We have differnt implementation for `xl save/restore` and > `xl migrate/migrate-receive`. Can we merge them? > > > Proposal > =======> > * Implement dedicated daemons for ssl and non-ssl migration receive > (`socat <http://www.dest-unreach.org/socat/>`_ can be used). > > Example patch for dedicated migrate receive daemon: > xen-xl-migrate-socat.patch > > `socat` will call `xl restore` instead of `xl migrate-receive` after the > following change. > > * Merge `xl migrate/migrate-receive` to `xl save/restore`: > > - To save a VM:: > > # xl save [-c] <domain> -f vm.chk > > Or:: > > # xl save [-c] <domain> >vm.chk > > - To restore a VM:: > > # xl restore -f vm.chk > > Or:: > > # cat vm.chk | xl restore > > - To migrate a VM using ssh/sshd:: > > # xl save -c <domain> | ssh root@<remote-host> xl restore > > We can implement a wrapper to make `xl migrate <domain> <remote-host>` to > call the above command. > > - To migrate a VM using dedicated migrate receive daemon:: > > # xl save -c <domain> | socat - TCP:<remote-host>:8004" > > Or with SSL:: > > # xl save -c <domain> | socat - OPENSSL:<remote-host>:8005,verify=0 > > - Localhost migration:: > > # xl save <domain> | xl restrore > > - Localhost live migration:: > > # xl save -c <domain> | xl restrore > > Patch: I don''t have the capability and time to implement it yet. > > > ---- xl-migrate-socat.rst ---- > > =========> XL Migrate > =========> > :Date: 2013-09-16 > > Current Status > =============> > * xl migrate leverages ssh/sshd:: > > xl migrate <domain> <host> > > * In order to migrate a VM without user interactive, we have to configure ssh > keys for all Servers in a pool. Key management with dynamic Server Pools is > error prone. > > * In certain cases, customers need non-ssl migrate, which greatly improves the > migration speed. There''s no way to do it with ssh. > > > Proposal > =======> > * Implement dedicated daemons for ssl and non-ssl migration receive. > > * `socat <http://www.dest-unreach.org/socat/>`_ can be used. > > * Patch: xen-xl-migrate-socat.patch > > > Testing > ======> > * Install the new Xen. > > * Start the services:: > > service xl-migrate-received start > service xl-ssl-migrate-received start > > * Start a VM:: > > xl create vm.cfg > > * Migrate:: > > xl migrate -s xl-migrate-command <domid> localhost > > * SSL migrate:: > > xl migrate -s xl-ssl-migrate-command <domid> localhost > > * Speed for PV guest with 1024M memory: > > - ssh: 45s > - socat: 9s > - socat ssl: 57s > > > ---- xen-xl-migrate-socat.patch ---- > > diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile > index 47655f6..586728b 100644 > --- a/tools/hotplug/Linux/Makefile > +++ b/tools/hotplug/Linux/Makefile > @@ -55,6 +55,8 @@ endif > $(INSTALL_PROG) $(XENCOMMONS_INITD) $(DESTDIR)$(INITD_DIR) > $(INSTALL_DATA) $(XENCOMMONS_SYSCONFIG) $(DESTDIR)$(SYSCONFIG_DIR)/xencommons > $(INSTALL_PROG) init.d/xen-watchdog $(DESTDIR)$(INITD_DIR) > + $(INSTALL_PROG) init.d/xl-migrate-received $(DESTDIR)$(INITD_DIR) > + $(INSTALL_PROG) init.d/xl-ssl-migrate-received $(DESTDIR)$(INITD_DIR) > > .PHONY: install-scripts > install-scripts: > diff --git a/tools/hotplug/Linux/init.d/xl-migrate-received b/tools/hotplug/Linux/init.d/xl-migrate-received > new file mode 100644 > index 0000000..d67cf7d > --- /dev/null > +++ b/tools/hotplug/Linux/init.d/xl-migrate-received > @@ -0,0 +1,106 @@ > +#!/bin/bash > +# > +# xl-migrate-received: xl migrate receive daemon > +# > +# chkconfig: 2345 80 20 > +# description: xl migrate receive daemon > + > +. /etc/rc.d/init.d/functions > + > +start() { > + socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + action $"Starting xl migrate receive daemon: " true > + touch /var/lock/subsys/xl-migrate-received > + else > + action $"Starting xl migrate receive daemon: " false > + fi > +} > + > +stop() { > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + action $"Stopping xl migrate receive daemon: " kill -9 $pid > + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received > +} > + > +status() { > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + echo $"xl migrate receive daemon is running" > + else > + echo $"xl migrate receive daemon is stopped" > + fi > +} > + > +case "$1" in > + start) > + start > + ;; > + stop) > + stop > + ;; > + restart) > + stop > + start > + ;; > + status) > + status > + ;; > + *) > + echo $"Usage: $0 {start|stop|restart|status}" > + exit 1 > +esac > +#!/bin/bash > +# > +# xl-migrate-received: xl migrate receive daemon > +# > +# chkconfig: 2345 80 20 > +# description: xl migrate receive daemon > + > +. /etc/rc.d/init.d/functions > + > +start() { > + socat -ly TCP-LISTEN:8004,reuseaddr,fork EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + action $"Starting xl migrate receive daemon: " true > + touch /var/lock/subsys/xl-migrate-received > + else > + action $"Starting xl migrate receive daemon: " false > + fi > +} > + > +stop() { > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + action $"Stopping xl migrate receive daemon: " kill -9 $pid > + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received > +} > + > +status() { > + pid=$(ps -eo pid,args | grep "socat -ly TCP-LISTEN:8004" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + echo $"xl migrate receive daemon is running" > + else > + echo $"xl migrate receive daemon is stopped" > + fi > +} > + > +case "$1" in > + start) > + start > + ;; > + stop) > + stop > + ;; > + restart) > + stop > + start > + ;; > + status) > + status > + ;; > + *) > + echo $"Usage: $0 {start|stop|restart|status}" > + exit 1 > +esac > diff --git a/tools/hotplug/Linux/init.d/xl-ssl-migrate-received b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received > new file mode 100644 > index 0000000..272ed9c > --- /dev/null > +++ b/tools/hotplug/Linux/init.d/xl-ssl-migrate-received > @@ -0,0 +1,106 @@ > +#!/bin/bash > +# > +# xl-ssl-migrate-received: xl ssl migrate receive daemon > +# > +# chkconfig: 2345 80 20 > +# description: xl ssl migrate receive daemon > + > +. /etc/rc.d/init.d/functions > + > +start() { > + socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + action $"Starting xl ssl migrate receive daemon: " true > + touch /var/lock/subsys/xl-migrate-received > + else > + action $"Starting xl ssl migrate receive daemon: " false > + fi > +} > + > +stop() { > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid > + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received > +} > + > +status() { > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + echo $"xl ssl migrate receive daemon is running" > + else > + echo $"xl ssl migrate receive daemon is stopped" > + fi > +} > + > +case "$1" in > + start) > + start > + ;; > + stop) > + stop > + ;; > + restart) > + stop > + start > + ;; > + status) > + status > + ;; > + *) > + echo $"Usage: $0 {start|stop|restart|status}" > + exit 1 > +esac > +#!/bin/bash > +# > +# xl-ssl-migrate-received: xl ssl migrate receive daemon > +# > +# chkconfig: 2345 80 20 > +# description: xl ssl migrate receive daemon > + > +. /etc/rc.d/init.d/functions > + > +start() { > + socat -ly OPENSSL-LISTEN:8005,reuseaddr,fork,key=/etc/ovs-agent/cert/key.pem,cert=/etc/ovs-agent/cert/certificate.pem,verify=0 EXEC:"xl migrate-receive" >/dev/null 2>&1 </dev/null & > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + action $"Starting xl ssl migrate receive daemon: " true > + touch /var/lock/subsys/xl-migrate-received > + else > + action $"Starting xl ssl migrate receive daemon: " false > + fi > +} > + > +stop() { > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + action $"Stopping xl ssl migrate receive daemon: " kill -9 $pid > + [ $? -eq 0 ] && rm -f /var/lock/subsys/xl-migrate-received > +} > + > +status() { > + pid=$(ps -eo pid,args | grep "socat -ly OPENSSL-LISTEN:8005" | grep -v grep | awk ''{print $1}'') > + if [ -n "$pid" ]; then > + echo $"xl ssl migrate receive daemon is running" > + else > + echo $"xl ssl migrate receive daemon is stopped" > + fi > +} > + > +case "$1" in > + start) > + start > + ;; > + stop) > + stop > + ;; > + restart) > + stop > + start > + ;; > + status) > + status > + ;; > + *) > + echo $"Usage: $0 {start|stop|restart|status}" > + exit 1 > +esac > diff --git a/tools/misc/Makefile b/tools/misc/Makefile > index 59def7a..88aaf20 100644 > --- a/tools/misc/Makefile > +++ b/tools/misc/Makefile > @@ -16,7 +16,7 @@ TARGETS := $(TARGETS-y) > > SUBDIRS := $(SUBDIRS-y) > > -INSTALL_BIN-y := xencons xencov_split > +INSTALL_BIN-y := xencons xencov_split xl-migrate-command xl-ssl-migrate-command > INSTALL_BIN-$(CONFIG_X86) += xen-detect > INSTALL_BIN := $(INSTALL_BIN-y) > > diff --git a/tools/misc/xl-migrate-command b/tools/misc/xl-migrate-command > new file mode 100644 > index 0000000..0558ff9 > --- /dev/null > +++ b/tools/misc/xl-migrate-command > @@ -0,0 +1,3 @@ > +#!/bin/sh > + > +socat - TCP:$1:8004 > diff --git a/tools/misc/xl-ssl-migrate-command b/tools/misc/xl-ssl-migrate-command > new file mode 100644 > index 0000000..628c554 > --- /dev/null > +++ b/tools/misc/xl-ssl-migrate-command > @@ -0,0 +1,3 @@ > +#!/bin/sh > + > +socat - OPENSSL:$1:8005,verify=0 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
xen@bugs.xenproject.org
2013-Sep-17 10:45 UTC
Processed: Re: Suggestion for merging xl save/restore/migrate/migrate-receive
Processing commands for xen@bugs.xenproject.org:> create ^Created new bug #19 rooted at `<5237291C.9090100@oracle.com>'' Title: `Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive''> title it xl migrate transport improvementsSet title for #19 to `xl migrate transport improvements''> thanksFinished processing. Modified/created Bugs: - 19: http://bugs.xenproject.org/xen/bug/19 (new) --- Xen Hypervisor Bug Tracker See http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen for information on reporting bugs Contact xen-bugs-owner@bugs.xenproject.org with any infrastructure issues
Zhigang Wang
2013-Sep-17 13:44 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 09/17/2013 05:26 AM, Ian Jackson wrote:> George Dunlap writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): >> On 09/16/2013 06:41 PM, Zhigang Wang wrote: >>> ... Also after this, all Servers in a pool can login to each >>> other. I don''t know whether it''s a security issue for our product. >>> >>> This is something we try to avoid at this time. >> >> ...so instead of allowing anyone on one of the hosts log in, you''re >> going to allow anyone with access to the network to create a VM without >> any kind of authentication? >> >> From a security perspective, that doesn''t really sound like an >> improvement... > > Note that if host B allows incoming migrations from host A, then host > B is trusting host A completely. This is because the migration data > contains not just the guest''s state (which is of course encapsulated > inside the Xen VM security boundary), but also the VM configuration. > The VM configuration specifies the mapping between guest resources and > host resources. > > So host B trusts host A to specify the correct set of host B''s own > resources to expose to the guest VM. If host A is malicious it can > send a VM whose configuration specifies (for example) that the whole > of host B''s disk is to be exposed to the guest, along with a guest > which will make whatever malicious changes host A desires. > > In summary: accepting incoming migration images is just as dangerous > as allowing root login (from the same source host). So switching the > transport from ssh to unauthenticated ssl makes the security against > malicious migration source hosts strictly worse. > > The only way unauthenticated ssl is better than simply unauthenticated > unencrypted TCP is protection against passive eavesdropping. This is > important for much general traffic on the public Internet (see recent > revelations about widespread eavesdropping), but probably not relevant > for the control plane of a VM hosting setup. If your control plane > network has bad people on it, you need authentication as well as > encryption. > > > So I don''t think we should be adding new code to xl which might > encourage the use of ssl. The proposed format-string based template > would be OK, but I think really that we should have better (more > convenient) support for unencrypted migration. > > Things that would be helpful: > > * An option to xl migrate which causes xl to make the TCP connection > itself. This is a not-quite-trivial SMOP and the specification > ought to be trivial. > > * A separate executable (or perhaps argv[0] mode) > "xl-migrate-receive" so that the hosts.{allow,deny} etc. files > used by tcpd can contain "xl-migrate-receive" and not just "xl". > The specification for this would need to be discussed, but the > implementation will be trivial. > > * A command line option for logging redirection so that /all/ the > error messages from an inetd-launched xl migrate-receive go > somewhere useful. The specification for this would need to be > discussed, but the implementation should be very simple. > > * Better documentation, particularly including a recipe for setting > this up, covering: inetd.conf, hosts.{allow,deny}, invocation at > the sending end, security considerations.Thanks for the good explanation. We really want to use the upstream solution. I will try the ssh solution first. Please go ahead and implement the non-security solution. I can help for testing. Zhigang
Konrad Rzeszutek Wilk
2013-Sep-24 16:46 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On Tue, Sep 17, 2013 at 09:25:12AM +0100, George Dunlap wrote:> On 09/16/2013 06:41 PM, Zhigang Wang wrote: > >On 09/16/2013 12:20 PM, Ian Jackson wrote: > >>Zhigang Wang writes ("Re: [Xen-devel] Suggestion for merging xl save/restore/migrate/migrate-receive"): > >>>---- xl-migrate.rst ---- > >>... > >>>* Current xl migrate command is not intuitive, especially the `-s` option:: > >>> > >>> # xl migrate > >>> Usage: xl [-v] migrate [options] <Domain> <host> > >>> > >>> Save a domain state to restore later. > >>> > >>> Options: > >>> > >>> -h Print this help. > >>> -C <config> Send <config> instead of config file from creation. > >>> -s <sshcommand> Use <sshcommand> instead of ssh. String will be passed > >>> to sh. If empty, run <host> instead of ssh <host> xl > >>> migrate-receive [-d -e] > >>> -e Do not wait in the background (on <host>) for the death > >>> of the domain. > >>> > >>> It''s a little hard to adapt other tools as transport. > >> > >>Perhaps the documentation needs to be improved. But you can just say > >> xl migrate -s '''' 42 ''nc remotehost 1234'' > >>and in the receiving host''s inetd.conf: > >> 1234 stream tcp nowait root /usr/bin/xl xl migrate-receive > >>(NB I haven''t tested this). If you want better logging then use a > >>better superserver than inetd. > >> > >>>* We have differnt implementation for `xl save/restore` and > >>> `xl migrate/migrate-receive`. Can we merge them? > >> > >>I''m afraid not. The migration protocol includes a confirmation that > >>the receiver is ready, to try to reduce the chance that a failed > >>migration ends up killing the domain. > >> > >>>Proposal > >>>=======> >>> > >>>* Implement dedicated daemons for ssl and non-ssl migration receive > >>> (`socat <http://www.dest-unreach.org/socat/>`_ can be used). > >>> > >>> Example patch for dedicated migrate receive daemon: > >>> xen-xl-migrate-socat.patch > >> > >>I think a one-line change to inetd.conf is probably better. Your > >>script is very complicated (and still throws away the error messages > >>from xl migrate-receive rather than logging them). > >> > >>As for the encrypted version: ssl has pretty awful security > >>properties, at least by default, which you need to work around. For > >>example, the default usually involves the X.509 root certificate > >>oligopoly, and doesn''t provide forward secrecy. If you need > >>encryption, ssh has a much better security model. > >> > >>If you don''t need encryption and authentication then default mode of > >>use for xl is rather heavyweight and you might want to use a simple > >>unencrypted unauthenticated TCP session as I describe above. > >> > >>>* In order to migrate a VM without user interactive, we have to configure ssh > >>> keys for all Servers in a pool. Key management brings complexity. > >> > >>Surely your automated server deployment system can manage this ? > > > >Yes, we can. > > > >keys are states; we need to make sure they are always sync. Also after this, > >all Servers in a pool can login to each other. I don''t know whether it''s > >a security issue for our product. > > > >This is something we try to avoid at this time. > > ...so instead of allowing anyone on one of the hosts log in, you''re > going to allow anyone with access to the network to create a VM > without any kind of authentication? > > From a security perspective, that doesn''t really sound like an > improvement... >How did this work with ''xend'' and its migration using SSL? Was it as simple as this ? Thanks.> -George > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
George Dunlap
2013-Sep-25 10:06 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On Tue, Sep 24, 2013 at 5:46 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> >>>* In order to migrate a VM without user interactive, we have to configure ssh >> >>> keys for all Servers in a pool. Key management brings complexity. >> >> >> >>Surely your automated server deployment system can manage this ? >> > >> >Yes, we can. >> > >> >keys are states; we need to make sure they are always sync. Also after this, >> >all Servers in a pool can login to each other. I don''t know whether it''s >> >a security issue for our product. >> > >> >This is something we try to avoid at this time. >> >> ...so instead of allowing anyone on one of the hosts log in, you''re >> going to allow anyone with access to the network to create a VM >> without any kind of authentication? >> >> From a security perspective, that doesn''t really sound like an >> improvement... >> > > How did this work with ''xend'' and its migration using SSL? Was it as > simple as this ?I have no idea -- Matt, do you know / would you care to take a look and find out (since you have expressed a willingness to maintain xend)? -George
Matt Wilson
2013-Oct-03 02:19 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On Wed, Sep 25, 2013 at 11:06:29AM +0100, George Dunlap wrote:> On Tue, Sep 24, 2013 at 5:46 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > >> >>>* In order to migrate a VM without user interactive, we have to configure ssh > >> >>> keys for all Servers in a pool. Key management brings complexity. > >> >> > >> >>Surely your automated server deployment system can manage this ? > >> > > >> >Yes, we can. > >> > > >> >keys are states; we need to make sure they are always sync. Also after this, > >> >all Servers in a pool can login to each other. I don''t know whether it''s > >> >a security issue for our product. > >> > > >> >This is something we try to avoid at this time. > >> > >> ...so instead of allowing anyone on one of the hosts log in, you''re > >> going to allow anyone with access to the network to create a VM > >> without any kind of authentication? > >> > >> From a security perspective, that doesn''t really sound like an > >> improvement... > >> > > > > How did this work with ''xend'' and its migration using SSL? Was it as > > simple as this ? > > I have no idea -- Matt, do you know / would you care to take a look > and find out (since you have expressed a willingness to maintain > xend)?It seems that you would just configure a ssl key file and cert file in xend-config.sxp http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=0f26d15c Zhigang: you wrote this code, correct? --msw
Zhigang Wang
2013-Oct-03 13:34 UTC
Re: Suggestion for merging xl save/restore/migrate/migrate-receive
On 10/02/2013 10:19 PM, Matt Wilson wrote:> On Wed, Sep 25, 2013 at 11:06:29AM +0100, George Dunlap wrote: >> On Tue, Sep 24, 2013 at 5:46 PM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >>>>>>> * In order to migrate a VM without user interactive, we have to configure ssh >>>>>>> keys for all Servers in a pool. Key management brings complexity. >>>>>> >>>>>> Surely your automated server deployment system can manage this ? >>>>> >>>>> Yes, we can. >>>>> >>>>> keys are states; we need to make sure they are always sync. Also after this, >>>>> all Servers in a pool can login to each other. I don''t know whether it''s >>>>> a security issue for our product. >>>>> >>>>> This is something we try to avoid at this time. >>>> >>>> ...so instead of allowing anyone on one of the hosts log in, you''re >>>> going to allow anyone with access to the network to create a VM >>>> without any kind of authentication? >>>> >>>> From a security perspective, that doesn''t really sound like an >>>> improvement... >>>> >>> >>> How did this work with ''xend'' and its migration using SSL? Was it as >>> simple as this ? >> >> I have no idea -- Matt, do you know / would you care to take a look >> and find out (since you have expressed a willingness to maintain >> xend)? > > It seems that you would just configure a ssl key file and cert file in > xend-config.sxp > > http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=0f26d15c > > Zhigang: you wrote this code, correct?Yes. That''s only a very basic implementation. The SSL relocation server will not do client cert verification and there''s no way to configure the client to use specific cert right now. I think SSL cert verification could be a way for security. But you need distribute the certs to all the servers in a pool and reload xend relocation server to use the new certificate. Thanks, Zhigang