Xen devel - Jul 2009 - Can the qemu-xen git repository be moved to a ''git://'' protocol? It can save a lot of time...

Hi,

Currently, xen clones the qemu-xen using an ''http://''
protocol, and
this takes a lot of time.  It can be improved a lot if it will be
moved to be shared by a ''git://'' protocol, which is much
faster.
Can this be done?

Tom

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tom Rotenberg writes ("[Xen-devel] Can the qemu-xen git repository be moved
to a ''git://'' protocol? It can save a lot of
time..."):
> Currently, xen clones the qemu-xen using an ''http://''
protocol, and
> this takes a lot of time.  It can be improved a lot if it will be
> moved to be shared by a ''git://'' protocol, which is much
faster.
> Can this be done?
Yes.  We were using http:// because we didn''t have a git server on
xenbits, but now we do.

Keir: this patch is for xen-unstable but given that the git daemon is
a property of the xenbits system configuration rather of the specific
branch, it wouldn''t be unreasonable to apply it to the older trees
too.

Ian.

use git smart server protocol for qemu tree by default

xenbits supports the git protocol now, which is much faster (and
produces less noise in the build log too).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff -r e07726c03d31 Config.mk
--- a/Config.mk	Thu Jul 16 10:32:55 2009 +0100
+++ b/Config.mk	Thu Jul 16 14:58:49 2009 +0100
@@ -141,7 +141,7 @@ XEN_EXTFILES_URL=http://xenbits.xensourc
 # the internet.  The original download URL is preserved as a comment
 # near the place in the Xen Makefiles where the file is used.
 
-QEMU_REMOTE=http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
+QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
 
 # Specify which qemu-dm to use. This may be `ioemu'' to use the old
 # Mercurial in-tree version, or a local directory, or a git URL.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thursday, 16 July 2009 at 11:36, Tom Rotenberg wrote:
> Hi,
> 
> Currently, xen clones the qemu-xen using an ''http://''
protocol, and
> this takes a lot of time.  It can be improved a lot if it will be
> moved to be shared by a ''git://'' protocol, which is much
faster.
> Can this be done?
Why not have xen use an hg mirror of the qemu-xen tree? These are easy
to set up (it''s a one-line "hg convert" cron job), and it
would
simplify life for xen hackers to need only one obscure version control
system.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Ian,
git clone http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
and 
git clone git://xenbits.xensource.com/qemu-xen-unstable.git 

should always return the exactly same repository, correct?

I can''t access the latter since I''m behind a firewall. I can
only access the http interface.

Thanks,
-- Dexuan



-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Jackson
Sent: 2009?7?16? 22:02
To: xen-devel@lists.xensource.com
Cc: Tom Rotenberg
Subject: [Xen-devel] [PATCH] use git smart server protocol for qemu tree by
default

Tom Rotenberg writes ("[Xen-devel] Can the qemu-xen git repository be moved
to a ''git://'' protocol? It can save a lot of
time..."):
> Currently, xen clones the qemu-xen using an ''http://''
protocol, and
> this takes a lot of time.  It can be improved a lot if it will be
> moved to be shared by a ''git://'' protocol, which is much
faster.
> Can this be done?
Yes.  We were using http:// because we didn''t have a git server on
xenbits, but now we do.

Keir: this patch is for xen-unstable but given that the git daemon is
a property of the xenbits system configuration rather of the specific
branch, it wouldn''t be unreasonable to apply it to the older trees
too.

Ian.

use git smart server protocol for qemu tree by default

xenbits supports the git protocol now, which is much faster (and
produces less noise in the build log too).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff -r e07726c03d31 Config.mk
--- a/Config.mk	Thu Jul 16 10:32:55 2009 +0100
+++ b/Config.mk	Thu Jul 16 14:58:49 2009 +0100
@@ -141,7 +141,7 @@ XEN_EXTFILES_URL=http://xenbits.xensourc
 # the internet.  The original download URL is preserved as a comment
 # near the place in the Xen Makefiles where the file is used.
 
-QEMU_REMOTE=http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
+QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
 
 # Specify which qemu-dm to use. This may be `ioemu'' to use the old
 # Mercurial in-tree version, or a local directory, or a git URL.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

It may make sense to document the git server path in Config.mk but not
enable it by default?

 -- Keir

On 17/07/2009 02:59, "Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> Hi Ian,
> git clone http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
> and 
> git clone git://xenbits.xensource.com/qemu-xen-unstable.git
> 
> should always return the exactly same repository, correct?
> 
> I can''t access the latter since I''m behind a firewall. I
can only access the
> http interface.
> 
> Thanks,
> -- Dexuan
> 
> 
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Jackson
> Sent: 2009?7?16? 22:02
> To: xen-devel@lists.xensource.com
> Cc: Tom Rotenberg
> Subject: [Xen-devel] [PATCH] use git smart server protocol for qemu tree by
> default
> 
> Tom Rotenberg writes ("[Xen-devel] Can the qemu-xen git repository be
moved to
> a ''git://'' protocol? It can save a lot of time..."):
>> Currently, xen clones the qemu-xen using an ''http://''
protocol, and
>> this takes a lot of time.  It can be improved a lot if it will be
>> moved to be shared by a ''git://'' protocol, which is
much faster.
>> Can this be done?
> 
> Yes.  We were using http:// because we didn''t have a git server on
> xenbits, but now we do.
> 
> Keir: this patch is for xen-unstable but given that the git daemon is
> a property of the xenbits system configuration rather of the specific
> branch, it wouldn''t be unreasonable to apply it to the older trees
> too.
> 
> Ian.
> 
> use git smart server protocol for qemu tree by default
> 
> xenbits supports the git protocol now, which is much faster (and
> produces less noise in the build log too).
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> diff -r e07726c03d31 Config.mk
> --- a/Config.mk Thu Jul 16 10:32:55 2009 +0100
> +++ b/Config.mk Thu Jul 16 14:58:49 2009 +0100
> @@ -141,7 +141,7 @@ XEN_EXTFILES_URL=http://xenbits.xensourc
>  # the internet.  The original download URL is preserved as a comment
>  # near the place in the Xen Makefiles where the file is used.
>  
> -QEMU_REMOTE=http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
> +QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
>  
>  # Specify which qemu-dm to use. This may be `ioemu'' to use the
old
>  # Mercurial in-tree version, or a local directory, or a git URL.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Cui, Dexuan writes ("RE: [Xen-devel] [PATCH] use git smart server protocol
for qemu tree by	default"):
> Hi Ian,
> git clone http://xenbits.xensource.com/git-http/qemu-xen-unstable.git
> and 
> git clone git://xenbits.xensource.com/qemu-xen-unstable.git 
> 
> should always return the exactly same repository, correct?
Yes, barring misfeatures in git.  Certainly in our case you should be
able to use http.
> I can''t access the latter since I''m behind a firewall. I
can only access the http interface.
I would recommend getting your firewall fixed.  The git protocol is
much more efficient.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser writes ("Re: [Xen-devel] [PATCH] use git smart server protocol
for qemu tree by default"):
> It may make sense to document the git server path in Config.mk but not
> enable it by default?
I think if many people have this problem we could document the http:
path.  Nowadays people really ought to arrange that git can go through
their firewall (after all it''s not as if http isn''t used to
tunnel all
manner of nonsense!)

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 17 Jul 2009, Brendan Cully wrote:
> On Thursday, 16 July 2009 at 11:36, Tom Rotenberg wrote:
> > Hi,
> > 
> > Currently, xen clones the qemu-xen using an
''http://'' protocol, and
> > this takes a lot of time.  It can be improved a lot if it will be
> > moved to be shared by a ''git://'' protocol, which is
much faster.
> > Can this be done?
> 
> Why not have xen use an hg mirror of the qemu-xen tree? These are easy
> to set up (it''s a one-line "hg convert" cron job), and
it would
> simplify life for xen hackers to need only one obscure version control
> system.
We already have an hg mirror for qemu-xen, it is just not used by
default:

http://xenbits.xensource.com/ext/qemu-xen-unstable.hg


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Friday, 17 July 2009 at 11:11, Stefano Stabellini
wrote:
> On Fri, 17 Jul 2009, Brendan Cully wrote:
> > On Thursday, 16 July 2009 at 11:36, Tom Rotenberg wrote:
> > > Hi,
> > > 
> > > Currently, xen clones the qemu-xen using an
''http://'' protocol, and
> > > this takes a lot of time.  It can be improved a lot if it will be
> > > moved to be shared by a ''git://'' protocol,
which is much faster.
> > > Can this be done?
> > 
> > Why not have xen use an hg mirror of the qemu-xen tree? These are easy
> > to set up (it''s a one-line "hg convert" cron job),
and it would
> > simplify life for xen hackers to need only one obscure version control
> > system.
> 
> We already have an hg mirror for qemu-xen, it is just not used by
> default:
> 
> http://xenbits.xensource.com/ext/qemu-xen-unstable.hg
Ah, wonderful. Would it make sense to use this tree by default? We can
assume users already have hg working to pull the master tree, and it
means one less confusing build tool. The git tree doesn''t seem to me
to offer any advantages over an hg mirror for the majority of users,
but it does increase the number of complicated moving parts in an
already difficult build system.

Of course the git tree should remain an option for those people that
want to use it to collaborate with upstream qemu development.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I think c/s 19956 should be reverted.

After struggling through a recent default build (and finally remembering
this discussion from two weeks ago),  I have to agree with Brendan.
It''s bad enough to have two different sourcecode control systems,
but those unfamiliar with git should not have to learn about
its features and deficiencies (e.g. inability to use an http
proxy) to build xen-unstable.

C''mon people, it''s critical that we make it as easy as
possible
for newcomers and people who are not top-notch linux sysadmins to
still do Xen development.
> From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com]
> I would recommend getting your firewall fixed.  The git protocol is
> much more efficient.
The git protocol is much LESS efficient if you don''t know what
this means:

git.... [0: 999.99.99.999]: errno=Connection timed out
fatal: unable to connect a socket (Connection timed out)

Or how to "get your firewall fixed" (even if that is possible
in your organization or university) or how to set up tunneling
software to go around the firewall.

So inefficient in fact as to take a lot more time to figure
it out and "fix it" than to use http to begin with.

The default Xen build should be braindead easy to use;
I realize we are far from that already but that''s no excuse
for making it worse.

My two cents.
Dan

P.S. Even if I lose this battle, it should be an absolute
minimum that changes to the build dependencies/process are
accompanied by changes to the main README.  I don''t see any
mention of qemu/ioemu at all and the KERNELS= info is now wrong
as well.  Are we trying to ensure the failure of newcomers
that want to contribute to Xen?
> -----Original Message-----
> From: Brendan Cully [mailto:brendan@cs.ubc.ca]
> Sent: Thursday, July 16, 2009 5:03 PM
> To: Tom Rotenberg
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Can the qemu-xen git repository be moved to a
> ''git://'' protocol? It can save a lot of time...
> 
> 
> On Thursday, 16 July 2009 at 11:36, Tom Rotenberg wrote:
> > Hi,
> > 
> > Currently, xen clones the qemu-xen using an
''http://'' protocol, and
> > this takes a lot of time.  It can be improved a lot if it will be
> > moved to be shared by a ''git://'' protocol, which is
much faster.
> > Can this be done?
> 
> Why not have xen use an hg mirror of the qemu-xen tree? These are easy
> to set up (it''s a one-line "hg convert" cron job), and
it would
> simplify life for xen hackers to need only one obscure version control
> system.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/17/09 02:55, Ian Jackson wrote:
> Keir Fraser writes ("Re: [Xen-devel] [PATCH] use git smart server
protocol for qemu tree by default"):
>   
>> It may make sense to document the git server path in Config.mk but not
>> enable it by default?
>>     
>
> I think if many people have this problem we could document the http:
> path.  Nowadays people really ought to arrange that git can go through
> their firewall (after all it''s not as if http isn''t used
to tunnel all
> manner of nonsense!)
>   
Pfft.  We can''t even get git through most of the Citrix networks.  The
git protocol is an interoperability disaster.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 07/17/09 10:11, Brendan Cully wrote:
> Ah, wonderful. Would it make sense to use this tree by default? We can
> assume users already have hg working to pull the master tree, and it
> means one less confusing build tool. The git tree doesn''t seem to
me
> to offer any advantages over an hg mirror for the majority of users,
> but it does increase the number of complicated moving parts in an
> already difficult build system.
>   
+1.  Presumably using something like the forest extension would help here.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Pfft.  We can''t even get git through most of the Citrix networks. 
The
> git protocol is an interoperability disaster.
If random things tunneled by pretending to be http work then your
firewall isn''t very secure. If it stops other protocols and you
can''t get
it fixed then perhaps your problems are not git.

Anything over http/https hacks are not "interoperability", but
"hey
wakeup your security your needs looking at"

Alan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 09:49 +0100 on 05 Aug (1249465754), Alan Cox wrote:
> > Pfft.  We can''t even get git through most of the Citrix
networks.  The
> > git protocol is an interoperability disaster.
> 
> If random things tunneled by pretending to be http work then your
> firewall isn''t very secure. If it stops other protocols and you
can''t get
> it fixed then perhaps your problems are not git.
Sure, but most people don''t control their own firewall; why put a fight
with the IT department between them and building Xen, when we could just
use the HG mirror that we already provide?

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 5 Aug 2009, Jeremy Fitzhardinge wrote:
> On 07/17/09 10:11, Brendan Cully wrote:
> > Ah, wonderful. Would it make sense to use this tree by default? We can
> > assume users already have hg working to pull the master tree, and it
> > means one less confusing build tool. The git tree doesn''t
seem to me
> > to offer any advantages over an hg mirror for the majority of users,
> > but it does increase the number of complicated moving parts in an
> > already difficult build system.
> >   
> 
> +1.  Presumably using something like the forest extension would help here.
> 
Given that so many people are having problems with the git protocol I think
we could at least make the old git http repo the default.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tuesday, 04 August 2009 at 16:59, Jeremy Fitzhardinge
wrote:
> On 07/17/09 10:11, Brendan Cully wrote:
> > Ah, wonderful. Would it make sense to use this tree by default? We can
> > assume users already have hg working to pull the master tree, and it
> > means one less confusing build tool. The git tree doesn''t
seem to me
> > to offer any advantages over an hg mirror for the majority of users,
> > but it does increase the number of complicated moving parts in an
> > already difficult build system.
> >   
> 
> +1.  Presumably using something like the forest extension would help here.
Mercurial 1.3 introduces subrepositories [1], which are ideal for
this. You''d just create a .hgsub file in the xen root like

ioemu-remote = http://xenbits.xensource.com/ext/qemu-xen-unstable.hg

and whenever you commit in the main xen tree, hg records the version
of the qemu tree in use automatically, and checks out the appropriate
version whenever you hg update.

Of course, this only works with 1.3+. In the meantime, here''s a patch
that imitates the qemu git Make logic for hg.

[1] http://mercurial.selenic.com/wiki/subrepos


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel