llvm dev - May 2016 - [llvm-foundation] IMPORTANT: APT repo temporary switched off

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone. During last 2
weeks the main services on llvm.org (svn, git, bugzilla) had serious
problems with overall connectivity.

We decided to temporary switch APT repo off to see if this would help.
Stay tuned for updates.

-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University

----- Original Message -----
> From: "Anton Korobeynikov via llvm-foundation"
<llvm-foundation at lists.llvm.org>
> To: "llvm-dev" <llvm-dev at lists.llvm.org>,
"clang-dev Developers" <cfe-dev at cs.uiuc.edu>,
> llvm-foundation at lists.llvm.org
> Cc: "Sylvestre Ledru" <sylvestre at debian.org>
> Sent: Tuesday, May 31, 2016 12:27:21 PM
> Subject: [llvm-foundation] IMPORTANT: APT repo temporary switched off
> 
> TL;DR: APT repo switched off due to excessive load / traffic
> 
> Recently we realized that APT repo generates almost 95% of I/O on
> llvm.org and more than 40% of network bandwidth alone.
Interesting. Is that mostly from metadata updates, or from people actually
installing LLVM packages?

 -Hal
> During last 2
> weeks the main services on llvm.org (svn, git, bugzilla) had serious
> problems with overall connectivity.
> 
> We decided to temporary switch APT repo off to see if this would
> help.
> Stay tuned for updates.
> 
> --
> With best regards, Anton Korobeynikov
> Department of Statistical Modelling, Saint Petersburg State
> University
> _______________________________________________
> llvm-foundation mailing list
> llvm-foundation at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-foundation
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

>> Recently we realized that APT repo generates almost 95% of I/O on
>> llvm.org and more than 40% of network bandwidth alone.
>
> Interesting. Is that mostly from metadata updates, or from people actually
installing LLVM packages?
The latter


-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University

On 31 May 2016 at 18:27, Anton Korobeynikov via llvm-foundation
<llvm-foundation at lists.llvm.org> wrote:
> We decided to temporary switch APT repo off to see if this would help.
> Stay tuned for updates.
Thanks Anton, there were a lot of timeouts on our bots for the last
few weeks, I hope this will make things better.

If we still want to be serving those packages, maybe we could talk to
someone willing to host them?

cheers,
--renato

On 31 May 2016 at 10:27, Anton Korobeynikov via llvm-foundation
<llvm-foundation at lists.llvm.org> wrote:
> TL;DR: APT repo switched off due to excessive load / traffic
>
> Recently we realized that APT repo generates almost 95% of I/O on
> llvm.org and more than 40% of network bandwidth alone. During last 2
> weeks the main services on llvm.org (svn, git, bugzilla) had serious
> problems with overall connectivity.
I'm not surprised by the heavy load but a little more notice would
have been nice. This has broken our CI builds for KLEE [1]. We (the
KLEE developers) can fix this by moving our CI to run in an Ubuntu
14.04 environment where the LLVM version we currently require is part
of the official Ubuntu repositories. So I guess this is the impetus we
need to make the switch :)

I (and suspect several others) really on the packages in the APT repo
for our CI builds. You can see see that the LLVM APT repo
is in TravisCI's source white list [2] so there are probably others using
it.


[1] https://travis-ci.org/klee/klee/jobs/134303870
[2]
https://github.com/travis-ci/apt-source-whitelist/blob/92dbd15f6a12e5f24c2f31f314060a349c37dc68/ubuntu.json#L168

On Wed, Jun 1, 2016 at 9:59 AM, Dan Liew via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 31 May 2016 at 10:27, Anton Korobeynikov via llvm-foundation
> <llvm-foundation at lists.llvm.org> wrote:
> > TL;DR: APT repo switched off due to excessive load / traffic
> >
> > Recently we realized that APT repo generates almost 95% of I/O on
> > llvm.org and more than 40% of network bandwidth alone. During last 2
> > weeks the main services on llvm.org (svn, git, bugzilla) had serious
> > problems with overall connectivity.
>
> I'm not surprised by the heavy load but a little more notice would
> have been nice.
>
Yes, I imagine that this broke quite a few builds on Travis CI; it
certainly broke a few that I rely on.

Regardless, thanks for having run this infrastructure for so long. Have you
considered publishing the packages to a PPA, so that the community can
continue to benefit from them without llvm.org having to support the
hosting costs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160601/078d96ac/attachment.html>

On 1 June 2016 at 17:59, Dan Liew via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> I'm not surprised by the heavy load but a little more notice would
> have been nice. This has broken our CI builds for KLEE [1]. We (the
> KLEE developers) can fix this by moving our CI to run in an Ubuntu
> 14.04 environment where the LLVM version we currently require is part
> of the official Ubuntu repositories. So I guess this is the impetus we
> need to make the switch :)
Hi Dan,

I understand the problems that this have created, and I can see that
we still have people on IRC complaining about it.

But the server were really on its knees and everything else in it
(including all our source repositories) were just not working at all.
It was a hard decision, but one that had to be made, and soon.

> Regardless, thanks for having run this infrastructure for so long. Have you
considered publishing the packages to a PPA, so that the community can continue
to benefit from them without llvm.org having to support the hosting costs?
John,

I'm not well versed with PPAs, but if you have an alternative that we
could do and fix the package problem without reusing the same server,
I think we should pursue it.

cheers,
--renato

Le 31/05/2016 à 19:27, Anton Korobeynikov a écrit :
> TL;DR: APT repo switched off due to excessive load / traffic
>
> Recently we realized that APT repo generates almost 95% of I/O on
> llvm.org and more than 40% of network bandwidth alone. During last 2
> weeks the main services on llvm.org (svn, git, bugzilla) had serious
> problems with overall connectivity.
Oups! I wasn't aware of this issue. Thanks for doing that.
At least, we have an answer to a question I had recently:
a lot of people are using it!
> We decided to temporary switch APT repo off to see if this would help.
> Stay tuned for updates.
OK, could you share the updates? Should I look for an alternative for
the hosting of the repository?

Cheers,
Sylvestre

On Tue, May 31, 2016 at 7:27 PM, Anton Korobeynikov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> TL;DR: APT repo switched off due to excessive load / traffic
>
> Recently we realized that APT repo generates almost 95% of I/O on
> llvm.org and more than 40% of network bandwidth alone. During last 2
> weeks the main services on llvm.org (svn, git, bugzilla) had serious
> problems with overall connectivity.
>
I think this correlates with the increase of package update frequency from
once a week to (sometimes more than) once a day. [1]
I thought that the CI systems like Travis do some caching of packages so it
is not re-downloading new packages for every build of every pull request?
If not, adding one commit to a PR in LDC means 6 downloads from llvm.org...

We decided to temporary switch APT repo off to see if this would
help.
> Stay tuned for updates.
>
Because LDC's core CI testing depends on the APT repo, I hope a replacement
solution is found quickly.
Thanks very much,
   Johan

[1] https://groups.google.com/forum/#!topic/llvm-dev/MjhxuNRx9sA
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160601/73631dcc/attachment.html>

Hello,
Le 31/05/2016 à 19:27, Anton Korobeynikov a écrit :
> TL;DR: APT repo switched off due to excessive load / traffic
>
> Recently we realized that APT repo generates almost 95% of I/O on
> llvm.org and more than 40% of network bandwidth alone. During last 2
> weeks the main services on llvm.org (svn, git, bugzilla) had serious
> problems with overall connectivity.
>
> We decided to temporary switch APT repo off to see if this would help.
> Stay tuned for updates.
>
Until we have a more permanent solution, I set up:

http://llvm-apt.ecranbleu.org/apt/

Please note that it is just a temporary solution, I might turn it off if
the load is too important.

(the server is only used for this).

The sync from the CI is not back yet (should be done in the next few days).

Cheers,
Sylvestre

On Fri, Jun 10, 2016 at 9:00 PM, Sylvestre Ledru via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello,
> Le 31/05/2016 à 19:27, Anton Korobeynikov a écrit :
> > TL;DR: APT repo switched off due to excessive load / traffic
> >
> > Recently we realized that APT repo generates almost 95% of I/O on
> > llvm.org and more than 40% of network bandwidth alone. During last 2
> > weeks the main services on llvm.org (svn, git, bugzilla) had serious
> > problems with overall connectivity.
> >
> > We decided to temporary switch APT repo off to see if this would help.
> > Stay tuned for updates.
> >
> Until we have a more permanent solution, I set up:
>
> http://llvm-apt.ecranbleu.org/apt/
>
> Please note that it is just a temporary solution, I might turn it off if
> the load is too important.
>
>
Thank you very much for setting up an APT repository with LLVM trunk in it.
LDC's CI systems are now working with the binaries available from llvm.org,
but for LLVM trunk we depend on the APT repo.

cheers,
  Johan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160611/0eefb521/attachment.html>