C Bergström via llvm-dev
2016-Aug-19 14:54 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
I remember reading all those examples about how a big repo isn't an
issue and when sitting at some ${LOCATION} I have 1010101010Gbps and
it takes like 30 seconds.. Well.. that may be true when I'm at home,
but I'm not.. and this laptop doesn't have llvm sources installed..
So real world feedback of the pita it can be..
/* Side rant - I wish I didn't even need the llvm sources. I just want
to build libcxxrt */
/* Yes I'm about to do the smarter thing and fetch a shallow clone or
try to get the tarball snapshot from github.. I did however want to
show an example of how big repo on non-corporate infrastructure can be
a challenge. I'd hate for some ambitious community dev to run into the
same thing and turn away as a result */
------------
CMake Error at CMakeLists.txt:53 (message):
llvm-config not found and LLVM_MAIN_SRC_DIR not defined. Reconfigure with
-DLLVM_CONFIG=path/to/llvm-config or -DLLVM_PATH=path/to/llvm-source-root.
-- Configuring incomplete, errors occurred!
See also "/usr/local/src/libcxxabi/build/CMakeFiles/CMakeOutput.log".
asura at asura-X3:/usr/local/src/libcxxabi/build$ cd ../..
asura at asura-X3:/usr/local/src$ git clone git at
github.com:llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228082, done.
remote: Compressing objects: 100% (245/245), done.
Receiving objects: 13% (167280/1228082), 68.86 MiB | 20.00 KiB/s
^Cfatal: The remote end hung up unexpectedly
asura at asura-X3:/usr/local/src$ rm -rf llv^C
asura at asura-X3:/usr/local/src$ git clone git at
github.com:llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228082, done.
remote: Compressing objects: 100% (245/245), done.
^Cfatal: The remote end hung up unexpectedly1 KiB | 20.00 KiB/s
asura at asura-X3:/usr/local/src$ git clone
https://github.com/llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228082, done.
remote: Compressing objects: 100% (245/245), done.
error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
asura at asura-X3:/usr/local/src$ cd lib^C
asura at asura-X3:/usr/local/src$ git clone
https://github.com/llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228183, done.
remote: Compressing objects: 100% (346/346), done.
^Cceiving objects: 0% (1069/1228183), 388.01 KiB | 1024 bytes/s
asura at asura-X3:/usr/local/src$ git clone
https://github.com/llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228183, done.
error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
asura at asura-X3:/usr/local/src$ git clone
https://github.com/llvm-mirror/llvm.git
Cloning into 'llvm'...
remote: Counting objects: 1228271, done.
remote: Compressing objects: 100% (11/11), done.
error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
James Molloy via llvm-dev
2016-Aug-19 15:14 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
Hi, No matter what we do, I hardly think it likely we can optimize the LLVM clone speed for a 20KB/s connection. I agree that corporate internet connections being gigantic pipes means we run the risk of forgetting what home infrastructure looks like, but 20KBps is... around dial-up. Where are you, rural Tibet? Cheers, James On Fri, 19 Aug 2016 at 15:55 C Bergström <llvm-dev at lists.llvm.org> wrote:> I remember reading all those examples about how a big repo isn't an > issue and when sitting at some ${LOCATION} I have 1010101010Gbps and > it takes like 30 seconds.. Well.. that may be true when I'm at home, > but I'm not.. and this laptop doesn't have llvm sources installed.. > > So real world feedback of the pita it can be.. > /* Side rant - I wish I didn't even need the llvm sources. I just want > to build libcxxrt */ > > /* Yes I'm about to do the smarter thing and fetch a shallow clone or > try to get the tarball snapshot from github.. I did however want to > show an example of how big repo on non-corporate infrastructure can be > a challenge. I'd hate for some ambitious community dev to run into the > same thing and turn away as a result */ > ------------ > CMake Error at CMakeLists.txt:53 (message): > llvm-config not found and LLVM_MAIN_SRC_DIR not defined. Reconfigure > with > -DLLVM_CONFIG=path/to/llvm-config or > -DLLVM_PATH=path/to/llvm-source-root. > > > -- Configuring incomplete, errors occurred! > See also "/usr/local/src/libcxxabi/build/CMakeFiles/CMakeOutput.log". > asura at asura-X3:/usr/local/src/libcxxabi/build$ cd ../.. > asura at asura-X3:/usr/local/src$ git clone git at github.com: > llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > Receiving objects: 13% (167280/1228082), 68.86 MiB | 20.00 KiB/s > > ^Cfatal: The remote end hung up unexpectedly > > asura at asura-X3:/usr/local/src$ rm -rf llv^C > asura at asura-X3:/usr/local/src$ git clone git at github.com: > llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > ^Cfatal: The remote end hung up unexpectedly1 KiB | 20.00 KiB/s > > asura at asura-X3:/usr/local/src$ git clone > https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull > function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failed > asura at asura-X3:/usr/local/src$ cd lib^C > asura at asura-X3:/usr/local/src$ git clone > https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > remote: Compressing objects: 100% (346/346), done. > ^Cceiving objects: 0% (1069/1228183), 388.01 KiB | 1024 bytes/s > asura at asura-X3:/usr/local/src$ git clone > https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull > function. > fatal: The remote end hung up unexpectedly > fatal: protocol error: bad pack header > asura at asura-X3:/usr/local/src$ git clone > https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228271, done. > remote: Compressing objects: 100% (11/11), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull > function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failed > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160819/458abfab/attachment.html>
C Bergström via llvm-dev
2016-Aug-19 15:22 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
Hah! Woo is me you say... Have you ever stayed in a non-hilton hotel while traveling? I don't mean couch surfing, but just average place.. I'd even add some above average places to the list and when you get 100 people all on the same line.. it can get "slow"... Lets also factor in networking policies that favor normal internet traffic.. So they may allow youtube to stream like normal, but ssh or something else gets throttled a lot.. (which is what's happening in my case) I wish I was in Tibet.. I'll have to add that to my things to try if I visit some day.. ;) You may mock me and think this can't ever happen, but I'm happy to benchmark real world connection speeds while traveling. On Fri, Aug 19, 2016 at 11:14 PM, James Molloy <james at jamesmolloy.co.uk> wrote:> Hi, > > No matter what we do, I hardly think it likely we can optimize the LLVM > clone speed for a 20KB/s connection. I agree that corporate internet > connections being gigantic pipes means we run the risk of forgetting what > home infrastructure looks like, but 20KBps is... around dial-up. Where are > you, rural Tibet? > > Cheers, > > James > > On Fri, 19 Aug 2016 at 15:55 C Bergström <llvm-dev at lists.llvm.org> wrote: >> >> I remember reading all those examples about how a big repo isn't an >> issue and when sitting at some ${LOCATION} I have 1010101010Gbps and >> it takes like 30 seconds.. Well.. that may be true when I'm at home, >> but I'm not.. and this laptop doesn't have llvm sources installed.. >> >> So real world feedback of the pita it can be.. >> /* Side rant - I wish I didn't even need the llvm sources. I just want >> to build libcxxrt */ >> >> /* Yes I'm about to do the smarter thing and fetch a shallow clone or >> try to get the tarball snapshot from github.. I did however want to >> show an example of how big repo on non-corporate infrastructure can be >> a challenge. I'd hate for some ambitious community dev to run into the >> same thing and turn away as a result */ >> ------------ >> CMake Error at CMakeLists.txt:53 (message): >> llvm-config not found and LLVM_MAIN_SRC_DIR not defined. Reconfigure >> with >> -DLLVM_CONFIG=path/to/llvm-config or >> -DLLVM_PATH=path/to/llvm-source-root. >> >> >> -- Configuring incomplete, errors occurred! >> See also "/usr/local/src/libcxxabi/build/CMakeFiles/CMakeOutput.log". >> asura at asura-X3:/usr/local/src/libcxxabi/build$ cd ../.. >> asura at asura-X3:/usr/local/src$ git clone >> git at github.com:llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228082, done. >> remote: Compressing objects: 100% (245/245), done. >> Receiving objects: 13% (167280/1228082), 68.86 MiB | 20.00 KiB/s >> >> ^Cfatal: The remote end hung up unexpectedly >> >> asura at asura-X3:/usr/local/src$ rm -rf llv^C >> asura at asura-X3:/usr/local/src$ git clone >> git at github.com:llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228082, done. >> remote: Compressing objects: 100% (245/245), done. >> ^Cfatal: The remote end hung up unexpectedly1 KiB | 20.00 KiB/s >> >> asura at asura-X3:/usr/local/src$ git clone >> https://github.com/llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228082, done. >> remote: Compressing objects: 100% (245/245), done. >> error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull >> function. >> fatal: The remote end hung up unexpectedly >> fatal: early EOF >> fatal: index-pack failed >> asura at asura-X3:/usr/local/src$ cd lib^C >> asura at asura-X3:/usr/local/src$ git clone >> https://github.com/llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228183, done. >> remote: Compressing objects: 100% (346/346), done. >> ^Cceiving objects: 0% (1069/1228183), 388.01 KiB | 1024 bytes/s >> asura at asura-X3:/usr/local/src$ git clone >> https://github.com/llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228183, done. >> error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull >> function. >> fatal: The remote end hung up unexpectedly >> fatal: protocol error: bad pack header >> asura at asura-X3:/usr/local/src$ git clone >> https://github.com/llvm-mirror/llvm.git >> Cloning into 'llvm'... >> remote: Counting objects: 1228271, done. >> remote: Compressing objects: 100% (11/11), done. >> error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull >> function. >> fatal: The remote end hung up unexpectedly >> fatal: early EOF >> fatal: index-pack failed >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Mehdi Amini via llvm-dev
2016-Aug-19 15:29 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
> On Aug 19, 2016, at 7:54 AM, C Bergström via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > I remember reading all those examples about how a big repo isn't an > issue and when sitting at some ${LOCATION} I have 1010101010Gbps and > it takes like 30 seconds.. Well.. that may be true when I'm at home, > but I'm not.. and this laptop doesn't have llvm sources installed.. > > So real world feedback of the pita it can be.. > /* Side rant - I wish I didn't even need the llvm sources. I just want > to build libcxxrt */ >/* side rant: aren’t you the one complaining loudly about noisy non-technical thread on llvm-dev?? */> /* Yes I'm about to do the smarter thing and fetch a shallow clone or > try to get the tarball snapshot from github.. I did however want to > show an example of how big repo on non-corporate infrastructure can be > a challenge. I'd hate for some ambitious community dev to run into the > same thing and turn away as a result */ > ------------ > CMake Error at CMakeLists.txt:53 (message): > llvm-config not found and LLVM_MAIN_SRC_DIR not defined. Reconfigure with > -DLLVM_CONFIG=path/to/llvm-config or -DLLVM_PATH=path/to/llvm-source-root. > > > -- Configuring incomplete, errors occurred! > See also "/usr/local/src/libcxxabi/build/CMakeFiles/CMakeOutput.log". > asura at asura-X3:/usr/local/src/libcxxabi/build$ cd ../.. > asura at asura-X3:/usr/local/src$ git clone git at github.com:llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > Receiving objects: 13% (167280/1228082), 68.86 MiB | 20.00 KiB/s > > ^Cfatal: The remote end hung up unexpectedly > > asura at asura-X3:/usr/local/src$ rm -rf llv^C > asura at asura-X3:/usr/local/src$ git clone git at github.com:llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > ^Cfatal: The remote end hung up unexpectedly1 KiB | 20.00 KiB/s > > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failed > asura at asura-X3:/usr/local/src$ cd lib^C > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > remote: Compressing objects: 100% (346/346), done. > ^Cceiving objects: 0% (1069/1228183), 388.01 KiB | 1024 bytes/s > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: protocol error: bad pack header > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228271, done. > remote: Compressing objects: 100% (11/11), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failed > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Konstantin Tokarev via llvm-dev
2016-Aug-19 15:30 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
19.08.2016, 17:55, "C Bergström via llvm-dev" <llvm-dev at lists.llvm.org>:> I remember reading all those examples about how a big repo isn't an > issue and when sitting at some ${LOCATION} I have 1010101010Gbps and > it takes like 30 seconds.. Well.. that may be true when I'm at home, > but I'm not.. and this laptop doesn't have llvm sources installed.. > > So real world feedback of the pita it can be.. > /* Side rant - I wish I didn't even need the llvm sources. I just want > to build libcxxrt */ > > /* Yes I'm about to do the smarter thing and fetch a shallow clone or > try to get the tarball snapshot from github.. I did however want to > show an example of how big repo on non-corporate infrastructure can be > a challenge. I'd hate for some ambitious community dev to run into the > same thing and turn away as a result */ > ------------ > CMake Error at CMakeLists.txt:53 (message): > llvm-config not found and LLVM_MAIN_SRC_DIR not defined. Reconfigure with > -DLLVM_CONFIG=path/to/llvm-config or -DLLVM_PATH=path/to/llvm-source-root. > > -- Configuring incomplete, errors occurred! > See also "/usr/local/src/libcxxabi/build/CMakeFiles/CMakeOutput.log". > asura at asura-X3:/usr/local/src/libcxxabi/build$ cd ../.. > asura at asura-X3:/usr/local/src$ git clone git at github.com:llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > Receiving objects: 13% (167280/1228082), 68.86 MiB | 20.00 KiB/s > > ^Cfatal: The remote end hung up unexpectedly > > asura at asura-X3:/usr/local/src$ rm -rf llv^C > asura at asura-X3:/usr/local/src$ git clone git at github.com:llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > ^Cfatal: The remote end hung up unexpectedly1 KiB | 20.00 KiB/s > > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228082, done. > remote: Compressing objects: 100% (245/245), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failed > asura at asura-X3:/usr/local/src$ cd lib^C > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > remote: Compressing objects: 100% (346/346), done. > ^Cceiving objects: 0% (1069/1228183), 388.01 KiB | 1024 bytes/s > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228183, done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: protocol error: bad pack header > asura at asura-X3:/usr/local/src$ git clone https://github.com/llvm-mirror/llvm.git > Cloning into 'llvm'... > remote: Counting objects: 1228271, done. > remote: Compressing objects: 100% (11/11), done. > error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. > fatal: The remote end hung up unexpectedly > fatal: early EOF > fatal: index-pack failedPlease try following approach: 1. Clone with --depth 1 2. Run git fetch --unshallow In this case you will split download into 2 parts, which can succeed independently from each other. If you cannot get unshallow step working, you can split it further by running git fetch --depth=<number> with increasing numbers. -- Regards, Konstantin
C Bergström via llvm-dev
2016-Aug-19 15:33 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
On Fri, Aug 19, 2016 at 11:29 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:> >> On Aug 19, 2016, at 7:54 AM, C Bergström via llvm-dev <llvm-dev at lists.llvm.org> wrote: >> >> I remember reading all those examples about how a big repo isn't an >> issue and when sitting at some ${LOCATION} I have 1010101010Gbps and >> it takes like 30 seconds.. Well.. that may be true when I'm at home, >> but I'm not.. and this laptop doesn't have llvm sources installed.. >> >> So real world feedback of the pita it can be.. >> /* Side rant - I wish I didn't even need the llvm sources. I just want >> to build libcxxrt */ >> > > /* side rant: aren’t you the one complaining loudly about noisy non-technical thread on llvm-dev?? */Yes and please look at the subject - I clearly put OT as well as didn't cross post to every other ML on the planet. So I'm following my own advice..
C Bergström via llvm-dev
2016-Aug-19 15:35 UTC
[llvm-dev] OT: Cloning llvm repo over low speed connection != fun
On Fri, Aug 19, 2016 at 11:30 PM, Konstantin Tokarev <annulen at yandex.ru> wrote:> > > Please try following approach: > > 1. Clone with --depth 1 > 2. Run git fetch --unshallowYou'll see in my 1st email I said I was going to try this.. (shallow clone did work) It was meant more as a data point from real world than a cry for help. Thanks a lot for looking out though :) Long time no talk..