Galina Kistanova via llvm-dev
2019-Oct-15 01:16 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
Hello everyone, We are in the middle of porting the majority of zorg to GitHub/monorepo. The following build factories will be ported and if you use one of those for your bots, you are all covered: * ClangBuilder.getClangCMakeBuildFactory (31 bots) * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) * LibcxxAndAbiBuilder (23 bots) * SphinxDocsBuilder (7 bots) * UnifiedTreeBuilder (11 bots) * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder Some build factories will be deprecated. If you use one of these, please change your bot to use something else instead. Here is the list of deprecated build factories: * ClangBuilder.getClangBuildFactory (0 bots) * LLDBuilder (0 bots) * ClangAndLLDBuilder (0 bots) However, some special build factories and build factories with a few bots would need your attention. Here is the list of build factories in need of porting. Patches are welcome. * LLVMBuilder (3 bots) * PollyBuilder (3 bots) * LLDBBuilder (6 bots) * SanitizerBuilder (10 bots) * CUDATestsuiteBuilder (1 bot) - depends on ClangBuilder.getClangBuildFactory * AOSPBuilder (1 bot) - depends on PollyBuilder * AnnotatedBuilder (2 bots) * OpenMPBuilder (2 bots) * FuchsiaBuilder (1 bot) Please feel free to ask if you have questions. Thanks Galina -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191014/513f7251/attachment.html>
Eli Friedman via llvm-dev
2019-Oct-15 01:57 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
What steps are required to port a build factory? I guess any “SVN” steps have to go away, but I’m not sure how we’re supposed to replace them. (I’m interested in PollyBuilder and AOSPBuilder, specifically.) -Eli From: cfe-commits <cfe-commits-bounces at lists.llvm.org> On Behalf Of Galina Kistanova via cfe-commits Sent: Monday, October 14, 2019 6:16 PM To: LLVM Dev <llvm-dev at lists.llvm.org>; cfe-commits <cfe-commits at lists.llvm.org>; llvm-commits <llvm-commits at lists.llvm.org> Subject: [EXT] Zorg migration to GitHub/monorepo Hello everyone, We are in the middle of porting the majority of zorg to GitHub/monorepo. The following build factories will be ported and if you use one of those for your bots, you are all covered: * ClangBuilder.getClangCMakeBuildFactory (31 bots) * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) * LibcxxAndAbiBuilder (23 bots) * SphinxDocsBuilder (7 bots) * UnifiedTreeBuilder (11 bots) * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder Some build factories will be deprecated. If you use one of these, please change your bot to use something else instead. Here is the list of deprecated build factories: * ClangBuilder.getClangBuildFactory (0 bots) * LLDBuilder (0 bots) * ClangAndLLDBuilder (0 bots) However, some special build factories and build factories with a few bots would need your attention. Here is the list of build factories in need of porting. Patches are welcome. * LLVMBuilder (3 bots) * PollyBuilder (3 bots) * LLDBBuilder (6 bots) * SanitizerBuilder (10 bots) * CUDATestsuiteBuilder (1 bot) - depends on ClangBuilder.getClangBuildFactory * AOSPBuilder (1 bot) - depends on PollyBuilder * AnnotatedBuilder (2 bots) * OpenMPBuilder (2 bots) * FuchsiaBuilder (1 bot) Please feel free to ask if you have questions. Thanks Galina -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191015/cfdbb84a/attachment.html>
Galina Kistanova via llvm-dev
2019-Oct-15 23:23 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
Hello Eli, Thanks for helping with this! May be the easies is to refactor the PollyBuilder to use UnifiedTreeBuilder for creating a build factory, checkout the source code for dependent projects, and maybe make configure. The rest seems could be used as it currently is. As a quick example you could check how Alex Orlov has refactored the SphinxDocsBuilder - https://reviews.llvm.org/D68955. Hope this helps. And please feel free to ask if you have more questions. Thanks Galina On Mon, Oct 14, 2019 at 6:57 PM Eli Friedman <efriedma at quicinc.com> wrote:> What steps are required to port a build factory? I guess any “SVN” steps > have to go away, but I’m not sure how we’re supposed to replace them. (I’m > interested in PollyBuilder and AOSPBuilder, specifically.) > > -Eli > > > > *From:* cfe-commits <cfe-commits-bounces at lists.llvm.org> *On Behalf Of *Galina > Kistanova via cfe-commits > *Sent:* Monday, October 14, 2019 6:16 PM > *To:* LLVM Dev <llvm-dev at lists.llvm.org>; cfe-commits < > cfe-commits at lists.llvm.org>; llvm-commits <llvm-commits at lists.llvm.org> > *Subject:* [EXT] Zorg migration to GitHub/monorepo > > > > Hello everyone, > > > > We are in the middle of porting the majority of zorg to > GitHub/monorepo. The following build factories will be ported and if you > use one of those for your bots, you are all covered: > > > > * ClangBuilder.getClangCMakeBuildFactory (31 bots) > > * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) > > * LibcxxAndAbiBuilder (23 bots) > > * SphinxDocsBuilder (7 bots) > > * UnifiedTreeBuilder (11 bots) > > * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder > > * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder > > * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder > > > > Some build factories will be deprecated. If you use one of these, please > change your bot to use something else instead. Here is the list of > deprecated build factories: > > > > * ClangBuilder.getClangBuildFactory (0 bots) > > * LLDBuilder (0 bots) > > * ClangAndLLDBuilder (0 bots) > > > > However, some special build factories and build factories with a few bots > would need your attention. > > Here is the list of build factories in need of porting. Patches are > welcome. > > > > * LLVMBuilder (3 bots) > > * PollyBuilder (3 bots) > > * LLDBBuilder (6 bots) > > * SanitizerBuilder (10 bots) > > * CUDATestsuiteBuilder (1 bot) - depends on > ClangBuilder.getClangBuildFactory > > * AOSPBuilder (1 bot) - depends on PollyBuilder > > * AnnotatedBuilder (2 bots) > > * OpenMPBuilder (2 bots) > > * FuchsiaBuilder (1 bot) > > > > Please feel free to ask if you have questions. > > > > Thanks > > > > Galina > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191015/8779eaa4/attachment.html>
Galina Kistanova via llvm-dev
2019-Oct-18 00:04 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
Hello everyone, The build bot is almost ready to move to github. As the next step we would migrate the staging master to listen for the both changes, from SVN as it was before, and from github. Tonight I am going to update the staging (aka silent) master and it will start working with github. I will send an email when it is done. If you bots use one of the ported build factories from the list in my previous e-mail, please feel free to stage them for working with monorepo/github and once you are happy, let me know, and I'll configure them accordingly on the production master. Once that's done, you could move the bots back to the production master and you are done. When you stage your bot, please follow these steps: 1. Stop your bot between builds, if possible, 2. Remove the bot working directory (usually this directory has a builder name and is under directory where your bot is installed; if you are not sure, check the 'builddir' param of your bot in zorg/buildbot/osuosl/master/config/builders.py), 3. Edit buildbot.tac in the directory where your bot is installed. Change "port = 9990" line to "port = 9994". Save the change. 4. Start your bot, make sure it connects to the staging master. 5. Send me a mail with the staged bot names. Once you are happy with your bot building monorepo changes from github, please send me an e-mail and I'll respond with the instructions of how to get your bot back to production. Please be aware that staging master could restart often. Please let me know if you are having long running builds. Feel free to ask if you have questions. Please me know if you will see issues with the staging master. Thanks Galina On Mon, Oct 14, 2019 at 6:16 PM Galina Kistanova <gkistanova at gmail.com> wrote:> Hello everyone, > > > > We are in the middle of porting the majority of zorg to > GitHub/monorepo. The following build factories will be ported and if you > use one of those for your bots, you are all covered: > > > > * ClangBuilder.getClangCMakeBuildFactory (31 bots) > > * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) > > * LibcxxAndAbiBuilder (23 bots) > > * SphinxDocsBuilder (7 bots) > > * UnifiedTreeBuilder (11 bots) > > * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder > > * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder > > * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder > > > > Some build factories will be deprecated. If you use one of these, please > change your bot to use something else instead. Here is the list of > deprecated build factories: > > > > * ClangBuilder.getClangBuildFactory (0 bots) > > * LLDBuilder (0 bots) > > * ClangAndLLDBuilder (0 bots) > > > > However, some special build factories and build factories with a few bots > would need your attention. > > Here is the list of build factories in need of porting. Patches are > welcome. > > > > * LLVMBuilder (3 bots) > > * PollyBuilder (3 bots) > > * LLDBBuilder (6 bots) > > * SanitizerBuilder (10 bots) > > * CUDATestsuiteBuilder (1 bot) - depends on > ClangBuilder.getClangBuildFactory > > * AOSPBuilder (1 bot) - depends on PollyBuilder > > * AnnotatedBuilder (2 bots) > > * OpenMPBuilder (2 bots) > > * FuchsiaBuilder (1 bot) > > > > Please feel free to ask if you have questions. > > > > Thanks > > > > Galina > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191017/3bc80b3a/attachment.html>
Galina Kistanova via llvm-dev
2019-Oct-18 07:05 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
Hello build bot owners! The staging master is ready. Please feel free to use it to make sure your bots would work well with the monorepo and github. The following builders could be configured to build monorepo: * clang-atom-d525-fedora-rel * clang-native-arm-lnt-perf * clang-cmake-armv7-lnt * clang-cmake-armv7-selfhost-neon * clang-cmake-armv7-quick * clang-cmake-armv7-global-isel * clang-cmake-armv7-selfhost * clang-cmake-aarch64-quick * clang-cmake-aarch64-lld * clang-cmake-aarch64-global-isel * clang-ppc64be-linux-lnt * clang-ppc64be-linux-multistage * clang-ppc64le-linux-lnt * clang-ppc64le-linux-multistage * clang-ppc64be-linux * clang-ppc64le-linux * clang-s390x-linux * clang-s390x-linux-multistage * clang-s390x-linux-lnt * clang-hexagon-elf * clang-cmake-x86_64-avx2-linux * clang-cmake-x86_64-avx2-linux-perf * clang-cmake-x86_64-sde-avx512-linux * clang-solaris11-amd64 * clang-x64-ninja-win7 * clang-solaris11-sparcv9 * clang-cmake-armv7-full * clang-cmake-thumbv7-full-sh * clang-cmake-armv8-lld * clang-cmake-aarch64-full * clang-armv7-linux-build-cache * clang-aarch64-linux-build-cache * libcxx-libcxxabi-x86_64-linux-debian * libcxx-libcxxabi-x86_64-linux-debian-noexceptions * libcxx-libcxxabi-libunwind-x86_64-linux-debian * libcxx-libcxxabi-singlethreaded-x86_64-linux-debian * libcxx-libcxxabi-x86_64-linux-ubuntu-cxx03 * libcxx-libcxxabi-x86_64-linux-ubuntu-cxx11 * libcxx-libcxxabi-x86_64-linux-ubuntu-cxx14 * libcxx-libcxxabi-x86_64-linux-ubuntu-cxx17 * libcxx-libcxxabi-x86_64-linux-ubuntu-cxx2a * libcxx-libcxxabi-x86_64-linux-ubuntu-32bit * libcxx-libcxxabi-x86_64-linux-ubuntu-asan * libcxx-libcxxabi-x86_64-linux-ubuntu-ubsan * libcxx-libcxxabi-x86_64-linux-ubuntu-msan * libcxx-libcxxabi-libunwind-x86_64-linux-ubuntu * libcxx-libcxxabi-x86_64-linux-ubuntu-tsan * libcxx-libcxxabi-x86_64-linux-ubuntu-gcc5-cxx11 * libcxx-libcxxabi-x86_64-linux-ubuntu-gcc-tot-latest-std * libcxx-libcxxabi-libunwind-armv7-linux * libcxx-libcxxabi-libunwind-armv8-linux * libcxx-libcxxabi-libunwind-armv7-linux-noexceptions * libcxx-libcxxabi-libunwind-armv8-linux-noexceptions * libcxx-libcxxabi-libunwind-aarch64-linux * libcxx-libcxxabi-libunwind-aarch64-linux-noexceptions * ppc64le-lld-multistage-test These builders are already on the staging master. So, please ping me if you would like to configure any of them to work with monorepo: * clang-freebsd11-amd64 These builders have been already tested and could be reconfigured without staging as soon as public master is ready: * llvm-sphinx-docs * clang-sphinx-docs * clang-tools-sphinx-docs * lld-sphinx-docs * lldb-sphinx-docs * libcxx-sphinx-docs * libunwind-sphinx-docs * clang-x86_64-debian-fast * libcxx-libcxxabi-x86_64-linux-debian * libcxx-libcxxabi-x86_64-linux-debian-noexceptions * libcxx-libcxxabi-libunwind-x86_64-linux-debian * libcxx-libcxxabi-singlethreaded-x86_64-linux-debian * lld-x86_64-darwin13 * lld-x86_64-win7 * llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast * llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast * clang-x86_64-linux-abi-test * clang-with-lto-ubuntu * clang-with-thin-lto-ubuntu * llvm-clang-x86_64-expensive-checks-win * llvm-clang-x86_64-win-fast * lld-x86_64-ubuntu-fast * clang-lld-x86_64-2stage * lld-perf-testsuite Thanks for your patience and help! Galina On Thu, Oct 17, 2019 at 5:04 PM Galina Kistanova <gkistanova at gmail.com> wrote:> Hello everyone, > > The build bot is almost ready to move to github. > > As the next step we would migrate the staging master to listen for the > both changes, from SVN as it was before, and from github. > > Tonight I am going to update the staging (aka silent) master and it will > start working with github. > > I will send an email when it is done. > > If you bots use one of the ported build factories from the list in my > previous e-mail, please feel free to stage them for working with > monorepo/github and once you are happy, let me know, and I'll configure > them accordingly on the production master. Once that's done, you could move > the bots back to the production master and you are done. > > When you stage your bot, please follow these steps: > 1. Stop your bot between builds, if possible, > 2. Remove the bot working directory (usually this directory has a builder > name and is under directory where your bot is installed; if you are not > sure, check the 'builddir' param of your bot in > zorg/buildbot/osuosl/master/config/builders.py), > 3. Edit buildbot.tac in the directory where your bot is installed. Change > "port = 9990" line to "port = 9994". Save the change. > 4. Start your bot, make sure it connects to the staging master. > 5. Send me a mail with the staged bot names. > > Once you are happy with your bot building monorepo changes from github, > please send me an e-mail and I'll respond with the instructions of how to > get your bot back to production. > > Please be aware that staging master could restart often. Please let me > know if you are having long running builds. > > Feel free to ask if you have questions. > Please me know if you will see issues with the staging master. > > Thanks > > Galina > > On Mon, Oct 14, 2019 at 6:16 PM Galina Kistanova <gkistanova at gmail.com> > wrote: > >> Hello everyone, >> >> >> >> We are in the middle of porting the majority of zorg to >> GitHub/monorepo. The following build factories will be ported and if you >> use one of those for your bots, you are all covered: >> >> >> >> * ClangBuilder.getClangCMakeBuildFactory (31 bots) >> >> * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) >> >> * LibcxxAndAbiBuilder (23 bots) >> >> * SphinxDocsBuilder (7 bots) >> >> * UnifiedTreeBuilder (11 bots) >> >> * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder >> >> * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder >> >> * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder >> >> >> >> Some build factories will be deprecated. If you use one of these, please >> change your bot to use something else instead. Here is the list of >> deprecated build factories: >> >> >> >> * ClangBuilder.getClangBuildFactory (0 bots) >> >> * LLDBuilder (0 bots) >> >> * ClangAndLLDBuilder (0 bots) >> >> >> >> However, some special build factories and build factories with a few bots >> would need your attention. >> >> Here is the list of build factories in need of porting. Patches are >> welcome. >> >> >> >> * LLVMBuilder (3 bots) >> >> * PollyBuilder (3 bots) >> >> * LLDBBuilder (6 bots) >> >> * SanitizerBuilder (10 bots) >> >> * CUDATestsuiteBuilder (1 bot) - depends on >> ClangBuilder.getClangBuildFactory >> >> * AOSPBuilder (1 bot) - depends on PollyBuilder >> >> * AnnotatedBuilder (2 bots) >> >> * OpenMPBuilder (2 bots) >> >> * FuchsiaBuilder (1 bot) >> >> >> >> Please feel free to ask if you have questions. >> >> >> >> Thanks >> >> >> >> Galina >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191018/690fe99f/attachment-0001.html>
Vitaly Buka via llvm-dev
2019-Oct-18 21:35 UTC
[llvm-dev] Zorg migration to GitHub/monorepo
On Mon, Oct 14, 2019 at 6:16 PM Galina Kistanova via cfe-commits < cfe-commits at lists.llvm.org> wrote:> Hello everyone, > > > > We are in the middle of porting the majority of zorg to > GitHub/monorepo. The following build factories will be ported and if you > use one of those for your bots, you are all covered: > > > > * ClangBuilder.getClangCMakeBuildFactory (31 bots) > > * ClangBuilder.getClangCMakeGCSBuildFactory (2 bots) > > * LibcxxAndAbiBuilder (23 bots) > > * SphinxDocsBuilder (7 bots) > > * UnifiedTreeBuilder (11 bots) > > * ABITestsuitBuilder (1 bot) - based on UnifiedTreeBuilder > > * ClangLTOBuilder (2 bots) - based on UnifiedTreeBuilder > > * LLDPerformanceTesuiteBuilder (1 bot) - based on UnifiedTreeBuilder > > > > Some build factories will be deprecated. If you use one of these, please > change your bot to use something else instead. Here is the list of > deprecated build factories: > > > > * ClangBuilder.getClangBuildFactory (0 bots) > > * LLDBuilder (0 bots) > > * ClangAndLLDBuilder (0 bots) > > > > However, some special build factories and build factories with a few bots > would need your attention. > > Here is the list of build factories in need of porting. Patches are > welcome. > > > > * LLVMBuilder (3 bots) > > * PollyBuilder (3 bots) > > * LLDBBuilder (6 bots) > > * SanitizerBuilder (10 bots) >SanitizerBuilder already uses monorepo from github. Still it uses svn BUILDBOT_REVISION. 'BUILDBOT_REVISION' is WithProperties('%(revision:-None)s'). Not sure what is going to happen after migration.> * CUDATestsuiteBuilder (1 bot) - depends on > ClangBuilder.getClangBuildFactory > > * AOSPBuilder (1 bot) - depends on PollyBuilder > > * AnnotatedBuilder (2 bots) > > * OpenMPBuilder (2 bots) > > * FuchsiaBuilder (1 bot) > > > > Please feel free to ask if you have questions. > > > > Thanks > > > > Galina > > _______________________________________________ > cfe-commits mailing list > cfe-commits at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191018/4c1f2dfa/attachment.html>