On Mon, 13 Jun 2022 at 05:17, Peter Hutterer <peter.hutterer at who-t.net> wrote:> On Sun, Jun 12, 2022 at 05:57:05PM -0700, Jeremy Sequoia wrote: > > I was going to spend a little bit of time putting out an update to XQuartz > > to address a few bugs that I've been meaning to squash, but I'm having a bit > > of an issue pulling down sources. > > > > Fetching via ssh://git at gitlab.freedesktop.org is giving me Permission denied > > (publickey,keyboard-interactive). I'm not sure if the latter is an infra > > issue or if the ssh key I have stored in my gitlab account are out of date > > (it's been about a year since I touched this). Unfortunately, I can't seem > > to access https://gitlab.freedesktop.org to check as it's constantly > > presenting me a 504 Gateway timeout. > > > > Is anyone else able to pull sources via ssh://git at gitlab.freedesktop.org > > right now? Is someone looking into the 504 issue? > > not an fdo admin but judging by the chatter on #freedesktop: no and yes, in > that order. seems like the infrastructure is in various stages of depositing > fecal matter on itself and the fixes are involved enough that the admins have > to be mentally awake, not merely physically.Yes, that's what's happening. Our (multi-host-replicated etc) Ceph storage setup has entered a degraded mode due to the loss of a couple of disks - no data has been lost but the cluster is currently unhappy. We're walking through fixing this, but have bumped into some other issues since, including a newly-flaky network setup, and changes since we last provisioned a new storage host. We're working through them one by one and will have the service back up with all our data intact - hopefully in a matter of hours but we have no firm ETA right now. Cheers, Daniel
https://www.phoronix.com/scan.php?page=news_item&px=FreeDesktop-GitLab-2022-Crash On Mon, Jun 13, 2022 at 9:40 AM Daniel Stone <daniel at fooishbar.org> wrote:> On Mon, 13 Jun 2022 at 05:17, Peter Hutterer <peter.hutterer at who-t.net> > wrote: > > On Sun, Jun 12, 2022 at 05:57:05PM -0700, Jeremy Sequoia wrote: > > > I was going to spend a little bit of time putting out an update to > XQuartz > > > to address a few bugs that I've been meaning to squash, but I'm having > a bit > > > of an issue pulling down sources. > > > > > > Fetching via ssh://git at gitlab.freedesktop.org is giving me Permission > denied > > > (publickey,keyboard-interactive). I'm not sure if the latter is an > infra > > > issue or if the ssh key I have stored in my gitlab account are out of > date > > > (it's been about a year since I touched this). Unfortunately, I can't > seem > > > to access https://gitlab.freedesktop.org to check as it's constantly > > > presenting me a 504 Gateway timeout. > > > > > > Is anyone else able to pull sources via ssh:// > git at gitlab.freedesktop.org > > > right now? Is someone looking into the 504 issue? > > > > not an fdo admin but judging by the chatter on #freedesktop: no and yes, > in > > that order. seems like the infrastructure is in various stages of > depositing > > fecal matter on itself and the fixes are involved enough that the admins > have > > to be mentally awake, not merely physically. > > Yes, that's what's happening. Our (multi-host-replicated etc) Ceph > storage setup has entered a degraded mode due to the loss of a couple > of disks - no data has been lost but the cluster is currently unhappy. > We're walking through fixing this, but have bumped into some other > issues since, including a newly-flaky network setup, and changes since > we last provisioned a new storage host. > > We're working through them one by one and will have the service back > up with all our data intact - hopefully in a matter of hours but we > have no firm ETA right now. > > Cheers, > Daniel >-------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.freedesktop.org/archives/freedesktop/attachments/20220613/cb5c535d/attachment.htm>
Hi, On Mon, 13 Jun 2022 at 08:39, Daniel Stone <daniel at fooishbar.org> wrote:> Yes, that's what's happening. Our (multi-host-replicated etc) Ceph > storage setup has entered a degraded mode due to the loss of a couple > of disks - no data has been lost but the cluster is currently unhappy. > We're walking through fixing this, but have bumped into some other > issues since, including a newly-flaky network setup, and changes since > we last provisioned a new storage host. > > We're working through them one by one and will have the service back > up with all our data intact - hopefully in a matter of hours but we > have no firm ETA right now.Thanks mainly to Ben, everything is back up and running now. Cheers, Daniel