On 10/26/2016 02:54 PM, Lindsay Mathieson wrote:> Maybe a controversial question (and hopefully not trolling), but any > particularly reason you choose gluster over ceph for these larger > setups Joe? > > For myself, gluster is much easier to manage and provides better > performance on my small non-enterprise setup, plus it plays nice with > zfs. > > But I thought ceph had the edge on large, many node, many disk setups. > It would seem it handles adding/removing disks better that the > juggling you have to do with gluster to keep replication triads even. > > To complex/fragile maybe? > > Genuinely curious. >I need to put together a whole presentation on this, but I'm not yet sure how much I can say yet. For now I can say that gluster performs better and has a much better worst-case resolution. If everything else goes to hell, I have disks with files on them that I can recover on a laptop if I have to. Of course when you ask the Inktank consultants (now Red Hat) about "What happens when it fails?" the answer is "It doesn't fail." Well guess what... To be fair, though, I can't blame ceph. We had a cascading hardware failure with those storage trays. Even still, if it had been gluster - I would have had files on disks.
On 27/10/2016 8:14 AM, Joe Julian wrote:> To be fair, though, I can't blame ceph. We had a cascading hardware > failure with those storage trays. Even still, if it had been gluster - > I would have had files on disks.Ouch :( In that regard how do you view sharding? why not as simple as pulling a whole file off disk it still seems straightforward. I haven't tried it yet (really should) but it should just be a matter of getting the guid for a filename, then concatenating the shards in order by their extension. -- Lindsay Mathieson
2016-10-27 0:14 GMT+02:00 Joe Julian <joe at julianfamily.org>:> For now I can say that gluster performs better and has a much better > worst-case resolution. If everything else goes to hell, I have disks with > files on them that I can recover on a laptop if I have to.Totally agree.> Of course when you ask the Inktank consultants (now Red Hat) about "What > happens when it fails?" the answer is "It doesn't fail." Well guess what...It doesn't fail? There was a recent HUGE outage in DreamHost cluster, some days ago. It took (if i remember properly 2 days to fix, and this was dreamhost, not a "normal" company) There was another huge outage some weeks ago, as seen on ceph ML. Is not the first time that I see ceph clusters totally down or with huge corruption (LevelDB corrupted and so on) So, it fails, like any other software. As I always say, having the file untouched and easily recoverable (without shards, you can simply run "rsync", with shard you run rsync+some script to mege each chunk) is the real value added, when everything is gonna bad.