Hi all,
I'm new to OCFS2, but not so new to DRBD. I'd like to use the new
primary/primary feature of DRBDv8 to create a shared storage space and
concurrently access it from multiple clients, using OCFS2.
I configured two hosts with DRBD, allowed two primaries, and successfully
made each partition primary.
# cat /proc/drbd
version: 8.0pre4 (api:84/proto:82)
SVN Revision: 2375M build by root@moby, 2006-08-17 15:54:17
0: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate r---
ns:0 nr:1398278 dw:1398278 dr:98 al:0 bm:1895 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:86007 misses:1381 starving:0 dirty:0
changed:1381
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
1: cs:Unconfigured
I tried to format the volume with a traditionnal filesystem, and
successfully mounted it on both nodes.
I then tried with ocfs2. On the first node, mkfs and mount went without a
hitch, but on the second one, I systematically get an error when I try to
do anything on the volume (fsck'ing, starting ocfs2-heartbeat, mounting,
etc.). dmesg shows the following,
drbd0: role( Secondary -> Primary )
drbd0: Writing meta data super block now.
(6672,0):o2hb_setup_one_bio:290 ERROR: Error adding page to bio i = 1,
vec_len = 4096, len = 0
, start = 0
(6672,0):o2hb_read_slots:385 ERROR: status = -5
(6672,0):o2hb_populate_slot_data:1279 ERROR: status = -5
(6672,0):o2hb_region_dev_write:1379 ERROR: status = -5
It seems that the heartbeat process can't write to the device, for an
unknown reason:
open("/sys/kernel/config/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) =
4
fstat(4, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
getdents64(4, /* 3 entries */, 4096) = 88
getdents64(4, /* 0 entries */, 4096) = 0
close(4) = 0
mkdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135",
0755) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/block_bytes",
O_WRONLY) = 4
write(4, "512", 3) = 3
close(4) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/start_block",
O_WRONLY) = 4
write(4, "2176", 4) = 4
close(4) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/blocks",
O_WRONLY) = 4
write(4, "255", 3) = 3
close(4) = 0
open("/dev/drbd0", O_RDWR) = 4
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/dev",
O_WRONLY) = 5
write(5, "4", 1) = -1 EIO (Input/output error)
close(5) = 0
close(4) = 0
rmdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135")
= 0
semop(0, 0x7fff930bfe30, 1) = 0
close(3) = 0
write(2, "mkfs.ocfs2", 10mkfs.ocfs2) = 10
write(2, ": ", 2: ) = 2
write(2, "I/O error on channel", 20I/O error on channel) = 20
write(2, " ", 1 ) = 1
write(2, "while initializing the dlm", 26while initializing the dlm) =
26
write(2, "\r\n", 2
I can't figure if it's a DRBD- or a OCFS2-related issue, and I'd
take any
enlightenment with gratitude.
BTW, I use amd64, debian-provided 2.6.17 kernel, drbd8-module-source
8.0pre4-1 (I tried SVN trunk too), and ocfs2-tools 1.2.1-1.
Thanks in advance,
--
Kilian CAVALOTTI Administrateur r?seaux et syst?mes
UPMC / CNRS - LIP6 (C870)
8, rue du Capitaine Scott Tel. : 01 44 27 88 54
75015 Paris - France Fax. : 01 44 27 70 00
As far as ocfs2 is concerned, bio_add_page() is failing. The one thing that springs to mind is that o2hb sets bio->bi_sector to 512 bytes and not the block size. Kilian CAVALOTTI wrote:> Hi all, > > I'm new to OCFS2, but not so new to DRBD. I'd like to use the new > primary/primary feature of DRBDv8 to create a shared storage space and > concurrently access it from multiple clients, using OCFS2. > > I configured two hosts with DRBD, allowed two primaries, and successfully > made each partition primary. > > # cat /proc/drbd > version: 8.0pre4 (api:84/proto:82) > SVN Revision: 2375M build by root@moby, 2006-08-17 15:54:17 > 0: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate r--- > ns:0 nr:1398278 dw:1398278 dr:98 al:0 bm:1895 lo:0 pe:0 ua:0 ap:0 > resync: used:0/7 hits:86007 misses:1381 starving:0 dirty:0 > changed:1381 > act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0 > 1: cs:Unconfigured > > I tried to format the volume with a traditionnal filesystem, and > successfully mounted it on both nodes. > > I then tried with ocfs2. On the first node, mkfs and mount went without a > hitch, but on the second one, I systematically get an error when I try to > do anything on the volume (fsck'ing, starting ocfs2-heartbeat, mounting, > etc.). dmesg shows the following, > > drbd0: role( Secondary -> Primary ) > drbd0: Writing meta data super block now. > (6672,0):o2hb_setup_one_bio:290 ERROR: Error adding page to bio i = 1, > vec_len = 4096, len = 0 > , start = 0 > (6672,0):o2hb_read_slots:385 ERROR: status = -5 > (6672,0):o2hb_populate_slot_data:1279 ERROR: status = -5 > (6672,0):o2hb_region_dev_write:1379 ERROR: status = -5 > > > It seems that the heartbeat process can't write to the device, for an > unknown reason: > > open("/sys/kernel/config/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4 > fstat(4, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 > fcntl(4, F_SETFD, FD_CLOEXEC) = 0 > getdents64(4, /* 3 entries */, 4096) = 88 > getdents64(4, /* 0 entries */, 4096) = 0 > close(4) = 0 > mkdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135", > 0755) = 0 > open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/block_bytes", > O_WRONLY) = 4 > write(4, "512", 3) = 3 > close(4) = 0 > open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/start_block", > O_WRONLY) = 4 > write(4, "2176", 4) = 4 > close(4) = 0 > open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/blocks", > O_WRONLY) = 4 > write(4, "255", 3) = 3 > close(4) = 0 > open("/dev/drbd0", O_RDWR) = 4 > open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135/dev", > O_WRONLY) = 5 > write(5, "4", 1) = -1 EIO (Input/output error) > close(5) = 0 > close(4) = 0 > rmdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0650A1FEF09135") > = 0 > semop(0, 0x7fff930bfe30, 1) = 0 > close(3) = 0 > write(2, "mkfs.ocfs2", 10mkfs.ocfs2) = 10 > write(2, ": ", 2: ) = 2 > write(2, "I/O error on channel", 20I/O error on channel) = 20 > write(2, " ", 1 ) = 1 > write(2, "while initializing the dlm", 26while initializing the dlm) = 26 > write(2, "\r\n", 2 > > I can't figure if it's a DRBD- or a OCFS2-related issue, and I'd take any > enlightenment with gratitude. > > BTW, I use amd64, debian-provided 2.6.17 kernel, drbd8-module-source > 8.0pre4-1 (I tried SVN trunk too), and ocfs2-tools 1.2.1-1. > > Thanks in advance, >
I advice against it. Even if it is possible, you definitkly will run into
DRBD/o2cb conflict in case of any network glitch,
and will have all your nodes self-fencing. In addition, DRDB is not fast
enough for OCFSv2 default settings.
Through, it all may be very interesting to discover in the lab.
----- Original Message -----
From: "Kilian CAVALOTTI" <kilian.cavalotti@lip6.fr>
To: <ocfs2-users@oss.oracle.com>; <drbd-user@linbit.com>
Sent: Thursday, August 17, 2006 7:48 AM
Subject: [Ocfs2-users] OCFS2 over DRBDv8
Hi all,
I'm new to OCFS2, but not so new to DRBD. I'd like to use the new
primary/primary feature of DRBDv8 to create a shared storage space and
concurrently access it from multiple clients, using OCFS2.
I configured two hosts with DRBD, allowed two primaries, and successfully
made each partition primary.
# cat /proc/drbd
version: 8.0pre4 (api:84/proto:82)
SVN Revision: 2375M build by root@moby, 2006-08-17 15:54:17
0: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate r---
ns:0 nr:1398278 dw:1398278 dr:98 al:0 bm:1895 lo:0 pe:0 ua:0 ap:0
resync: used:0/7 hits:86007 misses:1381 starving:0 dirty:0
changed:1381
act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0
1: cs:Unconfigured
I tried to format the volume with a traditionnal filesystem, and
successfully mounted it on both nodes.
I then tried with ocfs2. On the first node, mkfs and mount went without a
hitch, but on the second one, I systematically get an error when I try to
do anything on the volume (fsck'ing, starting ocfs2-heartbeat, mounting,
etc.). dmesg shows the following,
drbd0: role( Secondary -> Primary )
drbd0: Writing meta data super block now.
(6672,0):o2hb_setup_one_bio:290 ERROR: Error adding page to bio i = 1,
vec_len = 4096, len = 0
, start = 0
(6672,0):o2hb_read_slots:385 ERROR: status = -5
(6672,0):o2hb_populate_slot_data:1279 ERROR: status = -5
(6672,0):o2hb_region_dev_write:1379 ERROR: status = -5
It seems that the heartbeat process can't write to the device, for an
unknown reason:
open("/sys/kernel/config/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) =
4
fstat(4, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
getdents64(4, /* 3 entries */, 4096) = 88
getdents64(4, /* 0 entries */, 4096) = 0
close(4) = 0
mkdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF
0650A1FEF09135",
0755) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0
650A1FEF09135/block_bytes",
O_WRONLY) = 4
write(4, "512", 3) = 3
close(4) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0
650A1FEF09135/start_block",
O_WRONLY) = 4
write(4, "2176", 4) = 4
close(4) = 0
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0
650A1FEF09135/blocks",
O_WRONLY) = 4
write(4, "255", 3) = 3
close(4) = 0
open("/dev/drbd0", O_RDWR) = 4
open("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF0
650A1FEF09135/dev",
O_WRONLY) = 5
write(5, "4", 1) = -1 EIO (Input/output error)
close(5) = 0
close(4) = 0
rmdir("/sys/kernel/config/cluster/ocfs2_cluster/heartbeat/D6F76726AFE4472CBF
0650A1FEF09135")
= 0
semop(0, 0x7fff930bfe30, 1) = 0
close(3) = 0
write(2, "mkfs.ocfs2", 10mkfs.ocfs2) = 10
write(2, ": ", 2: ) = 2
write(2, "I/O error on channel", 20I/O error on channel) = 20
write(2, " ", 1 ) = 1
write(2, "while initializing the dlm", 26while initializing the dlm) =
26
write(2, "\r\n", 2
I can't figure if it's a DRBD- or a OCFS2-related issue, and I'd
take any
enlightenment with gratitude.
BTW, I use amd64, debian-provided 2.6.17 kernel, drbd8-module-source
8.0pre4-1 (I tried SVN trunk too), and ocfs2-tools 1.2.1-1.
Thanks in advance,
--
Kilian CAVALOTTI Administrateur r?seaux et syst?mes
UPMC / CNRS - LIP6 (C870)
8, rue du Capitaine Scott Tel. : 01 44 27 88 54
75015 Paris - France Fax. : 01 44 27 70 00
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
On Friday 18 August 2006 15:04, Lars Ellenberg wrote:> we discovered an issue in the way we currently stack the devices limits > (max_segment_size, merge_bvec_fn), we not always report the true limits > of the underlying device correctly. > we currently are looking into that. > > what are your lower level devices? (sorry if thats somewhere in your > mails, I just short-circuited on the "Error adding page").No probs. :) /dev/drbd0 is defined as: on moby { device /dev/drbd0; disk /dev/md2; address 10.0.0.2:7788; flexible-meta-disk internal; } and /dev/md2 is a RAID0 MD device, 2.5TB in size, assembled from two hardware RAID5 volumes created on two ICP Vortex SATA-RAID cards [1] (/dev/sda1 and /dev/sda2) moby:~# mdadm -D /dev/md2 /dev/md2: Version : 00.90.03 Creation Time : Mon May 15 10:53:38 2006 Raid Level : raid0 Array Size : 2637013248 (2514.85 GiB 2700.30 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Mon May 15 10:53:38 2006 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 64K UUID : 5e5ff5d1:fbcae14a:ac747fac:987c5d3a Events : 0.2 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 [1] http://www.icp-vortex.com/english/product/pci/rz_sata_8/8586rz_e.htm -- Kilian CAVALOTTI Administrateur r?seaux et syst?mes UPMC / CNRS - LIP6 (C870) 8, rue du Capitaine Scott Tel. : 01 44 27 88 54 75015 Paris - France Fax. : 01 44 27 70 00