I recently (re)built a fileserver at home, using Ubuntu and zfs-fuse to create a ZFS filesystem (RAIDz1) on five 1.5TB drives. I had some serious issues with NFS not working properly (kept getting stale file handles), so I tried to switch to OpenSolaris/Nexenta, but my SATA controller wasn''t supported. I went to FreeBSD, and got ZFS working there, and was able to import the ZFS pool that I had created under Linux and zfs-fuse. But I had issues with kernel panics. Finally, I found a SATA card that would work with Solaris (an old LSI 150-6). I upgraded the firmware and turned off the BIOS (so it would act as a plain SATA card, rather than doing RAID), and I could finally access the drives under Solaris. Now my problem is that even though Solaris can see the drives, and recognizes that I have a ZFS pool, it won''t import it. This isn''t a case of using -f to force the import. Rather, even though the drives are all online and showing as "available", ''zpool import'' says I have "insufficient replicas" and that the raidz is unavailable due to "corrupted data." (I can post screen caps later today.) I can reboot into Linux and import the pools, but haven''t figured out why I can''t import them in Solaris. I don''t know if it makes a difference (I wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, where Nexenta is using version 14. Any ideas? Paul
On Thu, Sep 17, 2009 at 8:55 PM, Paul Archer <paul at paularcher.org> wrote:> I can reboot into Linux and import the pools, but haven''t figured out why I > can''t import them in Solaris. I don''t know if it makes a difference (I > wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, where > Nexenta is using version 14.Just a guess, but did you use the whole drive while creating the pool on Linux? Something like "zpool create poolname raidz sda sdb sdc" ? -- Fajar
10:09pm, Fajar A. Nugraha wrote:> On Thu, Sep 17, 2009 at 8:55 PM, Paul Archer <paul at paularcher.org> wrote: >> I can reboot into Linux and import the pools, but haven''t figured out why I >> can''t import them in Solaris. I don''t know if it makes a difference (I >> wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, where >> Nexenta is using version 14. > > Just a guess, but did you use the whole drive while creating the pool > on Linux? Something like "zpool create poolname raidz sda sdb sdc" ? >Yes, I did. I was under the impression that was the way to go. If it''s not (ie it needs to be a single disk-sized partion), I can try moving. I''m assuming if I add a partition, I can do something like: zpool replace datapool sda sda1 Paul
Paul Archer wrote:> 10:09pm, Fajar A. Nugraha wrote: > >> On Thu, Sep 17, 2009 at 8:55 PM, Paul Archer <paul at paularcher.org> wrote: >>> I can reboot into Linux and import the pools, but haven''t figured out >>> why I >>> can''t import them in Solaris. I don''t know if it makes a difference (I >>> wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, >>> where >>> Nexenta is using version 14. >> >> Just a guess, but did you use the whole drive while creating the pool >> on Linux? Something like "zpool create poolname raidz sda sdb sdc" ? >> > Yes, I did. I was under the impression that was the way to go. If it''s > not (ie it needs to be a single disk-sized partion), I can try moving. > I''m assuming if I add a partition, I can do something like: > zpool replace datapool sda sda1What kind of partition table is on the disks, is it EFI ? If not that might be part of the issue. -- Darren J Moffat
10:40am, Paul Archer wrote:>>> I can reboot into Linux and import the pools, but haven''t figured out why >>> I >>> can''t import them in Solaris. I don''t know if it makes a difference (I >>> wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, >>> where >>> Nexenta is using version 14. >> >> Just a guess, but did you use the whole drive while creating the pool >> on Linux? Something like "zpool create poolname raidz sda sdb sdc" ? >> > Yes, I did. I was under the impression that was the way to go. If it''s not > (ie it needs to be a single disk-sized partion), I can try moving. I''m > assuming if I add a partition, I can do something like: > zpool replace datapool sda sda1 >Or not. I see that if I try to replace sda with sda1, zpool complains that sda1 is too small. Any suggestions (that don''t include ''start over'')? Paul
5:08pm, Darren J Moffat wrote:> Paul Archer wrote: >> 10:09pm, Fajar A. Nugraha wrote: >> >>> On Thu, Sep 17, 2009 at 8:55 PM, Paul Archer <paul at paularcher.org> wrote: >>>> I can reboot into Linux and import the pools, but haven''t figured out why >>>> I >>>> can''t import them in Solaris. I don''t know if it makes a difference (I >>>> wouldn''t think so), but zfs-fuse under Linux is using ZFS version 13, >>>> where >>>> Nexenta is using version 14. >>> >>> Just a guess, but did you use the whole drive while creating the pool >>> on Linux? Something like "zpool create poolname raidz sda sdb sdc" ? >>> >> Yes, I did. I was under the impression that was the way to go. If it''s not >> (ie it needs to be a single disk-sized partion), I can try moving. I''m >> assuming if I add a partition, I can do something like: >> zpool replace datapool sda sda1 > > What kind of partition table is on the disks, is it EFI ? If not that might > be part of the issue. >I don''t believe there is any partition table on the disks. I pointed zfs to the raw disks when I setup the pool. Paul
Paul Archer wrote:>> What kind of partition table is on the disks, is it EFI ? If not that >> might be part of the issue. >> > I don''t believe there is any partition table on the disks. I pointed zfs > to the raw disks when I setup the pool.If you run fdisk on OpenSolaris against this disk what does it show as the partition type eg: fdisk -v /dev/rdsk/c7t4d0p0 Mine shows: 1 EFI 0 45600 45601 100 Which tells me I have an EFI label on the disk. My boot ZFS pool shows this: one one side of the mirror: 1 Diagnostic 0 3 4 0 2 Active Solaris2 4 45599 45596 100 and on the other: 1 Active Solaris2 1 45599 45599 100 -- Darren J Moffat
6:44pm, Darren J Moffat wrote:> Paul Archer wrote: >>> What kind of partition table is on the disks, is it EFI ? If not that >>> might be part of the issue. >>> >> I don''t believe there is any partition table on the disks. I pointed zfs to >> the raw disks when I setup the pool. > > If you run fdisk on OpenSolaris against this disk what does it show as the > partition type eg: > > fdisk -v /dev/rdsk/c7t4d0p0 > > Mine shows: > > 1 EFI 0 45600 45601 100 > > Which tells me I have an EFI label on the disk. > > My boot ZFS pool shows this: > > one one side of the mirror: > > 1 Diagnostic 0 3 4 0 > 2 Active Solaris2 4 45599 45596 100 > and on the other: > > 1 Active Solaris2 1 45599 45599 100 > > --I just took a look, and it seems that all the drives have a single partition on them. I''m looking under Linux, as I can''t reboot it into Solaris again until I get home tonight. root at ubuntu:~# fdisk -l /dev/sda Disk /dev/sda: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xce13f90b Device Boot Start End Blocks Id System /dev/sda1 1 182401 1465136001 83 Linux
Paul Archer wrote:> root at ubuntu:~# fdisk -l /dev/sda > > Disk /dev/sda: 1500.3 GB, 1500301910016 bytes > 255 heads, 63 sectors/track, 182401 cylinders > Units = cylinders of 16065 * 512 = 8225280 bytes > Disk identifier: 0xce13f90b > > Device Boot Start End Blocks Id System > /dev/sda1 1 182401 1465136001 83 LinuxThat is good enough. That is your problem right there. Solaris doesn''t recognise this partition type. FreeBSD I think does. I''m not sure what you can do to get Solaris to recognise this. If there is a non destructive way under Linux to change this to an EFI partition that would be a good way to start. I doubt that simply changing the tag from Linux (32) to Solaris2 (191) would be enough since you would lack the vtoc in there. Plus ideally you want this as EFI unless you need to put OpenSolaris into that pool to boot from it - but sounds like you don''t. -- Darren J Moffat
7:37pm, Darren J Moffat wrote:> Paul Archer wrote: >> root at ubuntu:~# fdisk -l /dev/sda >> >> Disk /dev/sda: 1500.3 GB, 1500301910016 bytes >> 255 heads, 63 sectors/track, 182401 cylinders >> Units = cylinders of 16065 * 512 = 8225280 bytes >> Disk identifier: 0xce13f90b >> >> Device Boot Start End Blocks Id System >> /dev/sda1 1 182401 1465136001 83 Linux > > That is good enough. That is your problem right there. Solaris doesn''t > recognise this partition type. FreeBSD I think does. > > I''m not sure what you can do to get Solaris to recognise this. If there is a > non destructive way under Linux to change this to an EFI partition that would > be a good way to start. > > I doubt that simply changing the tag from Linux (32) to Solaris2 (191) would > be enough since you would lack the vtoc in there. Plus ideally you want this > as EFI unless you need to put OpenSolaris into that pool to boot from it - > but sounds like you don''t. >I did a little research and found that parted on Linux handles EFI labelling. I used it to change the partition scheme on sda, creating an sda1. I then offlined sda and replaced it with sda1. I wish I had just tried a scrub instead of the replace, though, as I''ve gotta wait about 35 hours for the resilver to finish. (1.5TB data on five disks with a single PCI controller card.) Paul
Darren J Moffat wrote:> Paul Archer wrote: >> root at ubuntu:~# fdisk -l /dev/sda >> >> Disk /dev/sda: 1500.3 GB, 1500301910016 bytes >> 255 heads, 63 sectors/track, 182401 cylinders >> Units = cylinders of 16065 * 512 = 8225280 bytes >> Disk identifier: 0xce13f90b >> >> Device Boot Start End Blocks Id System >> /dev/sda1 1 182401 1465136001 83 Linux > > That is good enough. That is your problem right there. Solaris doesn''t > recognise this partition type. FreeBSD I think does. > > I''m not sure what you can do to get Solaris to recognise this. If there > is a non destructive way under Linux to change this to an EFI partition > that would be a good way to start.ZFS should just find this as /dev/dsk/c?t?d?p1 if the structures really are unrecognizable as a zpool device (on x86 only - won''t work on sparc). I haven''t tried importing in this way, but I''m pretty sure I have created zpools on raw primary fdisk partitions.> I doubt that simply changing the tag from Linux (32) to Solaris2 (191) > would be enough since you would lack the vtoc in there.You really don''t want it to be a partition type Solaris recognizes as something else.> Plus ideally > you want this as EFI unless you need to put OpenSolaris into that pool > to boot from it - but sounds like you don''t.-- Andrew
On Fri, Sep 18, 2009 at 4:08 AM, Paul Archer <paul at paularcher.org> wrote:> I did a little research and found that parted on Linux handles EFI > labelling. I used it to change the partition scheme on sda, creating an > sda1. I then offlined sda and replaced it with sda1. I wish I had just tried > a scrub instead of the replace, though, as I''ve gotta wait about 35 hours > for the resilver to finish. (1.5TB data on five disks with a single PCI > controller card.)There was a post from Ricardo on zfs-fuse list some time ago. Apparently if you do a "zpool create" on whole disks, Linux on Solaris behaves differently: - solaris will create EFI partition on that disk, and use the partition as vdev - Linux will use the whole disk without any partition, just like with a file-based vdev. The result is that you might be unable to import the pool on *solaris or *BSD. The recommended way to create a "portable" pool is to create the pool on a partition setup recognizable on all those OS. He suggested a simple DOS/MBR partition table. So in short, if you had created the pool on top of sda1 instead of sda, it will work. I''m surprised though that you can "offlined sda and replaced it with sda1" when previously you said "I see that if I try to replace sda with sda1, zpool complains that sda1 is too small" -- Fajar
Tomorrow, Fajar A. Nugraha wrote:> There was a post from Ricardo on zfs-fuse list some time ago. > Apparently if you do a "zpool create" on whole disks, Linux on > Solaris behaves differently: > - solaris will create EFI partition on that disk, and use the partition as vdev > - Linux will use the whole disk without any partition, just like with > a file-based vdev. > > The result is that you might be unable to import the pool on *solaris or *BSD. > > The recommended way to create a "portable" pool is to create the pool > on a partition setup recognizable on all those OS. He suggested a > simple DOS/MBR partition table. > > So in short, if you had created the pool on top of sda1 instead of > sda, it will work. I''m surprised though that you can "offlined sda and > replaced it with sda1" when previously you said "I see that if I try > to replace sda with sda1, zpool complains that sda1 is too small" >I was a bit surprised about that, too. But I found that a standard PC/Linux partition reserves around 24MB at the beginning of the disk, and an EFI (or actually, GPT) disklabel and partition only uses a few 100KB. Paul
On 18/09/2009, at 1:08 PM, Fajar A. Nugraha wrote:> There was a post from Ricardo on zfs-fuse list some time ago. > Apparently if you do a "zpool create" on whole disks, Linux on > Solaris behaves differently: > - solaris will create EFI partition on that disk, and use the > partition as vdev > - Linux will use the whole disk without any partition, just like with > a file-based vdev. > > The result is that you might be unable to import the pool on > *solaris or *BSD. > > The recommended way to create a "portable" pool is to create the pool > on a partition setup recognizable on all those OS. He suggested a > simple DOS/MBR partition table.I haven''t tried this on OSX yet, but I was fighting with this problem for quite a while with pools created on raw disk devices under FreeBSD. I''m still confused as to why Solaris won''t recognise them, but both Solaris and FreeBSD will happily import/export pools created on an EFI/GPT partition using FreeBSD''s gpart(1), so if Linux understands GPT partitions as well that''s probably the way to go for portable pools. FreeBSD 7.2-STABLE will NOT see pools created by OpenSolaris though, again I don''t understand why this is since it should in theory understand EFI partition tables fine. I''ll try importing a pool on a USB disk with a GPT table on my mac and see what happens. -- Alasdair Muckart | William de Wyke | http://wherearetheelves.blogspot.com "Any sufficiently advanced stupididty is indistinguishable from malice" -- James D. Macdonald
Paul Archer
2009-Sep-20 19:10 UTC
[zfs-discuss] SOLVED: Re: migrating from linux to solaris ZFS
Thursday, Paul Archer wrote:> Tomorrow, Fajar A. Nugraha wrote: > >> There was a post from Ricardo on zfs-fuse list some time ago. >> Apparently if you do a "zpool create" on whole disks, Linux on >> Solaris behaves differently: >> - solaris will create EFI partition on that disk, and use the partition as >> vdev >> - Linux will use the whole disk without any partition, just like with >> a file-based vdev. >> >> The result is that you might be unable to import the pool on *solaris or >> *BSD. >> >> The recommended way to create a "portable" pool is to create the pool >> on a partition setup recognizable on all those OS. He suggested a >> simple DOS/MBR partition table. >> >> So in short, if you had created the pool on top of sda1 instead of >> sda, it will work. I''m surprised though that you can "offlined sda and >> replaced it with sda1" when previously you said "I see that if I try >> to replace sda with sda1, zpool complains that sda1 is too small" >> > > I was a bit surprised about that, too. But I found that a standard PC/Linux > partition reserves around 24MB at the beginning of the disk, and an EFI (or > actually, GPT) disklabel and partition only uses a few 100KB. >As I mentioned above, I created GPT disklabels and partitions on all my disks, then one-by-one offlined the disk and replaced it with the partition from the same disk (eg ''zpool replace datapool ad1 ad1p1''). I did the first replacement with Linux and zfs-fuse. The resilver took 32 hours. I did the rest in FreeBSD, which took 5-6 hours for each disk. It was tedious, but the pool is available in Solaris (finally!), so hopefully no more NFS issues or kernel panics. (I had NFS issues with both Linux and BSD, and kernel panics with BSD.) Paul PS. Complicating matters was the fact that for some reason, BSD didn''t like my LSI 150-6 SATA card (which is the only one Solaris plays nice with), so I had to keep switching cards every time I went from one OS to the other. Blech. OTOH, here''s to Live CDs!