Jesse Reynolds
2010-Feb-26 14:03 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
Hello
I have an amd64 server running OpenSolaris 2009-06. In December I created one
container on this server named ''cpmail'' with it''s own
zfs dataset and it''s been running ever since. Until earlier this
evening when the server did a kernel panic and rebooted. Now, I can''t
see any contents in the zfs dataset for this zone!
The server has two disks which are root mirrored with ZFS:
# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c8t0d0s0 ONLINE 0 0 0
c8t1d0s0 ONLINE 0 0 0
errors: No known data errors
Here are the datasets:
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 161G 67.6G 79.5K /rpool
rpool/ROOT 3.66G 67.6G 19K legacy
rpool/ROOT/opensolaris 3.66G 67.6G 3.51G /
rpool/cpmail 139G 67.6G 22K /zones/cpmail
rpool/cpmail/ROOT 139G 67.6G 19K legacy
rpool/cpmail/ROOT/zbe 139G 67.6G 139G legacy
rpool/dump 2.00G 67.6G 2.00G -
rpool/export 7.64G 67.6G 7.49G /export
rpool/export/home 150M 67.6G 21K /export/home
rpool/export/home/jesse 150M 67.6G 150M /export/home/jesse
rpool/repo 6.56G 67.6G 6.56G /rpool/repo
rpool/swap 2.00G 69.4G 130M -
/zones/cpmail is where it should be mounting the zone''s dataset, I
believe.
Here''s what happens when I try and start the zone:
# zoneadm -z cpmail boot
could not verify zfs dataset mailtmp: dataset does not exist
zoneadm: zone cpmail failed to verify
So the zone is trying to find a dataset ''mailtmp'' and failing
because it doesn''t exist. So, what happened to it?
Here''s the zone config file, at /etc/zones/cpmail.xml (with IP address
obfuscated)
# cat /etc/zones/cpmail.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN"
"file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
<!--
DO NOT EDIT THIS FILE. Use zonecfg(1M) instead.
-->
<zone name="cpmail" zonepath="/zones/cpmail"
autoboot="false" brand="ipkg">
<network address="xxx.xxx.xxx.xxx" physical="bge1"/>
<dataset name="mailtmp"/>
</zone>
I just don''t understand where the dataset ''mailtmp''
went to. Perhaps it was an initial name I used for the dataset and I then
renamed it to cpmail, but then I can''t see any of the zones files in
/zones/cpmail :
# find /zones/cpmail/
/zones/cpmail/
/zones/cpmail/dev
/zones/cpmail/root
Does ZFS store a log file of all operations applied to it? It feels like someone
has gained access and run ''zfs destroy mailtmp'' to me, but
then again it could just be my own ineptitude.
Thank you
Jesse
--
This message posted from opensolaris.org
Andrew Gabriel
2010-Feb-26 14:14 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
Jesse Reynolds wrote:> Does ZFS store a log file of all operations applied to it? It feels like someone has gained access and run ''zfs destroy mailtmp'' to me, but then again it could just be my own ineptitude.Yes... zpool history rpool -- Andrew Gabriel
Jesse Reynolds
2010-Feb-26 14:31 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
Thanks Andrew! Sorry to be dumb. So, here''s the whole history: root at marmoset:/rpool/repo/updatelog# zpool history rpool History for ''rpool'': 2009-11-22.21:10:34 zpool create -f rpool c8t0d0s0 2009-11-22.21:10:35 zfs set org.opensolaris.caiman:install=busy rpool 2009-11-22.21:10:36 zfs create -b 4096 -V 2047m rpool/swap 2009-11-22.21:10:37 zfs create -b 131072 -V 2047m rpool/dump 2009-11-22.21:10:40 zfs set mountpoint=/a/export rpool/export 2009-11-22.21:10:40 zfs set mountpoint=/a/export/home rpool/export/home 2009-11-22.21:10:40 zfs set mountpoint=/a/export/home/jesse rpool/export/home/jesse 2009-11-22.21:20:08 zpool set bootfs=rpool/ROOT/opensolaris rpool 2009-11-22.21:21:14 zfs set org.opensolaris.caiman:install=ready rpool 2009-11-22.21:21:31 zfs set mountpoint=/export/home/jesse rpool/export/home/jesse 2009-11-22.21:21:31 zfs set mountpoint=/export/home rpool/export/home 2009-11-22.21:21:32 zfs set mountpoint=/export rpool/export 2009-11-27.16:29:42 zfs create -o compression=on rpool/repo 2009-11-30.01:08:03 zfs set atime=off rpool/repo 2009-12-09.13:12:37 zfs create -p rpool/zones/cpmail 2009-12-09.13:14:50 zfs create -o mountpoint=/zones/cpmail rpool/cpmail 2009-12-09.13:15:24 zfs destroy rpool/zones/cpmail 2009-12-09.13:15:34 zfs destroy rpool/zones 2009-12-09.13:17:53 zfs create -o mountpoint=legacy -o zoned=on rpool/cpmail/ROOT 2009-12-09.13:17:53 zfs create -o org.opensolaris.libbe:active=on -o org.opensolaris.libbe:parentbe=2e4070c4-8df3-4b4d-fc14-94164d8a3dcc -o canmount=noauto rpool/cpmail/ROOT/zbe 2009-12-13.17:15:52 zfs snapshot rpool/cpmail at newmail 2009-12-14.00:05:36 zpool attach -f rpool c8t0d0s0 c8t1d0s0 The creation of the zfs filesystem for this zone is here: 2009-12-09.13:17:53 zfs create -o mountpoint=legacy -o zoned=on rpool/cpmail/ROOT 2009-12-09.13:17:53 zfs create -o org.opensolaris.libbe:active=on -o org.opensolaris.libbe:parentbe=2e4070c4-8df3-4b4d-fc14-94164d8a3dcc -o canmount=noauto rpool/cpmail/ROOT/zbe So, how do I inspect that this is actually intact before proceeding with fixing up the zone configuration so it''s pointing to the right dataset? I''m scared to run zonecfg to ''fix it up'' to point to rpool/cpmail/ROOT in case it makes things worse. What''s the best way to proceed here? Indeed, how did the zone manifest get ''out of sync'' with reality I wonder? Thank you Jesse -- This message posted from opensolaris.org
Robert Milkowski
2010-Feb-26 14:43 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
On 26/02/2010 14:03, Jesse Reynolds wrote:> Hello > > I have an amd64 server running OpenSolaris 2009-06. In December I created one container on this server named ''cpmail'' with it''s own zfs dataset and it''s been running ever since. Until earlier this evening when the server did a kernel panic and rebooted. Now, I can''t see any contents in the zfs dataset for this zone! > > The server has two disks which are root mirrored with ZFS: > > # zpool status > pool: rpool > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > rpool ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c8t0d0s0 ONLINE 0 0 0 > c8t1d0s0 ONLINE 0 0 0 > > errors: No known data errors > > Here are the datasets: > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > rpool 161G 67.6G 79.5K /rpool > rpool/ROOT 3.66G 67.6G 19K legacy > rpool/ROOT/opensolaris 3.66G 67.6G 3.51G / > rpool/cpmail 139G 67.6G 22K /zones/cpmail > rpool/cpmail/ROOT 139G 67.6G 19K legacy > rpool/cpmail/ROOT/zbe 139G 67.6G 139G legacy > rpool/dump 2.00G 67.6G 2.00G - > rpool/export 7.64G 67.6G 7.49G /export > rpool/export/home 150M 67.6G 21K /export/home > rpool/export/home/jesse 150M 67.6G 150M /export/home/jesse > rpool/repo 6.56G 67.6G 6.56G /rpool/repo > rpool/swap 2.00G 69.4G 130M - > > /zones/cpmail is where it should be mounting the zone''s dataset, I believe. > > Here''s what happens when I try and start the zone: > > # zoneadm -z cpmail boot > could not verify zfs dataset mailtmp: dataset does not exist > zoneadm: zone cpmail failed to verify > > > So the zone is trying to find a dataset ''mailtmp'' and failing because it doesn''t exist. So, what happened to it? > > Here''s the zone config file, at /etc/zones/cpmail.xml (with IP address obfuscated) > > # cat /etc/zones/cpmail.xml > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1"> > <!-- > DO NOT EDIT THIS FILE. Use zonecfg(1M) instead. > --> > <zone name="cpmail" zonepath="/zones/cpmail" autoboot="false" brand="ipkg"> > <network address="xxx.xxx.xxx.xxx" physical="bge1"/> > <dataset name="mailtmp"/> > </zone> > > I just don''t understand where the dataset ''mailtmp'' went to. Perhaps it was an initial name I used for the dataset and I then renamed it to cpmail, but then I can''t see any of the zones files in /zones/cpmail : > > # find /zones/cpmail/ > /zones/cpmail/ > /zones/cpmail/dev > /zones/cpmail/root > > Does ZFS store a log file of all operations applied to it? It feels like someone has gained access and run ''zfs destroy mailtmp'' to me, but then again it could just be my own ineptitude. > > Thank you > Jesse >mailtmp had to be a separate pool and it was not a root-fs for your zone. can you first try to run ''zpool list; zpool import'' - it won''t do anything but listing imported pools and pools that could be imported. If it is still not there and you want to boot your zone wihout it anyway then make a copy of your cpmail.xml and edit the original one by deleting sataset-mailtmp line. Then your zone should boot. -- Robert Milkowski http://milek.blogspot.com
Enda O''Connor
2010-Feb-26 15:00 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
On 26/02/2010 14:03, Jesse Reynolds wrote:> Hello > > I have an amd64 server running OpenSolaris 2009-06. In December I created one container on this server named ''cpmail'' with it''s own zfs dataset and it''s been running ever since. Until earlier this evening when the server did a kernel panic and rebooted. Now, I can''t see any contents in the zfs dataset for this zone! > > The server has two disks which are root mirrored with ZFS: > > # zpool status > pool: rpool > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > rpool ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c8t0d0s0 ONLINE 0 0 0 > c8t1d0s0 ONLINE 0 0 0 > > errors: No known data errors > > Here are the datasets: > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > rpool 161G 67.6G 79.5K /rpool > rpool/ROOT 3.66G 67.6G 19K legacy > rpool/ROOT/opensolaris 3.66G 67.6G 3.51G / > rpool/cpmail 139G 67.6G 22K /zones/cpmail > rpool/cpmail/ROOT 139G 67.6G 19K legacy > rpool/cpmail/ROOT/zbe 139G 67.6G 139G legacy > rpool/dump 2.00G 67.6G 2.00G - > rpool/export 7.64G 67.6G 7.49G /export > rpool/export/home 150M 67.6G 21K /export/home > rpool/export/home/jesse 150M 67.6G 150M /export/home/jesse > rpool/repo 6.56G 67.6G 6.56G /rpool/repo > rpool/swap 2.00G 69.4G 130M - > > /zones/cpmail is where it should be mounting the zone''s dataset, I believe. > > Here''s what happens when I try and start the zone: > > # zoneadm -z cpmail boot > could not verify zfs dataset mailtmp: dataset does not exist > zoneadm: zone cpmail failed to verify > > > So the zone is trying to find a dataset ''mailtmp'' and failing because it doesn''t exist. So, what happened to it? > > Here''s the zone config file, at /etc/zones/cpmail.xml (with IP address obfuscated) > > # cat /etc/zones/cpmail.xml > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1"> > <!-- > DO NOT EDIT THIS FILE. Use zonecfg(1M) instead. > --> > <zone name="cpmail" zonepath="/zones/cpmail" autoboot="false" brand="ipkg"> > <network address="xxx.xxx.xxx.xxx" physical="bge1"/> > <dataset name="mailtmp"/> > </zone>not sure if above looks correct to me, surely this should be rpool/mailtmp, assuming you don''t have other pools it might live in. ( what does zpool import say by the way ) Did this get added to a running zone, and then fail on reboot perhaps,ie to me this never worked. Enda> > I just don''t understand where the dataset ''mailtmp'' went to. Perhaps it was an initial name I used for the dataset and I then renamed it to cpmail, but then I can''t see any of the zones files in /zones/cpmail : > > # find /zones/cpmail/ > /zones/cpmail/ > /zones/cpmail/dev > /zones/cpmail/root > > Does ZFS store a log file of all operations applied to it? It feels like someone has gained access and run ''zfs destroy mailtmp'' to me, but then again it could just be my own ineptitude. > > Thank you > Jesse
Jesse Reynolds
2010-Feb-26 15:03 UTC
[zfs-discuss] Help, my zone''s dataset has disappeared!
Ah, thanks Robert! Yes, I remember now. mailtmp was indeed a separate zfs dataset I created to use as the source for the mailbox migration I used when I built this server. It no longer exists, and is not needed. I see now, I can just remove reference to it and all should hopefully be good with the world. It''s booting!!! Woot!!! For some reason I thought the <dataset name=blah> line was necessary to tell it where to find the root filesystem for this zone, but obviously this is a way to give the zone access to ''auxiliary'' datasets. Thanks very much everyone who helped this zones on opensolaris with zfs dummy! Jesse -- This message posted from opensolaris.org