zfs discuss - Apr 2010 - how can I remove files when the fiile system is full?

During the IPS upgrade, the file system got full, then I cannot do anything to
recover it.

# df -kl
Filesystem           1K-blocks      Used Available Use% Mounted on
rpool/ROOT/opensolaris
                       4976642   4976642         0 100% /
swap                  14217564       244  14217320   1% /etc/svc/volatile
/usr/lib/libc/libc_hwcap2.so.1
                       4976642   4976642         0 100% /lib/libc.so.1
swap                  14217460       140  14217320   1% /tmp
swap                  14217344        24  14217320   1% /var/run

# cd /var/adm
# rm messages.?
rm: cannot remove `messages.0'': No space left on device
rm: cannot remove `messages.1'': No space left on device

Likely a similar issue was reported a few years ago like:
http://opensolaris.org/jive/thread.jspa?messageID=241580&#241580

However, my system is on snv_133.

Is there any way to work around the situation?
This is really critical since after the IPS gets the file system full, customers
seem not able to recover.

Thanks,

-Eiji

On 04/ 1/10 01:46 PM, Eiji Ota wrote:
> During the IPS upgrade, the file system got full, then I cannot do anything
to recover it.
>
> # df -kl
> Filesystem           1K-blocks      Used Available Use% Mounted on
> rpool/ROOT/opensolaris
>                         4976642   4976642         0 100% /
> swap                  14217564       244  14217320   1% /etc/svc/volatile
> /usr/lib/libc/libc_hwcap2.so.1
>                         4976642   4976642         0 100% /lib/libc.so.1
> swap                  14217460       140  14217320   1% /tmp
> swap                  14217344        24  14217320   1% /var/run
>
> # cd /var/adm
> # rm messages.?
> rm: cannot remove `messages.0'': No space left on device
> rm: cannot remove `messages.1'': No space left on device
>
> Likely a similar issue was reported a few years ago like:
> http://opensolaris.org/jive/thread.jspa?messageID=241580&#241580
>
> However, my system is on snv_133.
>
> Is there any way to work around the situation?
> This is really critical since after the IPS gets the file system full,
customers seem not able to recover.
>
> Thanks,
>
> -Eiji
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On opensolaris?  Did you try deleting any old BEs?

-tim

On Thu, Apr 1, 2010 at 12:46 PM, Eiji Ota <eiji.ota at oracle.com> wrote:
> # cd /var/adm
> # rm messages.?
> rm: cannot remove `messages.0'': No space left on device
> rm: cannot remove `messages.1'': No space left on device
>
I think doing "cat /dev/null > /var/adm/messages.1" will work.

-B

-- 
Brandon High : bhigh at freaks.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/93dd469c/attachment.html>

Thanks. It worked, but yet the fs says it''s full. Is it normal and I
can get some space eventually (if I continue this)?

# cat /dev/null >./messages.1 
# cat /dev/null >./messages.0 

# df -kl 
Filesystem 1K-blocks Used Available Use% Mounted on 
rpool/ROOT/opensolaris 
4976123 4976123 0 100% / <============== the availabe space is yet 0. 
swap 14218704 244 14218460 1% /etc/svc/volatile 
/usr/lib/libc/libc_hwcap2.so.1 
4976123 4976123 0 100% /lib/libc.so.1 
swap 14218600 140 14218460 1% /tmp 
swap 14218472 12 14218460 1% /var/run 

-Eiji 
> 
> On Thu, Apr 1, 2010 at 12:46 PM, Eiji Ota < eiji.ota at oracle.com >
wrote:
> 
# cd /var/adm 
> # rm messages.? 
> rm: cannot remove `messages.0'': No space left on device 
> rm: cannot remove `messages.1'': No space left on device 
> 
> 
I think doing "cat /dev/null > /var/adm/messages.1" will work. 
> 
-B 
> -- 
> Brandon High : bhigh at freaks.com 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/b3f94260/attachment.html>

On Thu, Apr 1, 2010 at 1:39 PM, Eiji Ota <eiji.ota at oracle.com> wrote:
> Thanks. It worked, but yet the fs says it''s full. Is it normal and
I can
> get some space eventually (if I continue this)?
>
You may need to destroy some snapshots before the space becomes available.

zfs list -t snapshot will how approximately how much space will be freed for
each snapshot.

-B

-- 
Brandon High : bhigh at freaks.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/5b9dd059/attachment.html>

Thanks, Brandon. 

Now that the issue goes away, I could recover my host. 

-Eiji 
> 
> On Thu, Apr 1, 2010 at 1:39 PM, Eiji Ota < eiji.ota at oracle.com >
wrote:
> 

> Thanks. It worked, but yet the fs says it''s full. Is it normal and
I can get some space eventually (if I continue this)?
> 
> 
You may need to destroy some snapshots before the space becomes available. 
> 
zfs list -t snapshot will how approximately how much space will be freed for
each snapshot.
> 
-B 
> -- 
> Brandon High : bhigh at freaks.com 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/feda971a/attachment.html>

> On opensolaris?  Did you try deleting any old BEs?
Don''t forget to "zfs destroy rpool at snapshot"
In fact, you might start with destroying snapshots ... if there are any
occupying space.