freebsd stable - Aug 2017 - nullfs making contents of ZFS datasets invisible even after unmounting

I am running a FreeBSD 11.1 system with ZFS and jails and I mount a
file system on my storage pool (/storage/cloud) into the jail
(/jails/cloud/storage) via nullfs, and that works fine for what the
jail does. However I just noticed that outside the jail, I can only
see the mount points of the file system children, but not the
contents.

For example if I do: ls -la /storage/cloud/* (As root) then nothing
shows up, but they are present inside the jail. Even if I stop the
jail, they still are not present outside the jail.

I just tried disabling jails in /etc/rc.conf and rebooting the system
and EVEN THEN the files do not reappear outside the jail in their
original location, yet zfs list reports the space being used. But if I
then (without rebooting) enable the jail, the files show up in the
nullfs-mounted location inside the jail!

As an experiment, I tried creating /mnt/test and /mnt/test2 and
creates a folder and a few files in test, and then nullfs-mounted it
onto test2, and I could then see the files in both locations.

Does anybody know what might be causing this? The fact that my data
stays hidden after a reboot is rather worrisome.

With a little bit of help, I managed to figure out what the problem
was. It turned out that I had run, head first, into the wall of nullfs
not operating across file systems. Thus inside the jail, the data was
being saved to a regular directory on my "storage/cloud"-dataset
(Mountpoint: /storage/cloud), whereas outside the jail the child file
system "storage/cloud/bc", having been left empty, was mounted over
the folder "bc" on "storage/cloud", where the files actually
resided,
thus "hiding" them from view outside the jail. A simple "zfs
unmount
storage/cloud/bc" revealed them.

Anybody reading this in the future: If you are using ZFS and want to
put a file system and all of its children into a jail then you cannot
use nullfs. Instead you need to either change the actual mount point
using "zfs set mountpoint=/jails/jail1/storage
storage/path/to/filesystem" or to attach the entire dataset to the
jail, and allow it to be managed from within the jail using "zfs
jail".

2017-08-14 10:34 GMT+02:00  <tsuroerusu at
gmail.com>:
> I am running a FreeBSD 11.1 system with ZFS and jails and I mount a
> file system on my storage pool (/storage/cloud) into the jail
> (/jails/cloud/storage) via nullfs, and that works fine for what the
> jail does. However I just noticed that outside the jail, I can only
> see the mount points of the file system children, but not the
> contents.
>
> For example if I do: ls -la /storage/cloud/* (As root) then nothing
> shows up, but they are present inside the jail. Even if I stop the
> jail, they still are not present outside the jail.
>
> I just tried disabling jails in /etc/rc.conf and rebooting the system
> and EVEN THEN the files do not reappear outside the jail in their
> original location, yet zfs list reports the space being used. But if I
> then (without rebooting) enable the jail, the files show up in the
> nullfs-mounted location inside the jail!
>
> As an experiment, I tried creating /mnt/test and /mnt/test2 and
> creates a folder and a few files in test, and then nullfs-mounted it
> onto test2, and I could then see the files in both locations.
>
> Does anybody know what might be causing this? The fact that my data
> stays hidden after a reboot is rather worrisome.