freebsd stable - May 2021 - releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

On 2021-May-20, at 22:19, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> Ok, so it isn't related to "soft".
> I am wondering if it is something specific to what
> "diff -r" does?
> 
> Could you try:
> # cd /usr/ports
> # ls -R > /tmp/x
> # cd /mnt
> # ls -R > /tmp/y
> # cd /tmp
> # diff -u -p x y
> --> To see if "ls -R" finds any difference?
> 
# diff -u -p x y 
--- x   2021-05-20 22:35:48.021663000 -0700
+++ y   2021-05-20 22:39:03.691936000 -0700
@@ -227209,10 +227209,10 @@
patch-chrome_browser_background_background__mode__mana
 patch-chrome_browser_background_background__mode__optimizer.cc
 patch-chrome_browser_browser__resources.grd
 patch-chrome_browser_browsing__data_chrome__browsing__data__remover__delegate.cc
+patch-chrome_browser_chrome__browser
 patch-chrome_browser_chrome__browser__interface__binders.cc
 patch-chrome_browser_chrome__browser__main.cc
 patch-chrome_browser_chrome__browser__main__linux.cc
-patch-chrome_browser_chrome__browser__main__posix.cc
 patch-chrome_browser_chrome__content__browser__client.cc
 patch-chrome_browser_chrome__content__browser__client.h
 patch-chrome_browser_crash__upload__list_crash__upload__list.cc

# find /usr/ports/ -name 'patch-chrome_browser_chrome__browser*' -print
| more
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__posix.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc

 find /mnt/ -name 'patch-chrome_browser_chrome__browser*' -print | more
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc

So: patch-chrome_browser_chrome__browser appears to be a
truncated: patch-chrome_browser_chrome__browser__main__posix.cc
file name and find also gets the same oddity.

(Note: This had /usr/ports in a main context and /mnt/
referring to a release/13.0.0 context.)
> ps: I do not think that r367492 could cause this, but it would be
>     nice if you try a kernel with the r367492 patch reverted.
>     It is currently in all of releng13, stable13 and main, although
>     the patch to fix this is was just reviewed and may hit main soon.
Do you want a debug kernel to be used? Do you have a preference
for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
stick to the base version things are now based on --or do you
want me to update to more recent? (That last only applies if
main or stable/13 is to be put to use.)
> . . . old history deleted . . .
==Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

[Looks like the RPi4B genet0 handling is involved.]

On 2021-May-20, at 22:56, Mark Millard <marklmi at yahoo.com>
wrote:
> 
> On 2021-May-20, at 22:19, Rick Macklem <rmacklem at uoguelph.ca>
wrote:
> 
>> Ok, so it isn't related to "soft".
>> I am wondering if it is something specific to what
>> "diff -r" does?
>> 
>> Could you try:
>> # cd /usr/ports
>> # ls -R > /tmp/x
>> # cd /mnt
>> # ls -R > /tmp/y
>> # cd /tmp
>> # diff -u -p x y
>> --> To see if "ls -R" finds any difference?
>> 
> 
> # diff -u -p x y 
> --- x   2021-05-20 22:35:48.021663000 -0700
> +++ y   2021-05-20 22:39:03.691936000 -0700
> @@ -227209,10 +227209,10 @@
patch-chrome_browser_background_background__mode__mana
> patch-chrome_browser_background_background__mode__optimizer.cc
> patch-chrome_browser_browser__resources.grd
>
patch-chrome_browser_browsing__data_chrome__browsing__data__remover__delegate.cc
> +patch-chrome_browser_chrome__browser
> patch-chrome_browser_chrome__browser__interface__binders.cc
> patch-chrome_browser_chrome__browser__main.cc
> patch-chrome_browser_chrome__browser__main__linux.cc
> -patch-chrome_browser_chrome__browser__main__posix.cc
> patch-chrome_browser_chrome__content__browser__client.cc
> patch-chrome_browser_chrome__content__browser__client.h
> patch-chrome_browser_crash__upload__list_crash__upload__list.cc
> 
> # find /usr/ports/ -name 'patch-chrome_browser_chrome__browser*'
-print | more
>
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
>
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
>
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
>
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
>
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__posix.cc
> /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
>
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
>
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> 
> find /mnt/ -name 'patch-chrome_browser_chrome__browser*' -print |
more
>
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
> /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
>
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
>
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
>
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
>
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> 
> So: patch-chrome_browser_chrome__browser appears to be a
> truncated: patch-chrome_browser_chrome__browser__main__posix.cc
> file name and find also gets the same oddity.
> 
> (Note: This had /usr/ports in a main context and /mnt/
> referring to a release/13.0.0 context.)
> 
>> ps: I do not think that r367492 could cause this, but it would be
>>    nice if you try a kernel with the r367492 patch reverted.
>>    It is currently in all of releng13, stable13 and main, although
>>    the patch to fix this is was just reviewed and may hit main soon.
> 
> Do you want a debug kernel to be used? Do you have a preference
> for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
> stick to the base version things are now based on --or do you
> want me to update to more recent? (That last only applies if
> main or stable/13 is to be put to use.)
> 
>> . . . old history deleted . . .
I reversed the roles of the faster vs. somewhat slower
machine and so far my diff -r attempts for this found
no differences. The machines were using different types
of EtherNet devices.

So I've substituted a different EtherNet device onto
the slower machine: the same type of USB3 EtherNet
device in use on the faster machine (instead of
using the RPi4B's builtin EtherNet). So the below
testing is with both machines having a:

ugen0.6: <Realtek USB 10/100/1000 LAN> at usbus0
ure0 on uhub0
ure0: <Realtek USB 10/100/1000 LAN, class 0/0, rev 3.00/30.00, addr 5> on
usbus0
miibus1: <MII bus> on ure0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus1
rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX,
1000baseT-FDX-master, auto

in use.

I rebooted with this connected instead of the genet0
interface.

Mounting the slower machine's /usr/ports/ as /mnt/ from the faster machine:
No differences found by diff -r this way (expected result).

Mounting the faster machine's /usr/ports/ as /mnt/ from the slower machine:
No differences found by diff -r this way (expected result).

Doing diff -r's from both sides at the same time:
No differences found by diff -r this way (expected result).


So it looks like genet0 or its supporting software
is contributing to the problems that I had reported.

It is interesting that there were no examples of the
content of files reporting a mismatch, just some file
names/paths not finding matches, some with truncated
names or obvious-garbage bytes in names.


Note: The faster machine is a MACCCHIATObin Double
Shot. The slower machine is a RPi4B 8 GiByte.

==Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard wrote:
>On 2021-May-20, at 22:19, Rick Macklem <rmacklem at uoguelph.ca>
wrote:
[stuff snipped]
>> ps: I do not think that r367492 could cause this, but it would be
>>     nice if you try a kernel with the r367492 patch reverted.
>>     It is currently in all of releng13, stable13 and main, although
>>     the patch to fix this is was just reviewed and may hit main soon.
>
>Do you want a debug kernel to be used? Do you have a preference
>for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
>stick to the base version things are now based on --or do you
>want me to update to more recent? (That last only applies if
>main or stable/13 is to be put to use.)
Well, it sounds like you've isolated it to the genet interface.
Good sluething.
Unfortunately, NFS is only as good as the network fabric under it.
However, it's usually hangs or poor performance. Except maybe
for the readdir issue that Jason Bacon reported and resolved via
an upgrade, this is a first.
--> In the old days, I would have expected IP checksums to catch
       this, but I'm guessing the hardware/net driver are doing them
       these days?

W.r.t. reverting r367492...the patch to replace r367492 was just
committed to "main" by rscheff@ with a two week MFC, so it
should be in stable/13 soon. Not sure if an errata can be done
for it for releng13.0?

Thanks for isolating this, rick
ps: Co-incidentally, I've been thinking of buying an RBPi4 as a
toy.
> . . . old history deleted . . .
==Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)