nsd users - May 2022 - NSD 4.5.0rc1 pre-release

Hi Andreas,

Thank you for trying the new version.

About the libev troubles.

I managed to run the regression test suite for NSD on libev. Two
different versions of it, but it does not produce that failure.
Everything works, apart from a signal handling test that fails, but
downloading an axfr does not involve signal handling, so that is not the
issue here.

Not sure how to proceed, since I cannot reproduce. I wonder what the
cause is.

Best regards, Wouter

On 06/05/22 23:35, A. Schulze via nsd-users wrote:
> 
> 
> Am 06.05.22 um 13:42 schrieb Wouter Wijngaards via nsd-users:
>> Hi,
>>
>> NSD 4.5.0rc1 pre-release is available
>> https://nlnetlabs.nl/downloads/nsd/nsd-4.5.0rc1.tar.gz
> 
> Hello Wouter,
> 
> this version compile without problems here and works at my small lab
> environment.
> 
>> sha256 2143268818f0f840f9fbb99a9350eaa553ee9d0b3b325851dd14a7b815b0a6e7
>> pgp https://nlnetlabs.nl/downloads/nsd/nsd-4.5.0rc1.tar.gz.gpg
> should end with .asc, not .gpg
> 
>> This release fixes a couple of minor bugs and adds IXFR out
>> functionality. With this functionality NSD can respond to IXFR queries
>> and serve IXFR transfers downstream.
>>
>> It is default disabled,
> make sense ...
> 
> 
> But I would like the moment to point to segfaults I'm seeing more for
years
> (https://lists.nlnetlabs.nl/pipermail/nsd-users/2019-June/002699.html)
> but not further discussed on this list yet.
> 
> Today I found a new data point. I use to build NSD with libev. I see
> failures.
> I would say, they are related to process termination.
> Today I rebuild NSD with libevent and the failure goes away!
> 
> To reproduce that I wrote two Dockerfile (attached)
> 
> $ docker build -t nsd:libevent -f Dockerfile.libevent .
> ...
> Successfully tagged nsd:libevent
> $ docker run --rm -ti nsd:libevent
> [2022-05-06 21:18:42.056] nsd[1]: notice: nsd starting (NSD 4.5.0)
> [2022-05-06 21:18:42.096] nsd[7]: notice: nsd started (NSD 4.5.0), pid 1
> <PRESS CTRL+C>
> [2022-05-06 21:18:43.979] nsd[7]: warning: signal received, shutting
> down...
> 
> -> this is ok
> 
> now let's build using libev
> $ docker build -t nsd:libev -f Dockerfile.libev .
> ...
> Successfully tagged nsd:libev
> $ docker run --rm -ti nsd:libev
> [2022-05-06 21:21:14.724] nsd[1]: notice: nsd starting (NSD 4.5.0)
> [2022-05-06 21:21:14.743] nsd[7]: notice: nsd started (NSD 4.5.0), pid 1
> <PRESS CTRL+C>
> [2022-05-06 21:21:32.879] nsd[7]: warning: server 8 died unexpectedly,
> restarting
> [2022-05-06 21:21:32.879] nsd[7]: warning: signal received, shutting
> down...
> [2022-05-06 21:21:32.884] nsd[9]: error: mode bad value 2, back to service.
> 
> -> I think, this should not happen...
> 
> I mentioned process termination. I see similar segfaults if NSD as slave
> receive new data via axfr.
> One server's log fill up with segfaul messages, but the data received
> via axfr are valid!
> The segfault seem to happen very close to a NSD process end.
> 
> Andreas
> -
> 
> 
> _______________________________________________
> nsd-users mailing list
> nsd-users at lists.nlnetlabs.nl
> https://lists.nlnetlabs.nl/mailman/listinfo/nsd-users
>

Wouter Wijngaards:
> I managed to run the regression test suite for NSD on libev. Two
> different versions of it, but it does not produce that failure.
> Everything works, apart from a signal handling test that fails, but
> downloading an axfr does not involve signal handling, so that is not the
> issue here.
>
> Not sure how to proceed, since I cannot reproduce. I wonder what the
> cause is.
Hello Wouter,

thanks for having a look into my trouble. I wonder about your "could  
not reproduce"
Did you build both containers with the Dockerfiles I send?

I've two servers receiving AXFF from a BIND master (no version known)
where I see the segfault messages constantly after AXFR.
I also could not reproduce any segfault on AXFR in a lab. I tried  
NSD-provider -> AXFR -> NSD-consumer
and used even the same zone. That makes the segfaults something strage  
to me, too.
They really did only happen on my prod servers.
Using the Dockerfiles was the first time I could trigger a similar error
but yes, the setup is not the same.

I now switched on server to a version built with libevent instead of libev
and the segfaults no longer happen!

In both cases the error happen very late when a process terminate.
So maybe it's related.

I plan to replace libev by libevent on /all/ of my numerous NSD instances.
That would solve this issues for me.

Could you describe the differences between libevent and libev?
What was the reason you implemented NSD in a way to use one OR the  
other library?
What may I expect when switching from libev back to libevent?
More load? Latency? Memory usage? Watever?

Andreas