CentOS - Oct 2017 - how to prevent files and directories from being deleted?

On Tuesday 03 October 2017 18:24:01 Mark Haney wrote:
> What issue? That the PID is dropped on reboot?? What else are you
> putting in there?? I'm beginning to question whether you know what
> you're doing or not.? Lighttpd doesn't store any persistent info in
> /var/run/ because, like everything else, /var/run isn't for persistent
> data.
Mark, Many Non-Centos originated packages create directories in /var/run as 
part of the install, and expect them to still exist after a reboot.

They then fail when starting the service because they're trying to create a 
PID / Lock file in a directory that no longer exists.  This problem has been 
around ever since /var/run was moved to tmpfs.

Unfortunately, sometimes we have to use packages other than the official 
Centos ones, usually as in this case because we need newer versions.

There is a solution that saves /var/run to disk at shutdown and restores it at 
bootup but I can't remember what it is.

On 04/10/2017 10:23, Gary Stainburn wrote:

Hi Gary,
> Mark, Many Non-Centos originated packages create directories in /var/run as
> part of the install, and expect them to still exist after a reboot.
Those packages have been built poorly.
> They then fail when starting the service because they're trying to
create a
> PID / Lock file in a directory that no longer exists.  This problem has
been
> around ever since /var/run was moved to tmpfs.
Yes, and those packages should know how to work with CentOS 7.
> Unfortunately, sometimes we have to use packages other than the official 
> Centos ones, usually as in this case because we need newer versions.
Sure, that can be.
> There is a solution that saves /var/run to disk at shutdown and restores it
at
> bootup but I can't remember what it is.
There's no need to do that (and it's also messy). Instead, if a package
needs a directory to exist in /var/run, then create your own config for
systemd-tmpfiles, and drop it into /etc/systemd/tmpfiles.d. Work with
CentOS 7, instead of fighting with it.

Anand

On Wednesday 04 October 2017 09:42:13 Anand Buddhdev
wrote:
> There's no need to do that (and it's also messy). Instead, if a
package
> needs a directory to exist in /var/run, then create your own config for
> systemd-tmpfiles, and drop it into /etc/systemd/tmpfiles.d. Work with
> CentOS 7, instead of fighting with it.
>
> Anand
I saw reference to system-tmpfs in Paul's post so I had a quick look. YUM 
doesn't seem to know about it, but I'm sure Google will help.

On 10/04/2017 04:23 AM, Gary Stainburn wrote:
>
> Mark, Many Non-Centos originated packages create directories in /var/run as
> part of the install, and expect them to still exist after a reboot.
>
> They then fail when starting the service because they're trying to
create a
> PID / Lock file in a directory that no longer exists.  This problem has
been
> around ever since /var/run was moved to tmpfs.
>
> Unfortunately, sometimes we have to use packages other than the official
> Centos ones, usually as in this case because we need newer versions.
>
> There is a solution that saves /var/run to disk at shutdown and restores it
at
> bootup but I can't remember what it is.
Sorry, but if you have to use packages that don't originate from CentOS 
and they do that, then I wouldn't use them. Period.? I'd compile from 
source before I used something configured that way.

Why is it so hard for people to understand that var/run IS NOT 
PERSISTENT and was never meant to be?? Do they not teach basic Unix 
concepts anymore?? If you think that setup is acceptable, I wouldn't 
hire you to water my lawn as you'd likely water the electrical box along 
with said lawn.

These are VERY VERY basic concepts.? Banging a square peg into a round 
hole, even in a test environment is a good way to get fired and become 
unemployable.? And believe me, word gets around quickly in IT circles.? 
If you can't build from source to keep from using non-standard packages, 
then you really shouldn't be doing whatever it is you were hired to do.

This is extremely basic arithmetic here.? You don't do surgery with 
dirty scalpels, you don't drive without brakes, these are axiomatic just 
like /var/run isn't persistent.? It's been that way at least since I was
in HS and college in the 80s and very very likely since the early Unix 
days.

Honestly, I feel bad for your employer if you think this is an 
acceptable way to get a system working.

There, I've said my piece. Call it a flame if you want, truth hurts and 
ignoring basic rules is a good way to hurt yourself or other people.


-- 
Mark Haney
Network Engineer at NeoNova
919-460-3330 option 1
mark.haney at neonova.net
www.neonova.net

On Wednesday 04 October 2017 12:54:44 Mark Haney wrote:
> Sorry, but if you have to use packages that don't originate from CentOS
> and they do that, then I wouldn't use them. Period.? I'd compile
from
> source before I used something configured that way.
This perspective to some extent employs cutting your nose of dispite youre 
face.  Before Packages were introduced, everyone compiled from source. That 
was a pain, and a long process, especially when you had dependancies that you 
also had to compile.  Packages eased this process but kept the dependancy 
issue.

Package managers got round (mostly) both the dependancy problem and updating 
too. The problem with package maintainers not keeping up to date shows that 
this still isn't perfect.

However, if you go back to compiling from source then you lose all of these 
benefits.
>
> Why is it so hard for people to understand that var/run IS NOT
> PERSISTENT and was never meant to be?? Do they not teach basic Unix
> concepts anymore?? If you think that setup is acceptable, I wouldn't
> hire you to water my lawn as you'd likely water the electrical box
along
> with said lawn.
Thankfully I do not earn my keep by watering lawns.  I do not believe that 
this is acceptable, but by the same token I have to earn my keep and that 
involves having working production servers and services.

I have managed to get round this problem in the past through manually doing 
the same function as systemd-tmpfiles. It is a small price to pay to have a 
working, (relatively) up to date server.

On 10/04/2017 01:23 AM, Gary Stainburn wrote:
> There is a solution that saves /var/run to disk at shutdown and restores it
at
> bootup but I can't remember what it is.

The simplest solution that comes to mind at 7:30am is simply bind 
mounting a directory that's persistent.? You'll still need to define 
that directory using tmpfiles.d, but...

    echo "D /var/run/mariadb 0755 root root -" >
    /etc/tmpfiles.d/mariadb.conf
    mkdir /var/run-persistent/mariadb -p
    echo "/var/run-persistent/mariadb /var/run/mariadb none bind 0 0"
>>
    /etc/fstab

Obviously, test this somewhere before you do it to a system you care 
about.? It's early, and I haven't.? I'm not sure startup ordering
will
be correct for this sort of thing...

On 10/04/2017 04:54 AM, Mark Haney wrote:
> Why is it so hard for people to understand that var/run IS NOT 
> PERSISTENT and was never meant to be?? Do they not teach basic Unix 
> concepts anymore?

http://www.pathname.com/fhs/pub/fhs-2.3.html#VARRUNRUNTIMEVARIABLEDATA

While FHS notes that *files* should be cleared during the boot process, 
it does not indicate that directories should be, and that is the source 
of this problem.? Packages on recent Fedora, RHEL, or CentOS releases 
should use tmpfiles.d to define directory structure within /var/run, but 
that is a unique and recent development.

Please chill out.? There is no need to berate users, here.

Gary Stainburn <gary at ringways.co.uk> writes:
> On Tuesday 03 October 2017 18:24:01 Mark Haney wrote:
>> What issue? That the PID is dropped on reboot?? What else are you
>> putting in there?? I'm beginning to question whether you know what
>> you're doing or not.? Lighttpd doesn't store any persistent
info in
>> /var/run/ because, like everything else, /var/run isn't for
persistent
>> data.
>
> Mark, Many Non-Centos originated packages create directories in /var/run as
> part of the install, and expect them to still exist after a reboot.
I would also expect that.
> They then fail when starting the service because they're trying to
create a
> PID / Lock file in a directory that no longer exists.  This problem has
been
> around ever since /var/run was moved to tmpfs.
A ramdisk?  That?s an incredibly stupid idea.  Ramdisks should not be
used by default.

Is there a way to change that?
> Unfortunately, sometimes we have to use packages other than the official 
> Centos ones, usually as in this case because we need newer versions.
>
> There is a solution that saves /var/run to disk at shutdown and restores it
at
> bootup but I can't remember what it is.
That doesn?t protect you in case of a power failure and may increase
shutdown durations to a point where the amount of time it takes to
shutdown isn?t fully covered by the UPS.


-- 
"Didn't work" is an error.

Mark Haney <mark.haney at neonova.net> writes:
> On 10/04/2017 04:23 AM, Gary Stainburn wrote:
>>
>> Mark, Many Non-Centos originated packages create directories in
/var/run as
>> part of the install, and expect them to still exist after a reboot.
>>
>> They then fail when starting the service because they're trying to
create a
>> PID / Lock file in a directory that no longer exists.  This problem has
been
>> around ever since /var/run was moved to tmpfs.
>>
>> Unfortunately, sometimes we have to use packages other than the
official
>> Centos ones, usually as in this case because we need newer versions.
>>
>> There is a solution that saves /var/run to disk at shutdown and
restores it at
>> bootup but I can't remember what it is.
> Sorry, but if you have to use packages that don't originate from
> CentOS and they do that, then I wouldn't use them. Period.? I'd
> compile from source before I used something configured that way.
>
> Why is it so hard for people to understand that var/run IS NOT
> PERSISTENT and was never meant to be?
This isn?t true, the directory is persistent, and the FHS says:

"Files under this directory must be cleared (removed or truncated as
appropriate) at the beginning of the boot process."[1]

[2] doesn?t tell you that files in /var/run will disappear at shutdown.

Using a ramdisk to store such files is not compliant with the FHS
because the files are neither truncated, nor removed; they are being
disappeared, and not at the beginning of the boot process but at
shutdown.  Using a ramdisk is not appropriate.

The safe and compliant way would be to truncate the files and not to
remove or to disappear them.

The FHS doesn?t say /which/ files should be cleared or removed.  I would
say that all files the removal or truncation of is not explicitly
specified must neither be removed, nor truncated, and that automatically
removing files is generally questionable and needs to be done, if at
all, with great care.

I can only speculate (and hope) that the intention of the FHS here is
that programs creating files under /var/run are supposed to remove or
truncate the files they created during the previous runtime, and only
those, when the system boots, which would be the time when those
programs are being started.


[1]:
http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#VARRUNRUNTIMEVARIABLEDATA

[2]:
https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-filesystem-fhs.html

> Do they not teach basic Unix concepts anymore?? If you think that
> setup is acceptable, I wouldn't hire you to water my lawn as you'd
> likely water the electrical box along with said lawn.
>
> These are VERY VERY basic concepts.? Banging a square peg into a round
> hole, even in a test environment is a good way to get fired and become
> unemployable.? And believe me, word gets around quickly in IT
> circles.? If you can't build from source to keep from using
> non-standard packages, then you really shouldn't be doing whatever it
> is you were hired to do.
>
> This is extremely basic arithmetic here.? You don't do surgery with
> dirty scalpels, you don't drive without brakes, these are axiomatic
> just like /var/run isn't persistent.? It's been that way at least
> since I was in HS and college in the 80s and very very likely since
> the early Unix days.
Then how come that the first time I?m seeing an issue like this is only
after someone made the utterly stupid decision to use a ramdisk for
/var/run?

It is a change that has been made at some time, and we weren?t told
about it.  Assuming that people not being informed about a change are
stupid because they don?t know about it is a stupid thing to do.
> Honestly, I feel bad for your employer if you think this is an
> acceptable way to get a system working.
>
> There, I've said my piece. Call it a flame if you want, truth hurts
> and ignoring basic rules is a good way to hurt yourself or other
> people.
Making things worse by providing dirty scalpels or vehicles without
brakes --- with or without telling those who are going to use them ---
doesn?t make things better, and it can be argued that someone providing
those should be fired because of their stupidity.

Alas, the only thing that helps against stupidity is more stupidity.
Getting upset about it does not.


-- 
"Didn't work" is an error.