CentOS - Sep 2023 - conclusion of: old website keep reappering in /var/www/html

Greetings,
absurd subject, but I don't know how else I could describe what is
happening.
I have a small static website hosted on a Centos 7 VPS (that I'll update as
soon as possible, but what follows is much more urgent, I must absolutely fix it
asap).
The website is generated by Hugo. I run hugo on my computer, then rsync
everything on the server, in the /var/www/html/hugo/MYWEBSITE folder.
Everything worked great for years.
This week I rebuilt the whole website almost from scratch, using a different
hugo theme and updating many pages.
Then I ran rsync, checked the new website in my browser, jumping from page to
page, and everything worked as expected: new theme, new content, new menus,
everything.
2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran
rsync at all. For example, in the new website there is a new subfolder called
"bio", that did not exist in the old version.

If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE  and run "ls
-l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do
"ls -l" 3 minutes later, the "bio" subfolder is no more.

I have repeated the rsync process 5/6 times before writing this email, and
it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes,
then the WHOLE folder on the server is erased and refilled with the previous
full version of the website.

It's as if there were some hidden cron job somewhere that runs hugo on the
OLD source files, but I can't find it. All the standard methods one can find
by googling "how to list all cron jobs" don't show anything that
may be the reason.
THANKS in advance for any help,
Marco

On 9/7/2023 11:19 AM, Marco Fioretti wrote:
> It's as if there were some hidden cron job somewhere that runs hugo on 
> the OLD source files, but I can't find it. All the standard methods 
> one can find by googling "how to list all cron jobs" don't
show
> anything that may be the reason.
Also check for systemd timer units in all the systemd unit directories.

Check the system log files when this happens ("ls -lt /var/log |
head")
to see if something's running at that moment.

Run top as it happens to see what jumps to the top of the process list.

Use chattr to mark the files immutable and see what complains when it 
fails to change the content.

From what your describing, a few things to check, some may be overly 
obvious so I apologize if you have checked these:

1.? Did you clear your browser cache?
2.? Is there a proxy server that has an old version of the site cached 
on it, and it's serving up the old site content instead of a refresh?
3.? Are there multiple web servers hosting the content in a cluster and 
you only rsync'd to one of them, and requests for connections is doing a 
round robin approach?
4.? Is there a backup mechanism or security appliance that is checking 
for the presence of pages on the site, doing a checksum or a diff or 
something, and if things have changed, then it's putting everything back 
to it's secured copy, perhaps an antimalware security tool or auto 
backup tool or some other tool reaching into the server and doing this 
check.

Check the host names, IP addresses, mac addresses when you see the 
inconsistent data, and check for your bio folder w/ it present and w/o 
it present see if anything changes.

Set up a new CentOS VPS w/ a LAMP stack with the same base packages 
installed as the existing VM (don't clone the existing one), then put 
the new site there and see if it's happening there.? If it isn't then 
some kind of security mechanism is checking the data and restoring a backup.

Just my 2 cents.

Chris


On 9/7/2023 1:19 PM, Marco Fioretti wrote:
> Greetings,
> absurd subject, but I don't know how else I could describe what is
happening.
> I have a small static website hosted on a Centos 7 VPS (that I'll
update as soon as possible, but what follows is much more urgent, I must
absolutely fix it asap).
> The website is generated by Hugo. I run hugo on my computer, then rsync
everything on the server, in the /var/www/html/hugo/MYWEBSITE folder.
> Everything worked great for years.
> This week I rebuilt the whole website almost from scratch, using a
different hugo theme and updating many pages.
> Then I ran rsync, checked the new website in my browser, jumping from page
to page, and everything worked as expected: new theme, new content, new menus,
everything.
> 2/3 minutes later, the OLD website reappeared, all of it, as if I had not
ran rsync at all. For example, in the new website there is a new subfolder
called "bio", that did not exist in the old version.
>
> If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE  and run
"ls -l" RIGHT AFTER running rsync, I see the "bio"
subfolder. If I do "ls -l" 3 minutes later, the "bio"
subfolder is no more.
>
> I have repeated the rsync process 5/6 times before writing this email, and
it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes,
then the WHOLE folder on the server is erased and refilled with the previous
full version of the website.
>
> It's as if there were some hidden cron job somewhere that runs hugo on
the OLD source files, but I can't find it. All the standard methods one can
find by googling "how to list all cron jobs" don't show anything
that may be the reason.
> THANKS in advance for any help,
> Marco
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
-- 
Christopher Wensink
IS Administrator
Five Star Plastics, Inc
1339 Continental Drive
Eau Claire, WI 54701
Office:  715-831-1682
Mobile:  715-563-3112
Fax:  715-831-6075
cwensink at five-star-plastics.com
www.five-star-plastics.com

At Thu, 07 Sep 2023 18:19:33 +0000 CentOS mailing list <centos at
centos.org> wrote:
> 
> Greetings,
> absurd subject, but I don't know how else I could describe what is
happening.
> I have a small static website hosted on a Centos 7 VPS (that I'll
update as soon as possible, but what follows is much more urgent, I must
absolutely fix it asap).
> The website is generated by Hugo. I run hugo on my computer, then rsync
everything on the server, in the /var/www/html/hugo/MYWEBSITE folder.
> Everything worked great for years.
> This week I rebuilt the whole website almost from scratch, using a
different hugo theme and updating many pages.
> Then I ran rsync, checked the new website in my browser, jumping from page
to page, and everything worked as expected: new theme, new content, new menus,
everything.
> 2/3 minutes later, the OLD website reappeared, all of it, as if I had not
ran rsync at all. For example, in the new website there is a new subfolder
called "bio", that did not exist in the old version.
> 
> If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE  and run
"ls -l" RIGHT AFTER running rsync, I see the "bio"
subfolder. If I do "ls -l" 3 minutes later, the "bio"
subfolder is no more.
> 
> I have repeated the rsync process 5/6 times before writing this email, and
it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes,
then the WHOLE folder on the server is erased and refilled with the previous
full version of the website.
Are you running cPanel?
> 
> It's as if there were some hidden cron job somewhere that runs hugo on
the OLD source files, but I can't find it. All the standard methods one can
find by googling "how to list all cron jobs" don't show anything
that may be the reason.
> THANKS in advance for any help,
> Marco
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
>                                   
> 
-- 
Robert Heller             -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

Hello all, and my apologies for the late answer, but as I "fixed" this
I had to patch other problems until this week...

TL;DR : I did fix the problem by "brute force", but don't really
know what
happened. See bottom of email for more details.

On Thu, Sep 07, 2023 18:19:33 PM +0000, Marco Fioretti
wrote:
> Greetings,
> absurd subject, but I don't know how else I could describe what is
happening.
> I have a small static website hosted on a Centos 7 VPS (that I'll
update as soon as possible, but what follows is much more urgent, I must
absolutely fix it asap).
> The website is generated by Hugo. I run hugo on my computer, then rsync
everything on the server, in the /var/www/html/hugo/MYWEBSITE folder.
> Everything worked great for years.
> This week I rebuilt the whole website almost from scratch, using a
different hugo theme and updating many pages.
> Then I ran rsync, checked the new website in my browser, jumping from page
to page, and everything worked as expected: new theme, new content, new menus,
everything.
> 2/3 minutes later, the OLD website reappeared, all of it, as if I had not
ran rsync at all. For example, in the new website there is a new subfolder
called "bio", that did not exist in the old version.
> 
> If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE  and run
"ls -l" RIGHT AFTER running rsync, I see the "bio"
subfolder. If I do "ls -l" 3 minutes later, the "bio"
subfolder is no more.
> 
> I have repeated the rsync process 5/6 times before writing this email, and
it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes,
then the WHOLE folder on the server is erased and refilled with the previous
full version of the website.
> 
> It's as if there were some hidden cron job somewhere that runs hugo on
the OLD source files, but I can't find it. All the standard methods one can
find by googling "how to list all cron jobs" don't show anything
that may be the reason.
> THANKS in advance for any help,
> Marco
> _______________________________________________
In synthesis:


- whatever the problem was, it was only on the server, no browser
stuff. I had said since the beginning that I could see the HTML
folders reappear by doing "ls -l" on the server.

- the server has no cpanel, no proxies, no security appliance etc,
  nothing of the sort. It is a VPS with the bare minimum, administered
  exclusively via SSH and cron jobs that never dido anything to
  **generate** web pages

- I did look inside /var/log, top, systemd timer units in the systemd
  unit directories, but could not recognize anything connected to this
  problem

How I "solved" the problem:

I do have a shell script on the server that I used to run hugo with
the right parameters to regenerate the website, but I NEVER included
it in any cron job. I only executed it manually when needed, because
the site was seldom updated. Well, when I changed name to that script
and made it NOT executable, the problem disappeared.

So, I probably I did miss something in the checks above, but no idea
what...

Marco