From: SilverTip257 <silvertip257 at gmail.com>
> To: CentOS mailing list <centos at centos.org>
> Cc:
> Sent: Friday, March 21, 2014 5:40 PM
> Subject: Re: [CentOS] rsync triggers oomkiller
>
> I added a subject so we can track this message on the list easier. ;)
>
> On Fri, Mar 21, 2014 at 12:19 PM, John Doe <jdmls at yahoo.com>
wrote:
>
>> Hey,
>>
>> ? kernel: rsync invoked oom-killer: gfp_mask=0x200da, order=0,
oom_adj=0,
>> oom_score_adj=0
>> ? ...
>> ? kernel: Out of memory: Kill process 27974 (mysqld) score 361 or
>> sacrifice child
>> ? kernel: Killed process 27974, UID 27, (mysqld) total-vm:3804672kB,
>> anon-rss:2890828kB, file-rss:3324kB
>>
>> rsync whines he wants more RAM and... mysql gets killed...
>> That makes me a bit sad!
>> And from my nagios graphs, at that time, ram usage was only 75%
>> (4.5GB/6GB) and swap usage 0.3% (/2GB)!
>> I cannot believe an rsync suddenly needed 1.5GB+2GB and was not
>> satisfied...
>> And even after the out of memory, the graphs show no swap usage change
at
>> all.
>> Anybody would have an idea about this "fake" out of memory?
>>
>
> Wild.
> I've not encountered oomkiller being triggered when the server has free
> memory and hasn't swapped to disk yet.
>
> After the fact, it will probably be almost impossible to figure out the
> list of files that rsync was storing in memory.
>
> This makes me wonder if there's an option to have rsync log the list of
> files (to be synced) to a log file of some sort.? I'll have to look
into it
> later today when I have a moment.
Thx for the subject!
After more investigation, I found:
- a vm.swappiness=0 in sysctl.conf, which should not prevent the kernel to swap
to prevent an oom.
- the rsync was part of 8 *sequential* rsyncs on 8 servers, rsyncing between 500
and 1000 files at most...
Thx,
JD