Johnny Hughes
2016-Feb-17 14:39 UTC
[CentOS] New glibc for CentOS-6 and CentOS-7 and CVE-2015-7547
On 02/17/2016 08:10 AM, Michael H wrote:>> The easy answer is yes .. glibc requires so many things to be restarted, >> that is the best bet. Or certainly the easiest. >> >> Note: in CentOS 7, there is also a kernel update which is rated as >> Important .. so you should boot to that anyway: >> https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html >> >> Here is a good link to figure out what to restart if you don't want to >> reboot: >> >> https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-an-upgrade/ >> >> and there is this thread: >> http://markmail.org/message/dodinyrhwgey35mh >> >> But generalyl, after a glibc update or a kernel update .. rebooting is >> easiest and it ensures everything is protected. > > Wow, so, I updated my server (yum update -y) which applied a new kernel > and the new glibc among other things, After the update completed it > knocked my master postgresql database offline. > > > Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... > Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter > "max_stack_depth": 16384 > Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed > 7680kB. > Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth > limit via "ulimit -s" or local equivalent. > Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file > "/var/lib/pgsql/data/postgresql.conf" contains errors > Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server > Feb 17 13:46:16 db1 pg_ctl: Examine the log output. > Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, > code=exited status=1 > Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. > Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. > Feb 17 13:46:16 db1 systemd: postgresql.service failed. > > > I have kernel parameters specified in /etc/sysctl.conf > > vm.swappiness=0 > vm.overcommit_memory=2 > vm.overcommit_ratio=90 > kernel.shmmax=35433480192 > kernel.shmall=8650752 > > After the update my postgresql service could not start because these > parameters had been reset, I promptly rebooted to server to re-apply them. > > Has something changed?!? after a reboot the service still complained > that my max_stack_depth was too high because kernel shmmax and shmall > were too low with the same error shown above. > > [root at db1 ~]# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 514616 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 514616 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > confirms that my entries in /etc/sysctl.conf were ignored. > > Why would these not work anymore? > > Are the parameters specified elsewhere now? > > any information would be very helpful! > > Thanks > > Michael > (slightly more grey now)Since you are talking about SystemD .. I assume c7. In c7 .. there is a symlink to /etc/sysctl.d/99-sysctl.conf to /etc/sysctl.conf Have you verified your sysctl.conf actually contains those settings still. Your best bet on CentOS-7 is to create a new file in /etc/sysctl.d/ called something like 99-postgres.conf and put youjr mods in there. That way it will never change. Also .. verify all the files in /etc/sysctl.d/ and /etc/sysctl.conf are set to this label for selinux: unconfined_u:object_r:etc_t:s0 See this for labeling: red.ht/1ooTpiI But, /etc/sysctl.conf should still work in centos-7. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos/attachments/20160217/230f25d9/attachment-0001.sig>
Johnny Hughes
2016-Feb-17 14:44 UTC
[CentOS] New glibc for CentOS-6 and CentOS-7 and CVE-2015-7547
On 02/17/2016 08:39 AM, Johnny Hughes wrote:> On 02/17/2016 08:10 AM, Michael H wrote: >>> The easy answer is yes .. glibc requires so many things to be restarted, >>> that is the best bet. Or certainly the easiest. >>> >>> Note: in CentOS 7, there is also a kernel update which is rated as >>> Important .. so you should boot to that anyway: >>> https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html >>> >>> Here is a good link to figure out what to restart if you don't want to >>> reboot: >>> >>> https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-an-upgrade/ >>> >>> and there is this thread: >>> http://markmail.org/message/dodinyrhwgey35mh >>> >>> But generalyl, after a glibc update or a kernel update .. rebooting is >>> easiest and it ensures everything is protected. >> >> Wow, so, I updated my server (yum update -y) which applied a new kernel >> and the new glibc among other things, After the update completed it >> knocked my master postgresql database offline. >> >> >> Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... >> Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter >> "max_stack_depth": 16384 >> Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed >> 7680kB. >> Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth >> limit via "ulimit -s" or local equivalent. >> Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file >> "/var/lib/pgsql/data/postgresql.conf" contains errors >> Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server >> Feb 17 13:46:16 db1 pg_ctl: Examine the log output. >> Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, >> code=exited status=1 >> Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. >> Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. >> Feb 17 13:46:16 db1 systemd: postgresql.service failed. >> >> >> I have kernel parameters specified in /etc/sysctl.conf >> >> vm.swappiness=0 >> vm.overcommit_memory=2 >> vm.overcommit_ratio=90 >> kernel.shmmax=35433480192 >> kernel.shmall=8650752 >> >> After the update my postgresql service could not start because these >> parameters had been reset, I promptly rebooted to server to re-apply them. >> >> Has something changed?!? after a reboot the service still complained >> that my max_stack_depth was too high because kernel shmmax and shmall >> were too low with the same error shown above. >> >> [root at db1 ~]# ulimit -a >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 514616 >> max locked memory (kbytes, -l) 64 >> max memory size (kbytes, -m) unlimited >> open files (-n) 1024 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 8192 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 514616 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> confirms that my entries in /etc/sysctl.conf were ignored. >> >> Why would these not work anymore? >> >> Are the parameters specified elsewhere now? >> >> any information would be very helpful! >> >> Thanks >> >> Michael >> (slightly more grey now) > > Since you are talking about SystemD .. I assume c7. > > In c7 .. there is a symlink to /etc/sysctl.d/99-sysctl.conf to > /etc/sysctl.conf > > Have you verified your sysctl.conf actually contains those settings still. > > Your best bet on CentOS-7 is to create a new file in /etc/sysctl.d/ > called something like 99-postgres.conf and put youjr mods in there. > That way it will never change. > > Also .. verify all the files in /etc/sysctl.d/ and /etc/sysctl.conf are > set to this label for selinux: > > unconfined_u:object_r:etc_t:s0 > > See this for labeling: > red.ht/1ooTpiI > > But, /etc/sysctl.conf should still work in centos-7.Looks like that is working (it seems to be reading your /etc/sysctl.conf file) based on your other post in this thread. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos/attachments/20160217/603c648e/attachment-0001.sig>
Michael H
2016-Feb-17 15:05 UTC
[CentOS] New glibc for CentOS-6 and CentOS-7 and CVE-2015-7547
On 17/02/16 14:44, Johnny Hughes wrote:> On 02/17/2016 08:39 AM, Johnny Hughes wrote: >> On 02/17/2016 08:10 AM, Michael H wrote: >>>> The easy answer is yes .. glibc requires so many things to be restarted, >>>> that is the best bet. Or certainly the easiest. >>>> >>>> Note: in CentOS 7, there is also a kernel update which is rated as >>>> Important .. so you should boot to that anyway: >>>> https://lists.centos.org/pipermail/centos-announce/2016-February/021705.html >>>> >>>> Here is a good link to figure out what to restart if you don't want to >>>> reboot: >>>> >>>> https://rwmj.wordpress.com/2014/07/10/which-services-need-restarting-after-an-upgrade/ >>>> >>>> and there is this thread: >>>> http://markmail.org/message/dodinyrhwgey35mh >>>> >>>> But generalyl, after a glibc update or a kernel update .. rebooting is >>>> easiest and it ensures everything is protected. >>> >>> Wow, so, I updated my server (yum update -y) which applied a new kernel >>> and the new glibc among other things, After the update completed it >>> knocked my master postgresql database offline. >>> >>> >>> Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... >>> Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter >>> "max_stack_depth": 16384 >>> Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed >>> 7680kB. >>> Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth >>> limit via "ulimit -s" or local equivalent. >>> Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file >>> "/var/lib/pgsql/data/postgresql.conf" contains errors >>> Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server >>> Feb 17 13:46:16 db1 pg_ctl: Examine the log output. >>> Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, >>> code=exited status=1 >>> Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. >>> Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failed state. >>> Feb 17 13:46:16 db1 systemd: postgresql.service failed. >>> >>> >>> I have kernel parameters specified in /etc/sysctl.conf >>> >>> vm.swappiness=0 >>> vm.overcommit_memory=2 >>> vm.overcommit_ratio=90 >>> kernel.shmmax=35433480192 >>> kernel.shmall=8650752 >>> >>> After the update my postgresql service could not start because these >>> parameters had been reset, I promptly rebooted to server to re-apply them. >>> >>> Has something changed?!? after a reboot the service still complained >>> that my max_stack_depth was too high because kernel shmmax and shmall >>> were too low with the same error shown above. >>> >>> [root at db1 ~]# ulimit -a >>> core file size (blocks, -c) 0 >>> data seg size (kbytes, -d) unlimited >>> scheduling priority (-e) 0 >>> file size (blocks, -f) unlimited >>> pending signals (-i) 514616 >>> max locked memory (kbytes, -l) 64 >>> max memory size (kbytes, -m) unlimited >>> open files (-n) 1024 >>> pipe size (512 bytes, -p) 8 >>> POSIX message queues (bytes, -q) 819200 >>> real-time priority (-r) 0 >>> stack size (kbytes, -s) 8192 >>> cpu time (seconds, -t) unlimited >>> max user processes (-u) 514616 >>> virtual memory (kbytes, -v) unlimited >>> file locks (-x) unlimited >>> >>> confirms that my entries in /etc/sysctl.conf were ignored. >>> >>> Why would these not work anymore? >>> >>> Are the parameters specified elsewhere now? >>> >>> any information would be very helpful! >>> >>> Thanks >>> >>> Michael >>> (slightly more grey now) >> >> Since you are talking about SystemD .. I assume c7. >> >> In c7 .. there is a symlink to /etc/sysctl.d/99-sysctl.conf to >> /etc/sysctl.conf >> >> Have you verified your sysctl.conf actually contains those settings still. >> >> Your best bet on CentOS-7 is to create a new file in /etc/sysctl.d/ >> called something like 99-postgres.conf and put youjr mods in there. >> That way it will never change. >> >> Also .. verify all the files in /etc/sysctl.d/ and /etc/sysctl.conf are >> set to this label for selinux: >> >> unconfined_u:object_r:etc_t:s0 >> >> See this for labeling: >> red.ht/1ooTpiI >> >> But, /etc/sysctl.conf should still work in centos-7. > > Looks like that is working (it seems to be reading your /etc/sysctl.conf > file) based on your other post in this thread.Hi Johnny, Excuse my ignorance, if the sysctl.conf is working then why would my database throw this error still?>>> Feb 17 13:46:11 db1 systemd: Starting PostgreSQL database server... >>> Feb 17 13:46:11 db1 pg_ctl: LOG: invalid value for parameter >>> "max_stack_depth": 16384 >>> Feb 17 13:46:11 db1 pg_ctl: DETAIL: "max_stack_depth" must not exceed >>> 7680kB. >>> Feb 17 13:46:11 db1 pg_ctl: HINT: Increase the platform's stack depth >>> limit via "ulimit -s" or local equivalent. >>> Feb 17 13:46:11 db1 pg_ctl: FATAL: configuration file >>> "/var/lib/pgsql/data/postgresql.conf" contains errors >>> Feb 17 13:46:16 db1 pg_ctl: pg_ctl: could not start server >>> Feb 17 13:46:16 db1 pg_ctl: Examine the log output. >>> Feb 17 13:46:16 db1 systemd: postgresql.service: control process exited, >>> code=exited status=1 >>> Feb 17 13:46:16 db1 systemd: Failed to start PostgreSQL database server. >>> Feb 17 13:46:16 db1 systemd: Unit postgresql.service entered failedstate.>>> Feb 17 13:46:16 db1 systemd: postgresql.service failed.Should my output from ulimit -a not correspond to my sysctl.conf parameters? This server was tested heavily and rebooted tens of times before it moved into production, I can't understand what has changed other than now I get inconsistent output from sysctl -a and ulimit -a. I am quite confident this wasn't the case before I updated today. ulimit -s is definitely not showing the correct parameter that I specified in /etc/sysctl.conf. Thanks, Michael