libvirt users - Feb 2017 - Real time threads don't work in libvirt containers under CentOS 7.3

We've been using libvirt based containers under CentOS 7 and everything 
has been working fine. One application we run in our containers is ctdb, 
which uses SCHED_FIFO (real time) threads. This has been working without 
problems until our recent upgrade to CentOS 7.3. For some reason, ctdb 
is no longer able to create real time threads, and I've tried a simple 
program myself that confirms this. The same program works fine on the 
hypervisor so I know the kernel supports real time. Does anyone know 
what may have changed in CentOS 7.3 that breaks real time threads in 
libvirt containers?

This is the simple test program I created to verify the real time 
threads are failing. This same program works in a libvirt container in 
CentOS 7.2.

#include <stdio.h>
#include <string.h>
#include <pthread.h>

pthread_t test_thread;

void *test(void *arg)
{
     printf("Starting thread\n");
     sleep(1);
     printf("Thread complete\n");
     return 0;
}


int main(int argc, char *argv[])
{
     int rc;
     printf("Starting main\n");
     struct sched_param tsparam;
     pthread_attr_t tattr;
     memset(&tsparam, 0, sizeof(tsparam));
     pthread_attr_init(&tattr);
     pthread_attr_setinheritsched(&tattr, PTHREAD_EXPLICIT_SCHED);
     pthread_attr_setschedpolicy(&tattr, SCHED_FIFO);
     tsparam.sched_priority = sched_get_priority_max(SCHED_FIFO) - 7;
     pthread_attr_setschedparam(&tattr, &tsparam);
     if ((rc = pthread_create(&test_thread, &tattr, test, NULL)) != 0) {
         printf("Unable to start rt thread\n");
     }
     return 0;
}

On Mon, Feb 06, 2017 at 09:32:29AM -0800, Peter Steele
wrote:
> We've been using libvirt based containers under CentOS 7 and everything
has
> been working fine. One application we run in our containers is ctdb, which
> uses SCHED_FIFO (real time) threads. This has been working without problems
> until our recent upgrade to CentOS 7.3. For some reason, ctdb is no longer
> able to create real time threads, and I've tried a simple program
myself
> that confirms this. The same program works fine on the hypervisor so I know
> the kernel supports real time. Does anyone know what may have changed in
> CentOS 7.3 that breaks real time threads in libvirt containers?
If I had to guess, perhaps some tunable in cgroups is denying execution
time ?


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

On 02/06/2017 09:47 AM, Daniel P. Berrange wrote:
> On Mon, Feb 06, 2017 at 09:32:29AM -0800, Peter Steele wrote:
>> We've been using libvirt based containers under CentOS 7 and
everything has
>> been working fine. One application we run in our containers is ctdb,
which
>> uses SCHED_FIFO (real time) threads. This has been working without
problems
>> until our recent upgrade to CentOS 7.3. For some reason, ctdb is no
longer
>> able to create real time threads, and I've tried a simple program
myself
>> that confirms this. The same program works fine on the hypervisor so I
know
>> the kernel supports real time. Does anyone know what may have changed
in
>> CentOS 7.3 that breaks real time threads in libvirt containers?
> If I had to guess, perhaps some tunable in cgroups is denying execution
> time ?
>
In CentOS 7.2 the path /sys/fs/cgroup/cpu/cpu.rt_runtime_us is zero by 
default and I had to set it to a non-zero value, e.g.

echo 25000 >/sys/fs/cgroup/cpu/cpu.rt_runtime_us

in order to use real time threads in containers. In CentOS 7.3, this 
path is non-zero already (950000 on my system) and it doesn't even let 
me write a new value to it. For some reason though, real time threads 
are still not allowed. Apparently some other cgroup param has to be set 
in 7.3. The question is which one?

Peter