Found the problem. It's polling by drm_kms_helper, as detailed here:
http://souriguha.wordpress.com/2011/03/08/how-to-solve-problem-with-thinkpadkslowd-kworker-on-linux-kernel-2-35-2-36/
Putting "N" or "0" in
/sys/modules/drm_kms_helper/parameters/poll, or booting with the
"drm_kms_helper.poll=0" kernel parameter, fixes the issue (and a few
others, e.g. kworker threads consuming excessive CPU time).
KMS polling seems to be related to autodetecting and configuring new display
devices, so there may be issues with this fix if you have to use a projector or
other hotpluggable display device. Not sure what those issues would be though.
Sorry if resurrecting a thread like this is frowned upon BTW...
