I've done
echo 1 > /proc/sys/kernel/modules_disabled
on some servers that don't need to load additional modules after startup. I've just configured this to run 5 minutes after boot. (Timing was an arbitrary choice.)
This reduces the attack surface and should help mitigate against vulnerabilities exploitable via kernel modules that you don't normally use, at the expense of on-demand loading of modules of course (including e.g. usbhid for remote kvm, so make sure whatever you might need is loaded first).
The setting takes a reboot to undo.