Sometimes you have to profile debug builds, and as always @superluminal@mastodon.gamedev.place
is perfect for that.
My matrix multiplication was using variables on the stack, which caused a lot of stack checks.
Rewrote the code, debug performance went from 23ms to 6ms.