I suspect it has more to do with the incentives faced by those who set the speed limits.
I don't know exactly how it goes. If it's chosen (or heavily influenced) by politicians, I suspect politicians can win some votes by saying they'll improve safety by lowering speed limits (particularly around schools or other places where children might be); while they'd be less likely to win as many votes by saying they'll raise the limits (exposing them to the risk of their opponents calling them reckless/irresponsible). If it's chosen by non-elected officials, that's less of an issue, but something like it may still be there; or it may be that raising the limit and then there being a fatal accident will damage your career, while lowering the limit and irritating everyone will not damage your career.