Indeed they can be to many drivers, which means that less patient drivers will be deterred from driving through residential areas with a 20mph speed limit and use faster roads instead thus reducing the traffic volumes in residential areas.
… and they can be dangerous because frustrated drivers will start overtaking.
This happens on any road regardless of the speed limit. On National Speed Limit (60mph) stretches of road frustrated drivers will overtake because the vehicle in front is not travelling fast enough (for them). This problem exists regardless of what the speed limit is.