I believe the biggest problem is due to the design of bike lanes. They are often narrow and squeezed between car lanes and car parking. Due to this design bike lanes are also more often subject to being blocked by construction and forced to merge with car traffic. The visibility on bike lanes are way poorer than regular car lanes due to the above factors. In cities you also have to deal with the loading and unloading of goods. This means large, tall vehicles blocking the sight and often crates and people moving in your lane.

How often do you experience your whole lane blocked by the content of a whole appartement?

In Stockholm where I live snow, and ice make bike lanes hard to see and also makes them inaccessible.

I really agree that you shouldn't keep a speed where you can't control your vehicle, but so common design flaw puts an unreasonable amount of responsibility on the biker compared to heavier vehicles misusing the bike lane.