My fear is similar, but more along the lines of adversarial attacks as various weaknesses are exposed. Imagine people taking advantage of zero-day exploits that cause self driving cars to veer off the road, stop suddenly, collide, etc. It is really not that far fetched. This technology is very far away from maturity, IMO.
Your run of the mill hacker sure, but adversarial nations about to invade another nation and want to keep the Western World occupied? DC was brought to a crisis by the sniper; can you imagine if cars suddenly started plowing through school pickup lines or concert venues with no way for occupant to stop it? Or even without an occupant?