> We can also compare locales that implement do implement $policy and ones that don't. As with anything it's not perfect but it can be indicative.

It's so imperfect it often inverts the sign of the result, e.g. cities that implement some anti-X policy have even more X than cities that don't, but the reason is that the cities that implemented the policy were the ones expecting the biggest problem. Then a policy that was effective in reducing the growth rate of X from 30% to 10% is accused of increasing X by 10%.

> For the exact same reason it's often so difficult to tease out the impact of policies in data, it's even harder to reason through what will happen from first principles.

The point isn't that you can't get the reasoning wrong, it's that most of the data is just mud. It's knowing that X = Y + Z and then someone gives you the value of Y, which is data, and asks you to solve for X. You still have no ability to answer the question.

But okay, fair enough, you have some data, you're maybe one step closer to solving the problem (even though you haven't yet). Here's what gets me. If the data contradicts the expectation, you should now have a theory for why. Is it because the data has confounders and hasn't really demonstrated anything one way or the other? Is it because you made mass transit free but it still didn't cover the path to where people live, and in order to be effective you have to do both? What's the reason? That is an important question because it determines what you should do instead, if anything.

Just "data says that doesn't work" isn't really refuting anything. If the reason is confounders then the data isn't even a counterargument. If the reason is that it doesn't work as implemented there but there is a variant that works, that's important, but then you can still do it by adopting the variant. And if it actually doesn't work at all then where's the explanation? How do you know what to do instead?

For example, if people don't respond to price differences then congestion pricing shouldn't work either, right? If price differences do work but part of the price has to be an increase to trigger loss aversion and then it's the total price that matters, what if you make mass transit free and then increase the federal gas tax instead of using congestion pricing? People see the higher price, compare it to the lower price and you get a double effect but still don't need to pay for separate collections infrastructure or have the privacy cost and the gas tax also encourages fuel efficiency.

But if the data is just some convoluted mud that can't answer the questions then it's worse than nothing, because it makes you think you know something you don't actually know.