You always need to make some assumption about the causal process before you can infer causality from evidence.
For example, take a randomized trial, where half of patients are given a drug, and half a placebo. If we define Y to be the outcome of interest, and X to be 0 when the patient received the placebo and 1 when they received the drug, then a positive outcome would be a correlation between X and Y. A person might object that "correlation does not imply causation", however, we know enough about the causal process involved, to infer that if Y and X are correlated, this must be because X caused Y.
There are many other ways that causation can be inferred, such as instrumental variables, regression discontinuity, or panel regressions. However all these methods require making an assumption about the causal process underlying the statistical distribution. Whether these assumptions are correct is often a matter for debate.
Well, that's the point! Correlation is seen in the data, then causation calls to be tested for. Your test for it, and continue to see correlation. Yet, the causation stays as an "assumption" that is a matter of debate. This brings back the question, how do you establish causation then?
Your argument is basically saying that if there is understandable physics behind the correlation, then causation could be safely inferred. However, that does not ultimately answer the question still, since how would physics would have established that explanation to begin with.
We happen to live in a world where sometimes things can be ascribed specific causes, but that is due to the nature of the physical universe, not an abstract truth.
Causality in our best theories of physics (general relativity, quantum field theory) is extremely complex and subtle. But that isn't needed anyway, what we use in ordinary reasoning is "naive" physics, an approximation of physical reality. I don't think this "naive physics", which we could also call common sense, can be derived from a statistical model, although maybe it can.
Going back to the randomized trial, presumably the choice to give a drug to a patient vs a placebo was determined by a random number generator. This in turn is a physical process, and there is no reason why the physical process that generates this number could not also influence a person's health outcome. It is only our naive physics model that tells us that there is no such process, and the only way the random number generator affects the health outcome, is through the choice of drug.
So to summarize, doing causal inference relies on a model of reality, and this model is sufficiently complex that the model itself cannot (for now) be derived through purely statistical means, but rather is arrived at through ordinary human thought.
One thing though that is a bit hard to explain, and I would like to hear your thoughts/understanding on this.
Establishing causation includes at least establishing precedence in addition to correlation. In other words, the very definition of time is intertwined with the relationship between causes and effects. And Physics does not yet claim to have understood what time means (AFAIK): It is really real? Is it something just within our heads? ...
If you do not treat causation as a fundamental notion, do you treat time as one? If yes, how do you explain the dependence between the two. If not, well, that becomes a subject of even more basic discussion since then I would ask what do you treat to be fundamental notions. :-) (There does not seem to be a way for us to get around fundamental notions completely.)
When I studied physics as an undergrad, I was made to take a course in experimental particle physics (in Germany they call this "Phenomenology", not in English to my knowledge). I complained that I wanted to study the fundamental laws of physics, not how they were tested. They university advisor replied that in order to understand how the world works, first we have to know what it looks like.
This is basically my view on this matter. We can only take for granted the existence of the objective universe. Everything else must be derived from experiment, including concepts that seem fundamental like time or causality.
One interesting thing about causality is that the fact that the macroscopic world obeys cause-and-effect laws is usually attributed to the second law of thermodynamics. But where does the second law of thermodynamics come from? I've heard it stated (can't say if this is universally held by physicists) that the second law of thermodynamics holds because the initial state of the universe had very low entropy, while the final state of the universe will have very high entropy. That is the second law of thermodynamics is "caused" by both the initial and final states of the universe. So causality and time are something that we don't fully understand yet.
For example, take a randomized trial, where half of patients are given a drug, and half a placebo. If we define Y to be the outcome of interest, and X to be 0 when the patient received the placebo and 1 when they received the drug, then a positive outcome would be a correlation between X and Y. A person might object that "correlation does not imply causation", however, we know enough about the causal process involved, to infer that if Y and X are correlated, this must be because X caused Y.
There are many other ways that causation can be inferred, such as instrumental variables, regression discontinuity, or panel regressions. However all these methods require making an assumption about the causal process underlying the statistical distribution. Whether these assumptions are correct is often a matter for debate.