Actually we already know that people metabolize caffeine differently. https://pharmrev.aspetjournals.org/content/70/2/384 - but caffeine is also very transparent drug. You can use it for years without realizing the negative/positive impact it has on you. How long have you even quit caffeine for? Withdrawal effects tend to show up in 2 weeks or so for me. The anxiety that it causes for me isn't very acute either. It's more like just background anxiety.
I used and abused coffee for most of my twenties without ever realizing it had such an impact. Nowadays I just drink small cups of tea.
I've had pauses of a few months when there wasn't any good coffee around; sometimes there wasn't even any decent tea (green/black). There was mediocre chocolate (supermarket stuff) but I don't think I ate a whole lot of that either (as a caffeinated-coffee replacement). Mostly happened when I was doing a weird combination of travel for work and not knowing where the good coffee spots were, coming back home after being away for two weeks to then move to a house I bought that was finally ready to move in to so another few weeks of having too much stuff packed away to make any coffee (and too busy to do it anyway). And then when the move was over, on vacation for a few weeks, again with no clue where to get coffee and all there is, is Starbucks, gas stops, and restaurant coffee (which more often than not is terrible anyway).
Maybe it does have an effect but being too busy to notice masked it?
I've had shorter pauses as well, broke a few bones in a sporting accident and didn't really feel like getting or making coffee, was very into ginger infusions, sometimes tamed with a bit of honey or lemon.
On one hand this makes me very curious to just not drink coffee for a few weeks and see if it does anything, on the other hand I pretty much stopped drinking sugary drinks years ago so that just leaves water and tea for me, and tea would have to be paused as well because I have a variety of teas where some have caffeine as well. Maybe if there was a good decaf around I'd just drink that as an experiment, but I haven't found one that doesn't taste weird yet.
That said, I'm not drinking all that much coffee (1 or 2 filters or 2 espressos or maybe one of each is about average), and sometimes I just forget to make some if I'm busy doing stuff.
I used and abused coffee for most of my twenties without ever realizing it had such an impact. Nowadays I just drink small cups of tea.
Also would like to point out: https://www.audible.com/pd/Caffeine-Audiobook/B083MVZ91Y is a pretty fun read. Not super scientific but interesting nonetheless.