In most of Europe (and probably elsewhere), you can’t throw a rock without hitting some sort of historically significant thing.

We used to smoke weed on the local Roman wall. It was also a popular destination for rock climbers to practice. The wall has been there for 2000 years. It’s fine.