I saw Concorde taking off a few times up close at air shows when it was still in service. It was significantly louder than any other aircraft I've ever been near (it had 4 enormous engines with afterburners - not that surprising I guess). Takeoff noise really was a big issue.

It used to fly over my house when I was a teenager. I lived an hour or two (drive) from Dulles Airport and when it (rarely) passed overhead, it was still fairly loud. Nothing like you'd hear at an air show, but we always knew to look up if we heard it when we were out in the yard.

It was pretty cool because it was close enough to takeoff that you could easily make out the distinct shape of the plane.

Note the proposed Concorde B that never actually reached production due to cancelled orders of the A variant went without the afterburners.

I don't recall it being significantly louder than any other aircraft I've been near, but the vast majority of aircraft that were any where close were military aircraft!

I think you're right, though take-off noise was also the subject of many complaints about the Concorde, not just the sonic boom. Reducing take-off noise was already one of the design considerations of the Concorde, and supposedly the Concorde was better in this regard than the Russian Concordeski (Tu-144.)