I'm referring more to New York the state, to compare it to its neighboring state New Jersey

“The city” refers to Manhattan specifically, not New York City in general. Being in the other boroughs, you might say “I’m heading to the city later”.

Or if you're upstate and going to one of the boroughs, you'll generally refer to it by name. "I'm going to Brooklyn / Queens / the Bronx", not "I'm going to the city" or "I'm going to New York City".

I see what you are saying but one could easily go to two or more of those boroughs in single trip.