Yes, this whole line of argument that Snowden has put forward is beside the point. Whether you call it "exile" or not, it is perfectly reasonable for the US to revoke his passport. It is certainly not "extralegal." It's not even unusual. It is what happens when you're charged with a felony by a US court.
Revoking his passport is not extralegal. Nobody's arguing that.
Exerting unknown influence behind closed doors with those who have the ability to grant or deny his requests for asylum is an entirely different matter, though.