I implemented number porting for a startup I worked for. It's all pretty scary: yes, it's easy to impersonate someone if you have some information about them, but some carriers don't check any sort of porting pin at all. That is, attackers sometimes do not need to impersonate you, they just have to ask for your number.

It would be great if we all had something big more secure, but in the meantime understand your carrier's port out process and that you have your number secured.