I did something similar but then I had to press my hand on his back or butt so he felt like he was still being held, then count to 100 and release it as slowly as possible.

Babies are ridiculous.