There is no loss of opportunity. You can borrow against invested money at very low interest rates, lower than mortgage interest rates, and people often do. In today's crazy real estate market, when people make "all cash offers" that is often just cash borrowed against securities. It is some of the cheapest debt that exists.
It's hard to argue in the abstract without figures and expected returns. If paying off the debt opens the door higher returns, then I'm all for it. All things being equal, I'll keep $250k cash with a $250k mortgage at 2.7% today, as opposed to a paid-off house and a $250k hole in my account.
How is a bunch of money tied up in debt, and an equal amount invested to hedge that not a huge opportunity sink?