In your situation, it takes effort to take the leap of faith. I did it 10 years ago, and it was tough even if I had savings and no debts/mortgages. The main issue is managing the 24/7 mental stress for years; back then, I had two teenagers, and life was expensive. Most of the time, the founder's salary is not enough to support your family—at least in the Bay Area—and, besides, when you work 7 days a week, the situation becomes even harder to sustain. That puts much pressure on you and your partner for years before you can get something back.
My suggestion:
a) If you want to do that, be sure you have at least 2 years of savings, and not just to survive, but to live a good life.
b) Be sure that your partner is truly happy with your choice.
c) Startup vs life-style business. Creating a startup when you have a family and financial obligations is much more challenging than becoming an entrepreneur and creating a company to make a living. Be entirely sure that is the road you want to pursue. If you have doubts, wait.
d) Some people have a positive—and balanced—approach to life that can help a lot when your launch your startup. If you are one of those, maybe it's a Yes.