IMHO it will be more challenging than just building an orbital class rocket. You have most of the challenges of space flight in terms of exotic fuels etc. But you spend the entire flight in a hypersonic airflow, where any problem could escalate instantly. At least in orbit you have time to fix things.

And the vehicle needs to work at completely different altitudes and velocities. This could be very challenging in terms of structures, landing gear, braking etc. Particularly if the mass of the vehicle is constrained. Many component will need to be lighter and have orders of magntitude better performance than a traditional airliner.