I don't understand why phone manufacturers don't just get into the camera business, then. They certainly all seem to love making phones that brag about having large sensors and fancy (tiny) lenses... so why not just go one step further and make a "phone" that only runs a camera app, with a lens mount rather than a fixed lens, hardware mode switches, and a tripod screw mount?
The camera market small, quite crowded (before Sony kind of created the mirrorless market the only serious contenders were Nikon and Canon, now you got Nikon, Canon, Sony, Fuji and Panasonic), smartphones already killed the entry level and compact camera markets and the tech is quite different from smartphone cameras.
Phone cameras don't have sensors or optics at a quality comparable to basic point and shoots. They have to fit into a tiny space and performance is necessarily compromised. They're making it all up on software processing.