I wrote mine in plain LaTeX, never had an issue with it. Regarding code, you can use the listings package (\lstincludelisting) to include source code files directly. I fail to see the interest in having a bastardized, not quite TeX, not quite MD file (unless you're using something like Org, of course, and using its other functionalities).

Regarding having unicode for the maths, how did you deal with symbols that needed scaling (brackets, integrals, etc.?) They may look simpler when seen as text, but they certainly won't render nicely as math...