The two that come to mind are ConTeXt https://en.wikipedia.org/wiki/ConTeXt (official: https://contextgarden.net) and DocBook https://en.wikipedia.org/wiki/DocBook (official: https://docbook.org). There are others, but they're less-serious and often not general purpose. Then there's the entire class of Markdown-based solutions (I personally wouldn't attempt academic work with any of those).

The other typesetting system is troff https://en.wikipedia.org/wiki/Troff (official: https://troff.org), but it isn't a "tried to build something better than TeX/LaTex" since it predates TeX by a few years. Arguably TeX is an attempt at something better than troff.

The case for troff is pretty strong: https://rkrishnan.org/posts/2016-03-07-how-is-gopl-typeset.h...

But honestly LaTeX is so ingrained in academia that it is probably worth sticking with it. You might even learn to love it.