I think everyone agrees that a common build system is a necessary step if any of this is going to work.
Thinking about how much or how little would be required beyond a common build system in order to get a working package management system is still a valid thing to do.
I expect it is necessary to keep build configuration portable with respect to packaging and environment configuration. In other words, I expect downloading from URLs in Makefiles and CMakeLists to be a local maximum.
Keeping a parallel set of instructions or metadata that includes specific URLs and such might work, though. As long as you can skip all that when a system package, filesystem path, or git submodule is more appropriate.
Thinking about how much or how little would be required beyond a common build system in order to get a working package management system is still a valid thing to do.