That's not what you said at all. Your stated position is that rights are fundamental and government has to justify any imposition on those, yes? So you're saying that the bill of Rights > the Rest of the Constitution.
Go read the Federalist Papers, there is no way the founders intended it to work that way and courts have never interpreted it that way either. James Madison explains it beter than I can:
“If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself. A dependence on the people is, no doubt, the primary control on the government; but experience has taught mankind the necessity of auxiliary precautions.”
The Federalist papers argued against the need for things like the Bill of Rights, which were considered redundant and perfectly placeable in the law itself (and possibly counter-productive, if it was ever assumed that the Bill of Rights represented the sum of your rights, which is what the 9th Amendment was passed to try to prevent).
Either way, that doesn't answer the point of whether this argument by Madison is valid or not.
Go read the Federalist Papers, there is no way the founders intended it to work that way and courts have never interpreted it that way either. James Madison explains it beter than I can:
“If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself. A dependence on the people is, no doubt, the primary control on the government; but experience has taught mankind the necessity of auxiliary precautions.”