This is totally true for some models of computation (including some studied in academic computer science), but it's not true for the models of computation that the halting problem applies to, like the Turing machine, which explicitly has an infinite tape and therefore has the potential for an unbounded number of machine states.
In fact, this unboundedness is a core part of what makes these models of computation so expressive. In the Gödel's incompleteness theorem analogy, you can say "there is an integer such that ..." or "there is no integer such that ...". In the Turing machine model, you can write programs that search for counterexamples to mathematical claims. Because these programs are written to try every integer, you can only tell if they eventually halt by yourself resolving the mathematical claim.
For example, we can write a program that tests the Goldbach conjecture or the 3n+1 conjecture by brute force. Determining if these programs' search through all integers will halt or not is equivalent to resolving the status of these conjectures!
Here's an example of a Python program whose halting behavior -- if run on a hypothetical Python interpreter with infinite RAM and the ability to make use of it -- is currently unknown. It looks for counterexamples to the 3n+1 conjecture, and it tries every integer to see if it's a counterexample.
start = i = 1
visited = []
while True:
if i in visited:
if 1 not in visited:
break
else:
print(start, visited)
visited = []
start += 1
i = start
continue
visited.append(i)
if i % 2:
i = 3*i + 1
else:
i //= 2
On an actual computer's Python interpreter, I believe this will eventually halt with a MemoryError exception (although I'm not at all sure that you can actually run it long enough to hit that, as it will probably take many years!), failing to resolve the mathematical question. That potential for the MemoryError is why it matters whether your model has infinite memory or not!
(You can make a considerably more memory-efficient version of this which should be able to search much further before hitting a MemoryError, but I don't expect it changes the ultimate fate of the program when run on a real physical computer.)
In fact, this unboundedness is a core part of what makes these models of computation so expressive. In the Gödel's incompleteness theorem analogy, you can say "there is an integer such that ..." or "there is no integer such that ...". In the Turing machine model, you can write programs that search for counterexamples to mathematical claims. Because these programs are written to try every integer, you can only tell if they eventually halt by yourself resolving the mathematical claim.
For example, we can write a program that tests the Goldbach conjecture or the 3n+1 conjecture by brute force. Determining if these programs' search through all integers will halt or not is equivalent to resolving the status of these conjectures!