Of course there are debuggers outside of the IDE world. For instance in Ruby we usually use a debugger as a library (gem) and we call it as a method call wherever you would put the breakpoint in your IDE.
Typing in a method call on the line where you want to debug isn't much of a leap from clicking a breakpoint.
I am sure the situation is similar in other languages.
The big difference I'd point out is that breakpoints can be added and removed at runtime. Sounds like what you're describing is a situation where you have to modify your source code to add a breakpoint (similar to a "debugger" statement in JavaScript or C#). To me, the big value of an IDE is that it lets you set/unset breakpoints, step, inspect arbitrary data, and modify application state, all at runtime. Now, I still typically work in VIM and pop open an IDE/debugger only when I hit a hairy problem, but I still find them really useful in those cases.
Typing in a method call on the line where you want to debug isn't much of a leap from clicking a breakpoint.
I am sure the situation is similar in other languages.