Another good starting point is #xkcd. it is a large channel with a lot of tech-ish and math-y people.
EDIT: Note that channel discovery on IRC is largely a word-of-mouth thing so don't be shy to ask people for channel recommendations on irc. Most programming language have a dedicated channel, e.g. #D or #python or whatnot.
Are you talking about freenode? I just joined #xkcd and it's 8 people and no topic set. 3 other users joined and immediately left, I guess those were from HN aswell.
(A big channel for everyone to join is ##programming, by the way.)
EDIT: Note that channel discovery on IRC is largely a word-of-mouth thing so don't be shy to ask people for channel recommendations on irc. Most programming language have a dedicated channel, e.g. #D or #python or whatnot.