Are you on wireless? Apparently bittorrent tends to kill a lot of wireless routers, as it opens too many simultaneous connections for them to handle. I will regularly see my download speeds shoot up, only to crash to 0. Resetting my router fixes it, but the modem itself is fine.
Wireless has nothing to do with it. The problem is that most consumer routers don't have enough RAM to firewall more than a hundred connections. It only correlates with wireless usage because most consumer routers are bought and used specifically for wi-fi.
I don't suppose your router happens to be a blue-and-gray El Cheapo Walmart Linksys, like mine...? Haha. It's so ghetto; I can only forward 10 port ranges total, for example.
Out of curiosity, does uploading/seeding work correctly when you torrent?
Check out Tomato firmware: http://www.polarcloud.com/tomato
Much more control than stock firmware. It may be a matter of adjusting max connections and/or TCP timeout.