I am using tmux with a 200,000 line history and have no issues.

Also, in 2017-2018 I contributed a few fixes for memory leaks related to buffer history, so make sure you are using a recent version.