I totally agree, that is the main missing feature in my opinion and the reason why I added a select box on top.
At first I wanted to use polymaps, which supports zoom and pan, but I found it difficult to draw on top of it (for the "arrows") and it lacked ie support. So I went for Raphael.js, but found it to cumbersome to implement zoom and pan.