This would always piss me off, eventually I put this into my rc file https://gist.github.com/adamnemecek/bf9d8e4c0356bfdd86c2

What I then do is that I select a file in the directory I want to cd into, hit CMD + C, switch to terminal and run sd on CMD+V. The script basically cd's into the same dir as the file you selected.

I realize that this workflow might not be for everyone but it's kind of ok for me.