Yeah but when you are first starting out it really doesn't matter - what matters more is keeping interested enough to keep learning and growing from wherever you start.
It will affect who you become though - 'a java shop' vs. a Ruby-on-Rails start-up, say - what people think of you, or where you're hireable.
I at least glance over all the recruiter spam I get, and not much of it takes the 'language experience doesn't matter, transferable comprehension of fundamentals does' line.