It can be ignored (it's the equivalent to a "keep out" sign on a lawn), but I very much doubt Google et al. (Edit: Oops, Bing et al.) will actually ignore it.
I wonder how much they pay. Reddit profits a lot from showing up on the top for many search queries. I very often do "whatever I'm looking for reddit" (for e.g. product reviews), since the reddit results often provide higher quality information than normal results.
Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)
