If it wasn't for duplicates a lot of posts would be missed. I don't watch posts 24/7 so a lot of times its the duplicate post i get to see not the original. Although duplicates that appear on the same page can be annoying. So the algorithm should prevent duplicates if they are within 40 or so posts?

The dupematching is for the first few hours and the the url matching itself is not terribly strict and trivial to circumvent (by design).