Rugby is also a terrible sport to play casually with high impact tackling.
When you think about it even more casually, going down to the park with a few friends, rugby and american football are both hard to replicate. Sure you could pretend to both be quarterbacks and pass the ball back and forward but it's really nothing like the actual sport. Same with rugby which is mostly backwards passes.
Taking the American sports angle you can see why it's much more common to seeing friends playing basketball casually as getting a few friends together you can mostly replicate the game.
Most major american cities have one or more rugby clubs. It is great way to unwind after a long day at the office. You should definitely have good health insurance, and be prepared to accept some longer term consequences, but unlike football, there is a fairly active (albeit small) adult population that engages in the sport.
