This isn't based on anything except for intuition, take it or leave it :).

Like most things - eliminating the highest points of contact likely would suffice.

Daily habits are the ones that matter.

Are you drinking from a plastic water bottle everyday? Is your water filter plastic? Are you drinking coffee from a keurig or similar? Are you storing food in plastic containers? Are you heating food in plastic containers? Are you using plastic utencils? Are you using plastic lined pans? Are you buying food wrapped in plastic?

Etc etc.

It's a good starting place...then get your sperm tested, it's not prohibitively expensive.