You've never been to a bar/coffee house/grocery store/etc on a weekend? Lots of people work the weekend, often it is those who are paid less and have more rigid schedules. Also you run into religious days where some are not able to perform work.
Honestly it is a hard one to mandate as a single few hour window on a single day. Someone has to be working at hospitals for example.
It already is 'election week' in Australia. You can vote early, and you can postal-vote. The weekend vote is much easier to get the masses out voting, because most people don't have work (or school) to deal with on the weekend. At the last federal election I voted on the Wednesday beforehand, simply because I was passing by an early voting station.
And the queues at Australian voting stations on The Day aren't anywhere near as long as in the US. I've heard some horror stories where some people have had to wait 4+ hours (on a Tuesday) to vote in the US.