This is true - I rarely get asked for ID when entering Canada, but always am when returning to the US. I started carrying my passport after 9-11 just because I thought it would be easier. US Customs only asks for ID, though, not a passport.
Even though Canadian Customs is less likely to ask...