Similar situation with me. Drive up about 4 times per year and always cross in Vermont. I was, however, hassled a couple of times by Canadian border guards. It is unpleasant (and may last up to one hour or so), but so far they let me through both times. Last time they demanded (among other things) that I hand over my phone and messed around with it for a while (not in my presence), screwed up some of my settings as a result. And of course asked the same standard set of questions a few times over -- probably trying to catch you in some inconsistencies.
Same thing (also a couple of times) happened with Americans on my way home. A nuisance, but nothing really terrible.
I feel sometimes they are (both Canadians are Americans) just bored out of their wits and do that as some sort of entertainment. Or maybe they just have a daily quota of how many decent people they must piss off, who knows. Just stick to the same standard spiel (brief clear answers, ideally yes or no) and never lose your temper, that is my way of dealing with them.