I have attended the Jazz Festival every year except one I think since 2003. The hotels are a little fuller than usual, and traffic is rougher than usual, but those issues are not really a big deal if you are dumping your car in a garage somewhere. The Grand Hyatt is the best place to stay since it is "Ground Zero" and features a balcony from which you can see two stages.
The only year I remember when things were a bit out of control was when Stevie Wonder was the headline performer in 2009, performing at a free outdoor concert on the main stage. I could not get near the stage but I did hear the performance from around 2 blocks away. The Montreal police did not seem well organized in terms of their crowd control efforts. It was a bit of chaos.
My favorite Jazz Festival moments include seeing Paul Simon's band warming up in around 2006, watching the band Escort perform as a Disco group in 2011 or 2012 (a performance I watched from the balcony adjacent to the swimming pool at the Grand Hyatt, where a balcony bar had been opened) and a Quebec High School Band Champion performing the Hawaii Five-O theme song (that was around 2004 or 2005, can't remember, but probably the greatest TV theme song of all time and they did a great job with it).