It probably has nothing to do with your looks.. it probably has to do with your attitude surrounding your looks. If you are constantly primping in the mirror, showing your muscles off, talking about working out, always wearing tight clothes, etc. , then you are communicating to her on a subcontious level that you are superficial, and/or are overcompensating for something with your body.
in my experience, women care little about a mans' body (to a certain degree. please dont quote me and say 'yeah but what if im 500 pounds?-to a certain degree). BUT if a woman senses insecurity right off the bat, she will run.
I saw a buff guy walking on St. Catherine street last week. It must have been plus 4 degees at the most. He was wearing a t-shirt , carrying his jacket in his hands. was he walking with a girl? nope. were women stopping to admire him? I don't think so. they were laughing at him, at his lame attempt to impress them with something superficial.
my point to you is this: im not saying you do anything like this. however, I don't see any woman leaving a guy SIMPLY because his body is TOO GOOD. It either means there's something very wrong emotionally with this woman, or there is something else to it. And if she's hinting to you that it has to do with your body, that may be it.