I usually write back that I do not feel I am the right person for him...that way it takes the focus out of him,I do not reject him per se, I try to convey that he needs someone else ...
This is a good way to respond. It clearly gives sent the message loud and clear but in a diplomatic way.