My older man is saying he's too old for me! I love him, so confused, what do I do?

I'm seeing a man who is exactly 15 years older than me. (We have the same birthday believe it or not; FYI both Scorpios) It was love at first sight: I saw this gorgeous, dark haired, distinguished man sitting on a bench reading a book on campus, while I was a grad student and knew I had to meet him. He actually looked up at me as I ped but I was way too shy to say anything and kept walking. As luck would have it, we met a few months later at a luncheon and hit it off. I was on cloud nine! I didn't know his age until after we got to know each other. It has never been an issue for me. For the record he is in his early 50's and I'm in my late 30's. He's never been married and doesn't have children. He looks a lot younger though and takes very good care of himself. We have this amazing connection and feel he is my soul mate. Recently though, he saw me talking to this guy who is about my age and got very jealous. This guy was just an acquaintance I knew from grad school and nothing more. Now he's been making comments like why would a nice younger woman want an old man like him, or that he's old enough to be my father. (which is not true unless he was promiscuous 14 year old) We were talking about how his sister, who is two years younger than him wants her kids to get married soon because she wants grand kids and I made a comment that his nieces and nephews (who are in their 20's) are going to beat him to the altar. (we've talked about marriage before but nothing of a serious nature) He then got a sad look on his face and said that if he were to get married and have a kid right now, he would be in his 60's when the child would turn 10. He said everyone will think he's the grandpa. I told him there are lots of older parents out there and that when he's 60 chances are people will think he is younger because right now he could easily p for early 40's. We talked about kids again recently and now he says he doesn't think he wants to have kids because he is getting too old. Now he's been acting pive aggressive, being very closed and not communicating what is going on with him. He's been also acting very grouchy and did tell me I'm better off with someone my age. I love him so much and don't want to lose him. Yes I want to have kids but I rather be with him. How can I convince him that I don't care about his age. By the way, I am the first woman he's dated in years. The last one he dated when he was an undergrad in the late 70's. He's actually really shy and the girl broke his heart very bad so he shunned out love for years..until he met me. What we have is very special and to just let it go would be heartbreaking for both of us...