Is this just an excuse to break up with me for his babys mother?

I don't think it's got anything to do with the baby's mother. I think he's just not wanting to be with you anymore so I'd let him go!