first you have to be very sure of what you want. is it really affirmative in your heart that you can do without him?
start by communicating this with him. if it does not work, start cutting off all your forms of communication. resort to changing numbers etc.
you mention he comes over. you mean to your place?
try telling him (not quarreling) that you do not want him to come over anymore.
do you have other places to go to? like perhaps parents' place?
if not try to insist your way that you do not want him over anymore.
but be careful, he is of 40 years of age. he might want his way. he might be overbearing.
if necessary, take legal action. might make him hate you if you do.
but if you really want to end this relationship with him, you got to be hard hearted.
try to explain things to your daughter.
actually he might be going to you as a form of relief from his family, like maybe his wife.
because he finds that being with you might not require him to be burden with any form of responsibility.