I stick to my grandma too. I slept with her until I'm 19 years old.
I didn't know how my mum felt. I think she feel the same way as you.
But I think it is inevitable because my mum work most of the day. Comes home about 6.30pm, and I sleep at about 11pm. So there is only a few hours every week day.
If you work, I think spend the weekend fruitfully with your child.
I dont't recall my childhood with my mum...