Babies will take a few weeks to recognise people/places that they see only once a week or so. My dear daughter took quite awhile to get used to my mum and going to my mum's place. We bring her once or twice a week. She used to cry when carried by my mum last time and when we go to her house but now she just goes straight to play with the toys... haha... So dont't be too stressed. Babies will get it sorted out with time.