i heard that baby usually get used to someone if he is attached to her in the beginning. so i think he is used to the confinement nanny "touch". so you have to keep cuddling him, speak to him, let him get used to your voice, touch, smell etc. get your hubby to do that also, then your baby will also "know" the daddy too. else once you are "in", your hubby is "out".
how about checking with your mum for opinion ?