Maybe your baby does not like the herb smell and push you away? The food you eat?
You are lucky to have CL. I had to look after her myself at night...But it was ok... loved it..
Do you carry your baby often or more than the CL? Maybe your baby is used to her smell...try to carry more often for your baby to get used to your smell.. otherwise when your CL leaves, you will be in trouble.