Just enjoy your baby & being a parent.
I work in special needs. In the past, there are quite a number of adopted kids who turned out to have special needs. The parent-child bond is the same, whether the child was adopted or not.
My mother was also adopted. Out of respect for her adoptive parents, she didn't contact her biological family until they passed away.
My advice is dont't let other people's comments affect you when things dont't go well. Parenting is a special joy with many challenges, whether the kids are adopted or not.