Usually pacifier is not recommended for NB who're breastfed because people fear the chances of nipple confusion. But since your he is already bottle fed at night, it might be safe to introduce the pacifier.
My son was doing that initially as well. What I ended up doing was making sure he had a full feed before I allowed him to fall back asleep. Sleeping is impt for NBs to grow and I felt that if he didn't feed well, he wouldn't sleep well, which was true in my case.
Roughly what is the interval which you feed your baby? Usually, it's every 3 hours, and I encourage that at every 3 hours that you try to ensure your baby gets a full feed. If he falls asleep, you can try to wake him up. What I do to wake my baby up is change his diaper, try to burp him etc etc. I never introduced the pacifier to my baby.
Do note that during growth spurts (one of which happens at around 3 weeks) they will feed very frequently, like almost every hr and this will last a few days.
There're many methods out there used by many mothers and there's no one-size-fits-all solution. Try to get to know your baby's "pattern" and then try a few solutions to see which is best for your baby.