I have fibroids during my first pregnancy and I delivered my dear daughter through natural birth. My gynae did not recommend to remove the fibroids mainly becos the position of my fibroids did not affect my baby or womb. He told me that he would rather wait till I have completed my family tree then he would remove rather then removing them now and cause a scar inside.
I am now preggy with my no. 2 and the fibroids are still there. They will grow bigger throughout your pregnancy but will also shrink after you have delivered. As long as you dont have any discomforts during your pregnanacy due to the fibroids, I think should be ok.