First of all congrats! We are in the same year although I have yet to turn 23! Hehehhe. Its true first trimester is the most fragile state thats why we have to be extra careful to ourself and dont't do things that will lead to miscarriage.
I was like you at first, very worried cause we just got married and furthermore we're getting a house and with all the expense that will be adding in, having a baby at this time is not appropriate either but when I hear the baby's heartbeat and see the scan for the very first time, I know I love him/her even before I met him/her. So therefore from there the happiness bloom and hubby is also excited!
When you hold the baby in your arms, who knows your maternal instincts kicked in and we will become a changed person. So dont't worry much, just enjoy the pregnancy! The more you think about it the more you will feel depressed. Pregnancy is supposed to be a happy process!