Hi congrats on getting pregnant.
I will suggest you to seek 2nd opinion.
Do take note that usually fetus/sac can only be seen during the ultrasound scan is around 6 weeks. So based on your own calculation, if fetus is only 3-4 weeks old, then I am not surprised that it is extremely small.
First thing first, do not fret over what the gynae has told you. The conceived/EDD date is not always accurate. Take care.