virgerl: i have switched my gynae halfway when i am entering into my 2nd trimester and didn't look back since. My advice is that if you are uncomfortable with him right now, its best to switch. Remember to get all your medical records from him so that the new gynae can refer to it. If there is a need to, some tests will be conducted. Do note, you do not need a referral letter from him to switch, you can just switch on your own.
Hope you find a nice gynae soon.