I used to fly every 2 weeks for my job. After I got pregnant I requested to be grounded. After I gave birth, I requested not to fly at all.
I feel it is not wrong to want to fly around to see the world and learn more. It is good for your career prospects in the future as well. I agree that no matter how you look at it, somebody loses. Because we all only have 24hrs, if we choose one thing over another, something else would need to be compromised and you'd need to know if you can live with that.
For me, I gave up my flying because I am sure of what I want. I want to take care of my son and I have decided to leave the climbing of the corporate ladder to my husband.
We all have our preferences so I wouldn't say that your hubby is selfish or you are too career-minded. The grass will always look greener on the other side so you really need to decide what you are willing to compromise.
Frankly speaking, 3-4 times a yr is not alot of flying compared to mine. I was flying twice every month and I will be overseas for 1 week each time. But because I chose to forgo flying, my bosses understand, but my promotion has been shelved. I am fine with that because I have already decided in my heart what I want.