I didn't do this and no I dont't think you're selfish.
If you only latch on at night and dont't pump, you will get pretty engorged and uncomfy during the day. Also due to the drop in demand, your milk supply will also drop.
So if you plan to breastfeed for longer, you should try to pump some milk during working hours. You can surf around the forum as there're a few threads talking about pumping when back at work.
Also, you might want to try to start your baby on the bottle to get him used to it? Some babies may reject the bottle because it is unfamiliar to them.