Personally, I think this varies from baby to baby. I latch on Eva 100% from the beginning and she can go on for three hours between feeds and five hours at night (sometimes more as well). During growth spurts, she'll want to feed every 1.5 to 2 hours.
A lot of old people think that when a baby cries and fusses, the baby is hungry but that's not true - especially right after a feed. Babies cry because of discomfort, dirty diaper, wanting to be burped, night-day transition, just want to be carried and so forth.
Have you considered that maybe it's some thing in your diet that he doesn't like?
But really, if you decide to BF, then stick to it. BFing is not easy - I've been doing it for over six months - no bottles on top of that - and am still doing it; I've been through cracked nipples, blocked ducts, engorged boobs, distractable/fussy baby, and now teething, and I never ever think about giving up. You're doing this for your baby NOT for yourself. The way I see it is that I never had a choice to give up in the first place.