you did good already.. so please dont't feel bad about having to give FM!
I think you've traced why your supply is dropping already. One solution if you want to build up your supply is to stay home the whole weekend and feed as often as he'd let you. Remember demand = supply.. so the more demand, the more supply. Then pump as often as you can at work... (new 'toilet' break? *grin*)
other than that... getting more rest and trying to minimise stress would be the other key factor. dont't know if you are co-sleeping with your toddler? Co-sleeping for me was the ideal way to get a good nite's sleep. You see, you'll surface from sleep just as the toddler wakes up (this may take some time to establish), offer the breast, let him latch and drift off to sleep together again.
Hope these suggestions help...