It depends on where and which area I am shooting. Most of my hummingbird feeders are near a window and often, I can use the smaller lens. If I sit down in a chair for a while, I can be very close to the feeders and the hummers.
The 100-400 mm is definitely the workhorse. Depending on the time of day, I will shoot with the speed at 250+. Obviously, the more light means the faster I can shoot. I sometimes will bump up the ISO setting to allow for speeds over 1000 and stopping all of the action. You may have noticed, a majority of my shots are at very low apertures ..... whatever the meter will allow me to shoot. I have found that, for my style of photos, only the subject is in full focus and the backgrounds are blurred or out of focus. Someday, I will learn to work with subjects from here to infinity.