Eh, even the small difference between Middletown and Concord, about 350 students, still gives you ~175 more boys to choose from for your team. That could make the difference of 2-3 great athletes that the other team doesn't get.
I think the main reason Concord has even been able to be competitive the last few years is because they've got the reputation as the "Football school" in that district thanks to the run they went on in D2 when all the kids that attend now were little and watching the local team win championships. Mount was able to get the best of them a couple years ago but otherwise they've dominated those local match-ups for years now. All the public school players want to choice into CHS up there, so even though they're smaller than the top dogs, they get players that other similar sized schools over the last few years(AI, Delcaste, etc) don't. That talent allows them to beat up on all those teams and then at least hang around with the Sallies/Middletown/Penns of the world.
Probably the same reason why Middletown still dominates Appo even though Appo is the bigger school by ~150-200 kids. Middletown has the reputation, Appo still hasn't put it all together yet. If they ever do, it could definitely upset the balance of power as more kids see Appo as an option.