Hem... There is PerSourcePenaltyExemptList to whitelist specific hosts so while I agree it might create a bit more complexity (not much different than handling fail2ban and co) it's not "blocking" like described.