It's a bit confusing. In order to whitelist in fail2ban you need to put the IPs in the fail2ban configuration, for example:
And so on.
We manage our firewall elsewhere and have fail2ban disabled to avoid the issue that you had.
I believe the Access Control section is only for sofia configuration by default, fail2ban doesn't look at it. If you want to whitelist anything in fail2ban you have to do so within it's own configuration.