Higher CPU utilization after Debian/Freeswitch/PHP upgrade

[PrettyBrutalXperience-Guy]

Like (hopefully) all of us I've been busy with upgrades since last week. Specifically last Wednesday night after hours PHP rose from 7.x (shame on me) to 8.4, then over the weekend Debian graduated from 11 and/or 12 (previous upgrade attempt from 11 to 12 was partially incomplete), Freeswitch jumped to 1.11, and FusionPBX from 5.5 to 5.6. Although Postgresql 17 was installed seems it's empty and idle with 13 still being used, found good member documentation to use over the weekend to move data over and point to 17.

After the OS and Freeswitch upgrades this past weekend CPU utilization is elevated (ruled out PHP/FusionPBX being the culprit, Freeswitch appears to be the hungry one). Wondering if anyone else has experienced the same and what they may have done to resolve. May have lost previous tuning/optimizations during the upgrade so I'll certainly be scouting the public and member documentation over the weekend, but figured can't hurt to ask here after Super Intelligence let me down, perhaps a human may know of a default value changing which could explain it.

Dashboard panels clockwise from top left are histograms showing Registration count, Idle CPU, Disk use percentage, call setups per second at the moment of data collection, monitored systemctl service, scrape targets, bridged calls, and failed calls. The dashboards from left to right are 12 hour windows from: 2 Wednesdays ago (old everything), last Wednesday (web UI was 'offline'), last Thursday (PHP upgraded to 8.4), and finally this Wednesday (everything at latestish).



In case of curiosity the UI is grafana, datasource prometheus, and metrics from prometheus (targets), node exporter (disk space, critical services) and freeswitch exporter (the rest). Moved the UI and datasource to a dedicated host after a few self inflicted incidents when everything was running on the same tiny VM as Fusion/Freeswitch.

 

[PrettyBrutalXperience-Guy]

Upgraded to postgres 17 then noticed message_service causing the database to use excessive CPU, disabled it and message_events for some relief and next I'll look into fax as well as bad routing (the spike around noon was correlated with some infinite loop routing that eventually failed).

 

[markjcrane]

The app/messages message_queue and message_events feature are not as efficient as it should be it could cause a CPU spike.

Event Guard is more efficient on recent versions but you do need to make sure that when it triggers to block something that it actually is being blocked by iptables which these days is really a shim for nftables. You can tell if the firewall is not working if it blocks the same IP address over and over again.

 

[PrettyBrutalXperience-Guy]

Thanks, Event Gaurd is one of my favorite features as I’m always looking up how to unban with fail2ban and jailed myself today (nginx-dos)! I remember having a similar issue with messages in the past and following the member documentation helped significantly (db indexes and troubleshooting guide). Also correlated faxes to freeswitch owned elevated CPU which may be a result of my botching the freeswitch upgrade (mixed packages and source install), but it may just be more fax volume as I was only looking at call count on graphs which don’t show how many of those are faxes vs voice, maybe users send/receive more fax at the end of the month. Most (all) my problems seem to be self inflicted by either not following instructions properly or going off script and forgetting about my nonstandard modifications. Knocking on wood the platform without my chaos is quite solid and this forum is a most reliable place for keeping notes as I seem to forget easily and loose what I save locally!

 

[markjcrane]

I have noticed this on a few servers. If iptables is blocking are very large list of IP addresses it can affect the CPU.
Easily solved by clearing out the block list once in a while and let it rebuild again.

Clear iptables

iptables -F sip-auth-fail
iptables -F sip-auth-ip

Clear event guard logs

su postgresql
psql fusionpbx
DELETE FROM v_event_guard_logs;