High load average with low CCU

Hi everyone,
I'm using FreeSWITCH-1.6.20~64bit on Linux. We tested successfully on dev enviroment with 3000 CCU without any problem, but got issue on product server at 500 CCU.
- The scenario is very simple:
+ sipp send INVITE at 3000 CCU,
+ freeswitch play audio
+ after 10s, sipp send BYE to terminate the call

- On the dev server (16 core/20G RAM), the load avareage is about 0.2 at 3000 CCU (by SIPP)
1567474109940.png
1567474177251.png
- On the product server (16 core/32G RAM), the load average is getting higher very fast
1567474396596.png
And it cause some problem with freeswitch:
- The session is queued (it should be around 3000 CCU), the problem still happends at 500 CCU
1567474467518.png
- And there 's 5 seconds delayed between the time of receiving INVITE and processing. We do not see the problem with begining calls, but as time goes by, the delayed time is getting longer.
1567474841624.png

I think that we have missed something in the configuration of Freeswitch or OS but don't know exavtly what it is. We have already checked:
- ulimit
- max-session
- max call rate
- Ignore cdr-csv or xml

Please advise if you have any idea about this. I hereby attach the lua script running on freeswitch. Thank you very much.
 

Attached Files:

Hi DigitalDaz, the duration of each call is 10 seconds, so the call rate is about 300 calls per second (3000 CCU), and i think freeswitch is able to handle that. In the other hand, Freeswitch still working on dev enviroment at the same rate, but now on product server, it cannot handle even 100 CPS
 

DigitalDaz

Administrator
Staff member
Yep, I see that, have you tweaked php settings etc? I have never seen a freeswitch box do a sustained 100CPS and I have tried. Keep us posted on what you do get though as I'm very interested.
 
I think your testing method is not right. You are worried about CPU, however you need to be looking at your SIP retransmissions. Those are very high. From my understanding each SIP Profile runs as a single mod_sofia thread.

Slowly turn up your Calls Per Second rate in SIPP and look for retransmissions. Once you start seeing retransmissions you have become mod_sofia thread bound.

If you have devices registering to the same SIP Profile that will affect things too.

I typically think of it like "SIP Packets/Second/Profile".

This is the reason people use OpenSIPS/Kamalio for SIP and FreeSWITCH for RTP processing.
 
Thank you all , and very special thanks to KonradSC. As your suggestion, I have tested with differrent call rate level and it is running well at 2000 CCU. The problem is the high Calls Per Second in the previous testing, it caused the load average getting high and could not running at 2000 CCU, but now with the lower rate, it can running well . Thanks again.