Gateway failure alert script for Debian 8

Status
Not open for further replies.

EasyBB

Active Member
Oct 23, 2016
240
33
28
Australia
When something goes wrong with your provider (server issues, DNS issues etc), the corresponding gateway in FusionPBX fails and you won't probably notice this for a while until someone contacts you through other means. This has happened to me on many occasions, so I wrote this script to check gateway status using CRON and send out email alerts in case there's a failure. The script would also attempt to restart FreeSWITCH if it is not running.

gwfailalert.php
PHP:
<?php
/* Author: EasyBB on fusionpbxforums.
 * Permission is hereby granted to anyone to use this code for any purpose
 * they see fit. This code comes with no warranties of any kind,
 * implied or expressed and this author is not responsible for any damage
 * or loss resulting from using this code directly or indirectly.
 * You may edit or modify the code as you like, again, at your own risk.
 */

//************************ gwfailalert.php *****************************
// Last edited: 2016 11 01.
// Tested on: Debian 8 Jessie.
// Dependencies: mail transfer agent (MTA), mailutils, php5+
//
// This PHP script is to be used with FreeSWITCH/ FusionPBX server to send
// email alert upon Gateway failure.
//
// You'll need a working mail transfer agent (ssmtp, sendmail etc) running
// on the FusionPBX server. https://help.ubuntu.com/community/EmailAlerts
// ssmtp is the simplest of all. Remember to disable other MTAs if you wish
// to go with ssmtp:
// http://linuxpitstop.com/install-ssmtp-to-send-emails-to-gmail-and-office3655/
//
// Please note that the script needs to be invoked from CRON.
// Recommended to set 30 min interval.
// e.g. */30 * * * * /usr/bin/php /root/gwfailalert.php > /dev/null 2>&1
//**********************************************************************

/*******************************************
 ******* EDIT BELOW VALUES TO SUIT *********
 ******************************************/

// Enter the email to receive alerts (mandatory)
$mailto = "your_to_email@gmail.com";

// Enter the from email address or leave it as it is.
$mailfrom = "your_from_email@gmail.com";

// Enter a friendly sender name or leave it as it is.
$fromname = "FusionPBX";

// Enter an email subject or leave it as it is.
$mailsub = "FusionPBX failure alert!";

// Location of fs_cli command.
$fscli = "/usr/bin/fs_cli";
//$fscli = "/usr/local/fs_cli";

/*******************************************
 ****** DO NOT EDIT BELOW THIS LINE *******
 ******************************************/
function checkFailed(){
    global $fscli;
    $checkgw = trim(shell_exec($fscli . ' -x "sofia profile external gwlist down"'));
    if($checkgw == "-ERR no reply"){
        return false;
    }else{
        return explode(" ", $checkgw);
    }
}

function sendMail($msg){
    global $fromname, $mailfrom, $mailsub, $mailto;
    // Format from name
    $from = $fromname . "\<" . $mailfrom . "\>";
    // Email command structure
    $recipe = 'echo "%s" | mail -a "Content-Type: text/html" -s "%s" -aFrom:%s "%s"';
    // Format email command
    $cooked = sprintf($recipe,$msg,$mailsub,$from,$mailto);
    // Send email
    shell_exec($cooked);
}

function checkFS(){
    $fs = trim(shell_exec('pidof "freeswitch"'));
    if(! strlen($fs)){
        return false;
    }else{
        return true;
    }
}

$msg = "";

if(! checkFS()){
    $msg .= "FreeSWITCH isn't running, trying to restart... <br />";
    $msg .= shell_exec('systemctl restart freeswitch.service');
    sleep(5);
    if(! checkFS()){
        $msg .= "<br /> FreeSWITCH wouldn't start, ATTEND IMMEDIATELY; exiting script... <br />";
        sendMail($msg);
        exit;
    }else{
        $msg .= "<br /> FreeSWITCH is up and running... <br />";
    }
}

for($i = 1; $i <= 2; $i++){
    if(checkFailed()){
        foreach ($failed = checkFailed() as $gw){
            shell_exec($fscli . ' -x "sofia profile external killgw ' . $gw .'"');
        }
        sleep(2);
        shell_exec($fscli . ' -x "sofia profile external rescan"');
    } 
    if($i < 2){
        sleep(5);
    }else{
        sleep(1);
    }
}

if(checkFailed()){

    $xmlstatus = shell_exec($fscli . ' -x "sofia xmlstatus"');
    $xml = simplexml_load_string($xmlstatus);
    $msg .= "<br /> FusionPBX Status: <br /> <br />";

    // Get gateway status
    foreach($xml-> gateway as $element){
        $msg .= $element->type . " : " . $element->data . " : " . $element->state . "<br />";
    }
    $msg .= "<br />";

    // Get SIP profile status
    foreach($xml -> profile as $element){
        $msg .= $element->type . " : " . $element->name . " : " . $element->state . "<br />";
    }
    $msg .= "<br />";
  
    // Get ALIAS status
    foreach($xml -> alias as $element){
        $msg .= $element->type . " : " . $element->name . " : " . $element->data . " : " . $element->state . "<br />";
    }

    sendMail($msg);
}

?>

Note:- You'll need an MTA running on your system for the script to send out emails; I use sSMTP which is very easy to set up. Popular MTAs are Sendmail, Postfix and Exim in no particular order.
 
Last edited:

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,043
565
113
I assume this only works for registered gateways? Regardless, I'm sure it will be very useful :)
 

EasyBB

Active Member
Oct 23, 2016
240
33
28
Australia
I assume this only works for registered gateways?
This is correct, the script uses output of the command: "sofia profile external gwlist down"
Sometimes gateways fail due to internal/external network issues and then won't recover automatically. I've seen this quite a few times with DIDLogic when they do maintenance overnight.
I've also had instances where FreeSWITCH was killed by misbehaving modules etc; this hasn't happened on FS 1.6 yet so that's good. The script restarts FS in such cases.

moteus has an Lua script that works with events and it can send warnings in realtime. It won't restart FS or gateways though. The script will die if FS dies.
https://gist.github.com/moteus/b0623b08653333a267cbcde73971b7ce
Modify the email address on line 11.
 

EasyBB

Active Member
Oct 23, 2016
240
33
28
Australia
Last night I had a gateway failure from which the current script couldn't recover. So I modified the script so it kills the gateway after a couple of attempts, then restarts it next time CRON runs the script.

PHP:
<?php
/* Author: EasyBB on fusionpbxforums.
 * Permission is hereby granted to anyone to use this code for any purpose
 * they see fit. This code comes with no warranties of any kind,
 * implied or expressive and this author is not responsible for any damage
 * or loss resulting from using this code directly or indirectly.
 * You may edit or modify the code as you like, again, at your own risk.
 */

//************************ gwfailalert.php *****************************
// Last edited: 2017 01 01.
// Tested on: Debian 8 Jessie.
// Dependencies: mail transfer agent (MTA), mailutils, php5+
//
// This PHP script is to be used with FreeSWITCH/ FusionPBX server to send
// email alert upon Gateway failure.
//
// You'll need a working mail transfer agent (ssmtp, sendmail etc) running
// on the FusionPBX server. https://help.ubuntu.com/community/EmailAlerts
// ssmtp is the simplest of all. Remember to disable other MTAs if you wish
// to go with ssmtp:
// http://linuxpitstop.com/install-ssmtp-to-send-emails-to-gmail-and-office3655/
//
// Please note that the script needs to be invoked from CRON.
// Recommended to set 30 min interval.
// e.g. */30 * * * * /usr/bin/php /root/gwfailalert.php > /dev/null 2>&1
//**********************************************************************

/*******************************************
 ******* EDIT BELOW VALUES TO SUIT *********
 ******************************************/

// Enter the email to receive alerts (mandatory)
$mailto = "to-email@gmail.com";

// Enter the from email address or leave it as it is.
$mailfrom = "fusionpbx@gmail.com";

// Enter a friendly sender name or leave it as it is.
$fromname = "FusionPBX";

// Enter an email subject or leave it as it is.
$mailsub = "[ALERT] FusionPBX Failure!";

// Location of fs_cli command.
$fscli = "/usr/bin/fs_cli";
//$fscli = "/usr/local/fs_cli";

/*******************************************
 ****** DO NOT EDIT BELOW THIS LINE *******
 ******************************************/
function checkFailed(){
    global $fscli;
    shell_exec($fscli . ' -x "sofia profile external rescan"');
    sleep(5);
    $checkgw = trim(shell_exec($fscli . ' -x "sofia profile external gwlist down"'));
    if($checkgw == "-ERR no reply"){
        return false;
    }else{
        return explode(" ", $checkgw);
    }
}

function sendMail($msg){
    global $fromname, $mailfrom, $mailsub, $mailto;
    // Format from name
    $from = $fromname . "\<" . $mailfrom . "\>";
    // Email command line
    $recipe = 'echo "%s" | mail -a "Content-Type: text/html" -s "%s" -aFrom:%s "%s"';
    // Format email command
    $cooked = sprintf($recipe,$msg,$mailsub,$from,$mailto);
    // Send email
    shell_exec($cooked);
}

function checkFS(){
    $fs = trim(shell_exec('pidof "freeswitch"'));
    if(! strlen($fs)){
        return false;
    }else{
        return true;
    }
}

$msg = "";

if(! checkFS()){
    $msg .= "FreeSWITCH isn't running, trying a service restart... <br />";
    $msg .= shell_exec('systemctl restart freeswitch.service');
    sleep(5);
    if(! checkFS()){
        $msg .= "<br /> FreeSWITCH wouldn't start; exiting... <br />";
        sendMail($msg);
        exit;
    }else{
        $msg .= "<br /> FreeSWITCH is running... <br />";
    }
}

if($failed = checkFailed()){
    // Try to re-register the gateway twice
    for($i = 1; $i <= 2; $i++){
        foreach ($failed as $gw){
            shell_exec($fscli . ' -x "sofia profile external killgw ' . $gw .'"');
        }
        sleep(2);
        shell_exec($fscli . ' -x "sofia profile external rescan"');

        if($i < 2){
            sleep(5);
        }else{
            sleep(1);
        }

        if(checkFailed()){
            break;
        }
    }

    // Kill recalcitrant gateway(s) until the next run
    if($failed2 = checkFailed()){
        foreach ($failed2 as $gw){
            shell_exec($fscli . ' -x "sofia profile external killgw ' . $gw .'"');
        }
    }

    $xmlstatus = shell_exec($fscli . ' -x "sofia xmlstatus"');
    $xml = simplexml_load_string($xmlstatus);
    $msg .= "<br /> FusionPBX Status: <br /> <br />";

    // Get gateway status
    foreach($xml-> gateway as $element){
            $msg .= $element->type . " : " . $element->data . " : " . $element->state . "<br />";
            }
    $msg .= "<br />";

    // Get SIP profile status
    foreach($xml -> profile as $element){
            $msg .= $element->type . " : " . $element->name . " : " . $element->state . "<br />";
            }
    $msg .= "<br />";
    // Get ALIAS status
    foreach($xml -> alias as $element){
            $msg .= $element->type . " : " . $element->name . " : " . $element->data . " : " . $element->state . "<br />";
            }

    sendMail($msg);
}

?>
 

Guy Perera

New Member
May 3, 2017
5
0
1
46
Melbourne, Australia
Hi, I've been looking for a solution to monitor extensions similar to this method. In the event of an extension goes down, a script to pickup the absence of that extensions. The re-registration of an extension is set to 3600sec on all Polycom phones that I use in multiple locations.

Can the above script be modified to read the available extensions from freeswitch/fusionpbx and check the availability of that list extension's registration attempt from the log? If a registration attempt was not available (within 30mins log), means the extension is offline.

I have researched for many months, but couldn't find a favorable solution yet. But I have a feeling, a modification to this scrip is a promising solution. What I can't figure out is to how to read the created extensions. Any help or guidance is highly appreciated. Thank you. Guy
 

Guy Perera

New Member
May 3, 2017
5
0
1
46
Melbourne, Australia
Hi EasyBB. Thank you for the reply.

The registration of the extension drops when there are issues with internet or power. So the notification has to come from the server side.

My intention is to monitor each site and notify site personal when Internet or Power is down. Specially this is useful when you run security monitoring via voip line.

Thanks, Cheers.
 

EasyBB

Active Member
Oct 23, 2016
240
33
28
Australia
The registration of the extension drops when there are issues with internet or power. So the notification has to come from the server side.

My intention is to monitor each site and notify site personal when Internet or Power is down. Specially this is useful when you run security monitoring via voip line.

Since you've nicely explained your requirements, I thought I would just write a script for it:

http://www.pbxforums.com/threads/ex...itoring-script-with-email-alert.790/post-3244

Hope it helps :D
 

EasyBB

Active Member
Oct 23, 2016
240
33
28
Australia
My intention is to monitor each site and notify site personal when Internet or Power is down. Specially this is useful when you run security monitoring via voip line.

Thanks, Cheers.
I have added another version of the extension monitoring script in the new thread mentioned above. New script can take multiple domains and extesions under those domains.
Cheers.
 
Status
Not open for further replies.