Yet Another Take On Call Recordings Storage: AWS S3

Lately there seems to be a lot of chatter around moving call recordings off your local Fusion servers. I decided to go with AWS S3 using the S3FS-Fuse Client. This client allows you to mount a S3 Bucket right to your file system. The fuse client documents do not recommend it for production, but for our case we pretty much move the recordings to S3 and only occasionally access them. The "correct" method would be to move them to S3 using Amazon's API as well as access them using their API. That will require a lot of changes to Fusion's source code, so I'm using the poor man's method.

A few other thoughts before we get started. Since AWS charges you for reads and writes, I wanted to keep all my transactions to a minimum. That kept me from using commands like "find" or using rsync. I just used a simple 'mv' command to move the files from the default directory to the S3 mount. Also, I decided to record to the default directory on local disk, then move the files at night. Recording directly to AWS just didn't seem wise.

I wanted to make sure that I verified every file was moved correctly before I updated the database. Therefore, I'm using a horribly inefficient method which is "Move a file, update the database, move a file, update the database." It takes a bit longer and touches the database a lot more. I had to add some indexes to the v_call_recordings and v_xml_cdr tables. The big ones were 'record_path' and 'record_name' in v_xml_cdr. I have a couple of million records in those tables and the script caused my CPU to go up to 50%. Once I added the indexes it was only around 3%.

Lastly, this information is for version 4.5+ (currently Master) and above of FusionPBX. If you are on 4.4, then you will probably need to make some adjustments to the php scripts.

And now for fun stuff...

Install the S3FS-Fuse Client:
Create a bucket called "voiceprod" in the AWS console.

Create the Directory
Code:
mkdir -p /var/s3/recordings
Add a script to mount the s3 bucket:
Code:
mkdir /etc/s3
cd /etc/s3
touch mount_bucket.sh
echo "export AWSACCESSKEYID=zzzMyAccessKeyzzz" >> mount_bucket.sh
echo "export AWSSECRETACCESSKEY=zzzSecretAccessKeyzzz" >> mount_bucket.sh
echo "/usr/bin/s3fs voiceprod /var/s3/recordings  -o allow_other" >> mount_bucket.sh
chmod +x mount_bucket.sh
Add to crontab on all servers to mount the bucket at boot:
Code:
@reboot /bin/sh /etc/s3/mount_bucket.sh
Add the Scripts below to /var/www/fusionpbx/call_recordings/
call_recording_archive_move.php

PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    $new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    $default_path = $_SESSION['switch']['recordings']['dir'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '1 hour' \n";
    $sql .= "and call_recording_path like '" . $default_path . "%' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";

//add the temporary permission
    $p = new permissions;
    $p->add("call_recording_add", "temp");
    $p->add("call_recording_edit", "temp");
    $p->add("xml_cdr_add", "temp");
    $p->add("xml_cdr_edit", "temp");
                
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $update_path = str_replace($default_path,$new_path,$row[call_recording_path]);
                $cmd = "mkdir -p " . $update_path;
                exec($cmd, $output, $return);

                $cmd = "mv " . $row[call_recording_path] . "/" . $row[call_recording_name] . " " . $update_path . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                if ($return) {
                    //echo "mv failed for ". $row[call_recording_path] . "/" . $row[call_recording_name];
                    continue;
                }

                $cmd = "chown www-data:www-data " . $update_path . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);

                $sql = "UPDATE v_call_recordings \n";
                $sql .= "SET call_recording_path = '".$update_path."' \n";
                $sql .= "where call_recording_uuid = '".$row[call_recording_uuid]."' \n";
                $db->exec(check_sql($sql));
                unset($sql);
                
                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = '".$update_path."' \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $sql .= "and record_path = '".$row[call_recording_path]."' \n";
                $db->exec(check_sql($sql));
                unset($sql);
            
        }
    }
    
//remove the temporary permission
    $p->delete("call_recording_add", "temp");
    $p->delete("call_recording_edit", "temp");
    $p->delete("xml_cdr_add", "temp");
    $p->delete("xml_cdr_edit", "temp");   
                
?>
call_recording_archive_delete.php
PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    //$new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    //$default_path = $_SESSION['switch']['recordings']['dir'];
    $archive_days  = $_SESSION['recordings']['archive_days']['text'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '" . $archive_days ." days' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";
    
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $cmd = "rm " . $row[call_recording_path] . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                //if ($return) {
                //    break;
                //}


            //delete from the database
                $sql = "DELETE FROM v_call_recordings ";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $sql .= "and record_path = '".$row[call_recording_path]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);

                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = null \n";
                $sql .= "AND record_name = null \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
        }
    }
    
?>
Change the owner to www-data
chown -R www-data:www-data /var/www/fusionpbx

Add the following to Default Settings in Fusion:
Category: Recordings
Subcategory : Type : Value : Enabled : Description
archive_days : text : 90 : True : Days to keep recordings
archive_path : dir : /var/s3/recordings : True

Add to crontab on one server:
(If you have a cluster, just run the script on one server. )
Code:
#Move Call Recordings to S3 Bucket
0 * * * * cd /var/www/fusionpbx; /usr/bin/php /var/www/fusionpbx/app/call_recordings/call_recording_archive_move.php
#Delete Old Call Recordings
0 3 * * * cd /var/www/fusionpbx; /usr/bin/php /var/www/fusionpbx/app/call_recordings/call_recording_archive_delete.php >/dev/null 2>&1
 
The script to delete recordings had some wrong column names. This is the updated version.

PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    //$new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    //$default_path = $_SESSION['switch']['recordings']['dir'];
    $archive_days  = $_SESSION['recordings']['archive_days']['text'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '" . $archive_days ." days' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";
    
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $cmd = "rm " . $row[call_recording_path] . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                //if ($return) {
                //    break;
                //}


            //delete from the database
                $sql = "DELETE FROM v_call_recordings ";
                $sql .= "WHERE call_recording_name = '".$row[call_recording_name]."' \n";
                $sql .= "and call_recording_path = '".$row[call_recording_path]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
                
                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = null \n";
                $sql .= "AND record_name = null \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
        }
    }
    
?>