Yet Another Take On Call Recordings Storage: AWS S3

Status
Not open for further replies.

KonradSC

Active Member
Mar 10, 2017
166
98
28
Lately there seems to be a lot of chatter around moving call recordings off your local Fusion servers. I decided to go with AWS S3 using the S3FS-Fuse Client. This client allows you to mount a S3 Bucket right to your file system. The fuse client documents do not recommend it for production, but for our case we pretty much move the recordings to S3 and only occasionally access them. The "correct" method would be to move them to S3 using Amazon's API as well as access them using their API. That will require a lot of changes to Fusion's source code, so I'm using the poor man's method.

A few other thoughts before we get started. Since AWS charges you for reads and writes, I wanted to keep all my transactions to a minimum. That kept me from using commands like "find" or using rsync. I just used a simple 'mv' command to move the files from the default directory to the S3 mount. Also, I decided to record to the default directory on local disk, then move the files at night. Recording directly to AWS just didn't seem wise.

I wanted to make sure that I verified every file was moved correctly before I updated the database. Therefore, I'm using a horribly inefficient method which is "Move a file, update the database, move a file, update the database." It takes a bit longer and touches the database a lot more. I had to add some indexes to the v_call_recordings and v_xml_cdr tables. The big ones were 'record_path' and 'record_name' in v_xml_cdr. I have a couple of million records in those tables and the script caused my CPU to go up to 50%. Once I added the indexes it was only around 3%.

Lastly, this information is for version 4.5+ (currently Master) and above of FusionPBX. If you are on 4.4, then you will probably need to make some adjustments to the php scripts.

And now for fun stuff...

Install the S3FS-Fuse Client:
Create a bucket called "voiceprod" in the AWS console.

Create the Directory
Code:
mkdir -p /var/s3/recordings

Add a script to mount the s3 bucket:
Code:
mkdir /etc/s3
cd /etc/s3
touch mount_bucket.sh
echo "export AWSACCESSKEYID=zzzMyAccessKeyzzz" >> mount_bucket.sh
echo "export AWSSECRETACCESSKEY=zzzSecretAccessKeyzzz" >> mount_bucket.sh
echo "/usr/bin/s3fs voiceprod /var/s3/recordings  -o allow_other" >> mount_bucket.sh
chmod +x mount_bucket.sh

Add to crontab on all servers to mount the bucket at boot:
Code:
@reboot /bin/sh /etc/s3/mount_bucket.sh

Add the Scripts below to /var/www/fusionpbx/call_recordings/
call_recording_archive_move.php

PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    $new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    $default_path = $_SESSION['switch']['recordings']['dir'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '1 hour' \n";
    $sql .= "and call_recording_path like '" . $default_path . "%' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";

//add the temporary permission
    $p = new permissions;
    $p->add("call_recording_add", "temp");
    $p->add("call_recording_edit", "temp");
    $p->add("xml_cdr_add", "temp");
    $p->add("xml_cdr_edit", "temp");
                
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $update_path = str_replace($default_path,$new_path,$row[call_recording_path]);
                $cmd = "mkdir -p " . $update_path;
                exec($cmd, $output, $return);

                $cmd = "mv " . $row[call_recording_path] . "/" . $row[call_recording_name] . " " . $update_path . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                if ($return) {
                    //echo "mv failed for ". $row[call_recording_path] . "/" . $row[call_recording_name];
                    continue;
                }

                $cmd = "chown www-data:www-data " . $update_path . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);

                $sql = "UPDATE v_call_recordings \n";
                $sql .= "SET call_recording_path = '".$update_path."' \n";
                $sql .= "where call_recording_uuid = '".$row[call_recording_uuid]."' \n";
                $db->exec(check_sql($sql));
                unset($sql);
                
                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = '".$update_path."' \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $sql .= "and record_path = '".$row[call_recording_path]."' \n";
                $db->exec(check_sql($sql));
                unset($sql);
            
        }
    }
    
//remove the temporary permission
    $p->delete("call_recording_add", "temp");
    $p->delete("call_recording_edit", "temp");
    $p->delete("xml_cdr_add", "temp");
    $p->delete("xml_cdr_edit", "temp");   
                
?>

call_recording_archive_delete.php
PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    //$new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    //$default_path = $_SESSION['switch']['recordings']['dir'];
    $archive_days  = $_SESSION['recordings']['archive_days']['text'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '" . $archive_days ." days' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";
    
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $cmd = "rm " . $row[call_recording_path] . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                //if ($return) {
                //    break;
                //}


            //delete from the database
                $sql = "DELETE FROM v_call_recordings ";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $sql .= "and record_path = '".$row[call_recording_path]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);

                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = null \n";
                $sql .= "AND record_name = null \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
        }
    }
    
?>

Change the owner to www-data
chown -R www-data:www-data /var/www/fusionpbx

Add the following to Default Settings in Fusion:
Category: Recordings
Subcategory : Type : Value : Enabled : Description
archive_days : text : 90 : True : Days to keep recordings
archive_path : dir : /var/s3/recordings : True

Add to crontab on one server:
(If you have a cluster, just run the script on one server. )
Code:
#Move Call Recordings to S3 Bucket
0 * * * * cd /var/www/fusionpbx; /usr/bin/php /var/www/fusionpbx/app/call_recordings/call_recording_archive_move.php
#Delete Old Call Recordings
0 3 * * * cd /var/www/fusionpbx; /usr/bin/php /var/www/fusionpbx/app/call_recordings/call_recording_archive_delete.php >/dev/null 2>&1
 

KonradSC

Active Member
Mar 10, 2017
166
98
28
The script to delete recordings had some wrong column names. This is the updated version.

PHP:
<?php
/*
    FusionPBX
    Version: MPL 1.1

    The contents of this file are subject to the Mozilla Public License Version
    1.1 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at
    http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis,
    WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
    for the specific language governing rights and limitations under the
    License.

    The Original Code is FusionPBX

    The Initial Developer of the Original Code is
    Mark J Crane <markjcrane@fusionpbx.com>
    Portions created by the Initial Developer are Copyright (C) 2016
    the Initial Developer. All Rights Reserved.

    Contributor(s):
    Mark J Crane <markjcrane@fusionpbx.com>
    KonradSC <konrd@yahoo.com>
*/

//check the permission
    if(defined('STDIN')) {
        $document_root = str_replace("\\", "/", $_SERVER["PHP_SELF"]);
        preg_match("/^(.*)\/app\/.*$/", $document_root, $matches);
        $document_root = $matches[1];
        set_include_path($document_root);
        $_SERVER["DOCUMENT_ROOT"] = $document_root;
        require_once "resources/require.php";
    }
    else {
        include "root.php";
        require_once "resources/require.php";
        require_once "resources/pdo.php";
    }

//increase limits
    set_time_limit(3600);
    ini_set('memory_limit', '256M');
    ini_set("precision", 6);
    
//set some variable
    //$new_path = "/var/s3/recordings";
    //$new_path = $_SESSION['recordings']['archive_path']['dir'];
    //$default_path = "/var/lib/freeswitch/recordings";
    //$default_path = $_SESSION['switch']['recordings']['dir'];
    $archive_days  = $_SESSION['recordings']['archive_days']['text'];

//lookup the destinations
    $sql = "SELECT call_recording_uuid, call_recording_path, call_recording_name  \n";
    $sql .= "FROM v_call_recordings \n";
    $sql .= "WHERE call_recording_date < NOW() - INTERVAL '" . $archive_days ." days' \n";
    $database = new database;
    $database->select($sql);
    $recording_array = $database->result;
    //echo "recording_array =" . print_r($recording_array,true) ."\n";
    
    if (is_array($recording_array)) {
        $i=0;
        foreach($recording_array as $key => $row) {   
            //create the directory and the move file
                $cmd = "rm " . $row[call_recording_path] . "/" . $row[call_recording_name];
                exec($cmd, $output, $return);
                
                //if ($return) {
                //    break;
                //}


            //delete from the database
                $sql = "DELETE FROM v_call_recordings ";
                $sql .= "WHERE call_recording_name = '".$row[call_recording_name]."' \n";
                $sql .= "and call_recording_path = '".$row[call_recording_path]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
                
                $sql = "UPDATE v_xml_cdr \n";
                $sql .= "SET record_path = null \n";
                $sql .= "AND record_name = null \n";
                $sql .= "WHERE record_name = '".$row[call_recording_name]."' \n";
                $prep_statement = $db->prepare(check_sql($sql));
                $prep_statement->execute();
                unset($prep_statement, $sql);
        }
    }
    
?>
 
  • Like
Reactions: alexhir

wesleymininel

New Member
Feb 26, 2020
1
0
1
43
Good afternoon, I have been using call_recording_archive_move.php for some time and recently it stopped sending files to AWS S3. When I enable to show the recording_array variable (echo "recording_array =". Print_r ($ recording_array, true). "\ N";) it doesn't return anything, I did a new installation of FusionPBX to test the script and the same happens thing, was there any change in FusionPBX that needs to update the PHP script?
 

KonradSC

Active Member
Mar 10, 2017
166
98
28
Interesting. It still works for me. I'm on a newish version of Master branch. Perhaps grab the query at the "SELECT call_recording_uuid, call_recording_path, call_recording_name..." query and try that directly on the database.
 

John

Member
Jan 23, 2017
97
8
8
Why don't you just mount it as a directory and let the Fusion deal with it as a local directory, by changing call recording and voicemail location in default or domain level settings? If you go this long way for cost reduction, there are many s3 compatible and S3FS certified storages out there. It does not have to be AWS itself. Please enlighten me, if my comment is incorrect. I don't mind. I would love to learn.
 

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,038
556
113
I think others are using mod_http_cache. From what I read years ago, this should work quite well if handled correctly.

Historically, if I remember rightly, the problem in the past with S3 was that we didn't store the recording path in the db. The way the CDR page 'found' the recording and displayed the play/download button was to check the file system for numerous extensions for the uuid eg uuid.wav,uuid.mp3, uuid.ogg etc/etc. This happened for every CDR row and made using S3 like glue. I have never looked at it since but would be really interested to see how it is being done in 2021 :)
 

KonradSC

Active Member
Mar 10, 2017
166
98
28
John, that's a good question. My main reason for not mounting the recordings directory directly on S3FS has to do with reliability. If I lose the connection to Amazon or the connection is slow I don't want calls to be affected.
 
Status
Not open for further replies.