HA Across the Country

Status
Not open for further replies.

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Hey guys. It will be a bit till I get basic configuration going, in terms of time conditions, queues, shared VM, IVRs, broadcast, SMS, FAX, and whatever else is needed. However, I will want to implement High Availability. As of now, the organizations I am looking to host (our family business, a local nonprofit, and a family friend's business) are in Wichita, KS.
Now that I have come across BulkVS, I am wanting to host the "primary" service location on my servers in Dallas. I would setup a 3rd server in the New York area. I may look at clustering two servers (VMs) in Dallas for load balancing and high availability and the server in NY for the "fail-over" functionality.

Dallas is not only closer, but I have two server colo'd within the Infomart. There is not immediate hosting cost I would face starting out.

How likely is this to function efficiently?
 
Last edited:

sudoRmRf

New Member
May 31, 2019
17
3
3
33
FusionPBX can be setup HA fairly easily. You can use a master-master configuration with Postgres BDR and use any number of filesystem sync tools for your files. Most of the process is actually built into the Fusion install process you just have to know where to look. Note that all this was pulled from my documentation I wrote as I worked through this setup, so there may be better way of doing this but this seems to work. Additionally you may need to make tweaks places for it to work for you. First in download the install files with
wget -O - https://raw.githubusercontent.com/fusionpbx/fusionpbx-install.sh/master/debian/pre-install.sh | sh; and then edit the config file /usr/src/fusionpbx-install.sh/debian/resources/config.sh and edit your database settings.
Bash:
# Database Settings
database_password=<DB Password>           # random or a custom value (safe characters A-Z, a-z, 0-9)
database_repo=2ndquadrant                 # PostgreSQL official, system, 2ndquadrant
database_version=9.4                      # requires repo official
database_host=<NodeIP>                    # hostname or IP address
database_port=5432                        # port number
database_backup=false                     # true or false
You also can change you fusionpbx version from master to stable here if you wish. Now edit /usr/src/fusionpbx-install.sh/Debian/resources/postgresql.sh and change the password field to match the database password you set in config.sh
From This:
Bash:
#generate a random password
password=$(dd if=/dev/urandom bs=1 count=20 2>/dev/null | base64)
To This:
Bash:
#generate a random password
Password=<db Password>
I have issues with getting config.php to to take my settings, I am sure I am doing something wrong, but I just cahnge it in /usr/src/fusionpbx-install.sh/debian/resources/fusionpbx/config.php
PHP:
//pgsql: database connection information
        $db_host = '<DATABASE IP>';
        $db_port = '5432';
        $db_name = 'fusionpbx';
        $db_username = '{database_username}';
        $db_password = '{database_password}';
Also I change the database host in /usr/src/fusionpbx-install.sh/debian/resources/finish.sh. Now you can run /usr/src/fusionpbx-install.sh/debian/install.sh
Then you need to go to /usr/src/fusionpbx-install.sh/debian/resources/postgresql and run empty.sh on all but one of the databases then use node.sh to build your BDR cluster, this will need ran on all you databases. You will need to choose whether or not to replicate your freeswitch database as I understand it if you do you may reduce your scalability due to performance issues but without it you will not be able to track things like conference calls across nodes. Honestly I am not sure if node.sh actually makes changes to freeswitch or just sets up the BDR. If it doesn't you can make this change in /etc/freeswitch/autoload_configs/db.conf.xml
Change This:
XML:
<configuration name="db.conf" description="LIMIT DB Configuration">
    <settings>
        <!--<param name="odbc-dsn" value="$${dsn}"/>-->
    </settings>
</configuration>
To This:
XML:
<configuration name="db.conf" description="LIMIT DB Configuration">
    <settings>
        <param name="odbc-dsn" value="psql://hostaddr=<IP of DB> dbname=freeswitch user=freeswitch password='<DB Password>'"/>
        <!--<param name="odbc-dsn" value="$${dsn}"/>-->
    </settings>
</configuration>
Then delete /var/lib/freeswitch/db/core.db and restart freeswitch. Last thing is filesystem replication. I think officially syncthing is recomended but you can use any number of programs for this. Syncthing has a web gui which can be nice but you have to build an ssh tunnel to access it. To install it add your nodes to the hosts file in /etc/hosts and use the following to install syncthing:
Bash:
# Add the release PGP keys:
curl -s https://syncthing.net/release-key.txt | sudo apt-key add -

# Add the "stable" channel to your APT sources:
echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list

# Update and install syncthing:
sudo apt-get update
sudo apt-get install syncthing

chown www-data:www-data /usr/bin/syncthing
chown -R www-data:www-data /var/www

apt install apt-transport-https

systemctl enable --now syncthing@www-data
To access the Web GUI of Syncthing you need to setup an ssh tunnel before browsing to it. Im sure there is a way to change it so syncthing will listen on more then just local host but you can just use:
ssh -L 8384:127.0.0.1:8384 <USERNAME>@<node IP>
Then browse to 127.0.0.1:8384

I'm sure there are better ways to do this, but this is what I have found. I have never had occasion to put this setup into production, though I put a lot of effort into working out the kinks, but non the less I would recommend testing thing obviously, and if anyone else is reading this I would love feed back on this.
 
  • Like
Reactions: ict2842

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Thank you so much! I should be getting additional IPs assigned within the next few days. I will give this a try when I am able to provision more VMs.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
IP addresses have been assigned. I am starting the setup of VMs now and will complete tomorrow along with (hopefully) setting up the DB and Fusion for HA in Dallas. :D
 

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,048
569
113
Sorry to be the bearer of bad news but what is described above will leave you just to crash and burn.

I'm not getting into it all here but here are a couple of reasons.

Firstly, FusionnPBX cannot load balance except using per domain load balancing, ie where you lock domains to a single server otherwise many things will appear to work, some won't.

If you replicate the freeswitch DB over WAN, as soon as you get any load things are going to get slow, one of the first things will be blf lights etc there will be delays on their reaction to events.

Probably the most important one is caching. I see no mention of caching above. FusionPBX relies heavily on caching.
 

sudoRmRf

New Member
May 31, 2019
17
3
3
33
Sorry to be the bearer of bad news but what is described above will leave you just to crash and burn.

I'm not getting into it all here but here are a couple of reasons.

Firstly, FusionnPBX cannot load balance except using per domain load balancing, ie where you lock domains to a single server otherwise many things will appear to work, some won't.

If you replicate the freeswitch DB over WAN, as soon as you get any load things are going to get slow, one of the first things will be blf lights etc there will be delays on their reaction to events.

Probably the most important one is caching. I see no mention of caching above. FusionPBX relies heavily on caching.
Well that’s embarrassing . Its been about a year since I worked on this setup and as soon as you mentioned these thing I remembered those those concerns but like I said I never had the occasion to actually deploy this to testing. Additionally I should have mentioned my setup was with two data centers each DC having two Fusion, two DBs and, a kamailio.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Thank you both. It's better to not have started this than have to scrap and restart.
Firstly, FusionnPBX cannot load balance except using per domain load balancing, ie where you lock domains to a single server otherwise many things will appear to work, some won't.
I can scrap the load balance concept. The main goal is ensuring service is available in Dallas, that is the location nearest to us that I can host a server and a SIP trunking provider is present in. I did a few different latency tests between different locations and data centers. My servers in Dallas + BulkVS along with my same DC provider's NY location + BulkVS had the lowest latency compared to the other routes I tried. I am fine with more of a "fail-over" within Dallas from one server to the other. Again, I don't want calls to go to NY unless Dallas is burning or freezing.

Additionally I should have mentioned my setup was with two data centers each DC having two Fusion, two DBs and, a kamailio.
What I am going for is for the Dallas DC to have two Fusions and two DBs. Later on (within a year? maybe longer.) I would add a New York DC with one Fusion and one DB for a complete fail-over should Dallas go to hell, and even later on, add a second Fusion and DB in NY for further redundancy. At least that's what I would like to happen.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Is BDR still the way to go for the database? I will need to set it up and play with it, but I have read mixed things on here about it.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Let me ask this: If I cough up $1200 for the Green Membership, is the documentation actually there? Are the docs for HA something that works as opposed to all these forum discussions that are sketch and stop working after an update or two?

I am thinking to say screw it and pay for the Membership. My labor isn't something I am factoring in today, I don't need the money and I am not sure pricing would be enticing if I did - nor do I know how to price my labor as a college student with no actual work experience (almost all is "lab"). But I have these three orgs ready to move today and I am going back and forth trying to find out how to do things without the worry of it relying on a single server.
 

sudoRmRf

New Member
May 31, 2019
17
3
3
33
Let me ask this: If I cough up $1200 for the Green Membership, is the documentation actually there? Are the docs for HA something that works as opposed to all these forum discussions that are sketch and stop working after an update or two?

I am thinking to say screw it and pay for the Membership. My labor isn't something I am factoring in today, I don't need the money and I am not sure pricing would be enticing if I did - nor do I know how to price my labor as a college student with no actual work experience (almost all is "lab"). But I have these three orgs ready to move today and I am going back and forth trying to find out how to do things without the worry of it relying on a single server.i
It’s in videos. That’s where I got most of that stuff from.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
Okay, I will give it a bit to see if anyone provides a reason not to get the membership. I really wanted to avoid committing to a 12 month subscription before I officially started but it seems that is the only way to get proper and current information on a number of components. The forum helps, but a lot of what I find is outdated.
 

sudoRmRf

New Member
May 31, 2019
17
3
3
33
Okay, I will give it a bit to see if anyone provides a reason not to get the membership. I really wanted to avoid committing to a 12 month subscription before I officially started but it seems that is the only way to get proper and current information on a number of components. The forum helps, but a lot of what I find is outdated.
I will say there is some value to the membership. I just wish more of this stuff was available since it’s a open source program and all.
 

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
I will say there is some value to the membership. I just wish more of this stuff was available since it’s a open source program and all.
Exactly my POV here. I would be more inclined to purchase the subscription if I knew how things worked. I am struggling to set parts up and to understand some functionality. For $1200, quite honestly I expect a everything I need to configure it from scratch, but how knows. I also wish they did one off support requests. Bill me buy the hour a reasonable rate, I don't care... But upgrading if I need support and then upgrading if I go over the included support time is not the route I want to go, as they've made it.
 

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,461
393
83
Don't get too hung up about HA at this early stage in developing your platform. As you gradually learn more about FusionPBX and FreeSWITCH you will develop your own ideas of how you would like to implement backup and fail over etc. As with so many things in life there is no right or wrong way, just the way that works for you.

You do not have to pay anything to ask for help on this forum, and rest assured, people will help you!
 
  • Like
Reactions: ict2842

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
You do not have to pay anything to ask for help on this forum, and rest assured, people will help you!
I know; I have seen it time and time again and I greatly appreciate the assistance provided to me and others, because those discussions have helped me several times.

I am going to set things up again from scratch on my new subnet and will likely move my FreePBX system over for testing within the next two weeks or so. I want to get things going but also want to do this right. It's really the fear of taking down a phone system that is used daily that I have. Registering an LLC, finding an insurance provider in a bit (any recommendations for US businesses? I can reach out to some brokers eventually), the hosting itself, finding some clients (I have a list of family friends that'll likely switch if I have things working and they are out of contract), etc. is not a problem.
 

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,048
569
113
@ict2842 There is a big difference between load balancing and failover. I think I have failover working pretty sweet now.

I have active/passive pairs using BDR for the DB.

Despite what people say about BDR and it being old, EOL etc, it does the job. Even if it can't be installed on Debian for whatever reason in the future, I'm already thinking of putting it into docker.

The servers needn't be big at all, don't overdo your hardware.

Its bedtime now, but I'll tell you my flow tomorrow.
 
  • Like
Reactions: ict2842

ict2842

Member
Mar 2, 2021
140
11
18
Wichita, KS
I guess another question here: Do I need separate VMs for the DBs? If so, I'll make them. If not, I won't waste the time making them to destroy them. Personally, I would rather have a separate DB component.

I got Debian 10.8 installed on two fresh VMs. Whether HA is done or not, I am going to rebuild what I have done and leave my current install as lab.
 

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,461
393
83
leave my current install as lab.
Always good to have a "lab" I have several "testing" setups in various configurations depending what I want to test. Sometimes I will configure a test box in order to try and replicate an issue that a customer may be having.
 
  • Like
Reactions: ict2842
Status
Not open for further replies.