Working SMS Integration

Status
Not open for further replies.

Edson

Member
Aug 1, 2017
59
4
8
46
PG database lock only for cluster or would this be a possibility for standalone?
It was a BDR DDL issue replication causing this database lock. Somehow this was the reason the table could not be created. I dont think this a possiblitity regarding standalone install. What is the PostgreSql version running in your system? Mine is 9.4.
 
  • Like
Reactions: FuncVOIP

FuncVOIP

New Member
Apr 20, 2018
14
1
3
28
It was a BDR DDL issue replication causing this database lock. Somehow this was the reason the table could not be created. I dont think this a possiblitity regarding standalone install. What is the PostgreSql version running in your system? Mine is 9.4.
Hello Edson,

I am using version 9.6.5 . Maybe I have some bug elsewhere. I am going to attempt a fresh install to see if that does better.
 
Jan 9, 2018
140
12
18
54
I'll just put this out there for those of you attempting to use this module:
I alluded to an issue in the original post, where on some softphone clients it would duplicate the message 120+ times. It will then usually crash/freeze the freeswitch process. It turns out, this was not a problem with the softphone client. It is a problem with Freeswitch. And apparently, not even with the SMS module itself. This bug report indicates that it is an issue with the underlying handler. In spite of the lack of any apparent definite resolution to that bug report, we have only replicated the problem under 1.6.x versions of Freeswitch and not 1.8.x, so it looks like the problem has been fixed. I believe @MTR has also experienced similar results.

Unfortunately, our production systems are currently running 1.6.x, so we have had to disable the module while prepping a transition. There are some complicating factors that make this transition difficult, which I won't go into.

Suffice it to say that I strongly recommend that if you are considering using this module, make sure that you on Freeswitch 1.8.

Here's a follow-up: We've now rolled out upgrades on our servers to Freeswitch 1.8. While we are not seeing the crash/freeze in the Freeswitch process as mentioned previously, we are still seeing the duplicated messages (120+) when the message is sent to an invalid or unregistered extension. We have had to disable local sends to avoid this, because we are now running down another possible memory corruption problem that we suspect is deriving from this. This means no on-net, ext-to-ext chatting. If you are using a version of my chatplan, above, the relevant part now looks like this:
XML:
                <extension name="other">
                        <condition field="to" expression="^(.*)$">
                                <action application="set" data="final_delivery=true"/>
                                <!-- <action application="send"/> -->
                        </condition>
                </extension>

Ideally, the lua script would handle both sends, and would have a way to check to see if the extension is registered before attempting the send. That would be an ideal upgrade.
 

mtek

New Member
Jul 2, 2018
4
0
1
41
It looks like something is wrong in the \sms\app_config.php file. It's looking for two tables named "v", not finding them, and trying to create them. Instead, it should be looking for "v_sms_messages" and "v_sms_destinations". There may be updates to the 4.4 schema upgrade engine. I've tweaked a couple of settings in the attached. Give this a try and see how it works.

I just did a fresh install of Debian 9 and 4.4 and set up this just following this. https://github.com/fusionpbx/fusionpbx-apps/tree/master/sms and I was unable to add the APP / SMS destination so I found your app_config.php a few posts above and then I changed out what I just downloaded from git. This allowed me to run the upgrade schema and it added the correct tables. I verified this by running the schema upgrade again and nothing was changed. I was then able to add my sms destination.

Can you take the time to update the git repo with this file and also make a note that this shouldn't be used with freeswitch 1.6. If this would have worked out of the box and I didn't go searching I would have possibly crashed my production system due to the bug that you posted that causes high cpu usage.
 

yukon

Member
Oct 3, 2016
138
14
18
Hi. I've got the files from Jonathan and I am working on trying to resolve the bug. Once I figure it out, I'll update the repo.
 
Jan 9, 2018
140
12
18
54
I just did a fresh install of Debian 9 and 4.4 and set up this just following this. https://github.com/fusionpbx/fusionpbx-apps/tree/master/sms and I was unable to add the APP / SMS destination so I found your app_config.php a few posts above and then I changed out what I just downloaded from git. This allowed me to run the upgrade schema and it added the correct tables. I verified this by running the schema upgrade again and nothing was changed. I was then able to add my sms destination.

Can you take the time to update the git repo with this file and also make a note that this shouldn't be used with freeswitch 1.6. If this would have worked out of the box and I didn't go searching I would have possibly crashed my production system due to the bug that you posted that causes high cpu usage.
The data table error you quoted may be a bit of a red herring, though once @yukon gets this all updated, I think the point will be moot. The original error I was addressing turned out to be not fixed by that app_config.php file change, but rather an issue on the poster's system related to BDR locking.

Regarding compatibility with Freeswitch 1.6, please note the following:
  • In my testing, this will work fine with FS 1.6, provided you are only using it with a single server. I've only replicated the problem with servers set up in a cluster of two or more.
  • In that original bug report on the Freeswitch site, the problem occurred independently of this SMS app (I don't believe it even mentioned FusionPBX). This is a bug in Freeswitch 1.6 related to having two servers in a cluster, and can be triggered without even using this app, though it is less likely to be triggered without this app or at least the Freeswitch SMS module active.
That said, @yukon, it's probably a good idea to add a caution about this to the read-me on this app when you do the update to github.
 

MTR

Member
Oct 25, 2017
181
9
18
45
i am trying to use the system with twillio
i am having different customers with different access keys

if i add the default settings under the domain the sms program will not pick it up,it will only pick it up from the global default settings

any idea how i can fix it ?
 
Jan 9, 2018
140
12
18
54
Hmm, good question. I didn't write that portion of the code, but looking at it, apparently the author didn't envision that and didn't build in support for it. If you want to look at it, open /apps/sms/index.lua and search for v_default_settings. Looks like those domain-level settings are stored in v_domain_settings. The code will need to be modified to look there first in all those places. I'm planning some fairly big updates to this app in the next couple of weeks. I'll try to take a look at this then, unless someone else does so first.
 

MTR

Member
Oct 25, 2017
181
9
18
45
Hmm, good question. I didn't write that portion of the code, but looking at it, apparently the author didn't envision that and didn't build in support for it. If you want to look at it, open /apps/sms/index.lua and search for v_default_settings. Looks like those domain-level settings are stored in v_domain_settings. The code will need to be modified to look there first in all those places. I'm planning some fairly big updates to this app in the next couple of weeks. I'll try to take a look at this then, unless someone else does so first.
Thanks if I will have time today to fix it I will submit the code
 
Jan 9, 2018
140
12
18
54
Thanks if I will have time today to fix it I will submit the code
Ok. I'm working on other parts of it, but I'll look for any updates you may make.

Also, you may wish to look and see how Mark handles this idea in other parts of the code. I haven't looked, but in theory, it seems like this would be a good place for a dedicated function call for retrieving settings.
 

MTR

Member
Oct 25, 2017
181
9
18
45
it seem like this code talks direcly to SQL mark has a class for this
under /resources/function/settings.lua

require "resources.functions.settings";

i will need to rewrite the code block for twillio
 
Jan 9, 2018
140
12
18
54
it seem like this code talks direcly to SQL mark has a class for this
under /resources/function/settings.lua

require "resources.functions.settings";

i will need to rewrite the code block for twillio
Exactly! I think that will work much better. It should also clean up the code and even make it more efficient, because that one single function loads all of the default (global) and domain settings into an array. They can then be accessed as needed from the array, instead of doing a SQL request every time we need another setting.

You should be able to call the "settings" function at the top and then making the code changes for the individual settings should be fairly easy.
 

MTR

Member
Oct 25, 2017
181
9
18
45
i have noticed something weird in the sms module

if i send a sms with 3 lines meaning i start a new line in the sms the system will only process the first line
so in my HTTPS post i can see all 3 lines , the PHP code is processing all 3 lines as well (i see this in the logs)
in freeswitch logs i only see the first line

i am not sure where this is getting lost
 
Jan 9, 2018
140
12
18
54
I'll take a look at this. I'm beta-testing some fairly big changes to this code at the moment, so I'll see if I can replicate your situation.
 

MTR

Member
Oct 25, 2017
181
9
18
45
Thank you
Do you have an ETA for committing this new code?
What are the big changes?
 
Jan 9, 2018
140
12
18
54
Thank you
Do you have an ETA for committing this new code?
What are the big changes?
ETA: based on workload, probably next week at the earliest. I squashed some more bugs today, including the multi-line problem.

I have incorporated this settings change. I do need to go back and reconcile any other changes made to this module of code since I last synced--I think someone added another carrier.
The changes are as follows:
  1. Added limited MMS handling. It is limited in that it only works for inbound, and for the Telnyx carrier. Also, it doesn't pass through the attachment to the endpoint, but just sends the attachment to email.
  2. Moved email address location into the v_sms_destinations table. And to the Apps/SMS/Destination Edit page.
  3. Moved email handling into its own function.
  4. Change settings from direct SQL access to use library functions.
  5. Changed from dynamic to parameterized queries in several places--improves security.
  6. Added check for target extension's registration status before sending the message to the endpoint.
  7. Added filter for message type to catch delivery status notifications sent by some softphone clients. This resulted in “garbage messages” (xml in the body). Seems to be eliminating this problem so far.

Some notes:
  • The MMS handling is, of course, the major change. Our immediate use case dealt with receiving MMS attachments by email, and primarily pictures, so that's where I started. It likely is possible to deliver the attachments to the endpoints and even send outbound; however, I discovered that the main softphones that we are using utilize different and incompatible strategies/formats for this. That pretty much means that the code needs to be aware of the endpoint type, and tailor the message accordingly. If there is an industry standard format that works for all softphones, I'm not aware of it. Also, since we only deal with one carrier, I don't have access to test with any others. In theory, it shouldn't be too difficult for people to add other carriers, now that the framework is in place for this, but YMMV.
  • The change of email address location is non-trivial from a coding perspective, as it involved moving the processing from Lua to PHP. I did preserve backwards compatibility with the "Apps/Contacts" method, for those who are using this. However, the MMS attachment delivery requires using the Apps/SMS/Destination Edit location. The difference here is important functionally otherwise as well. Using the Contacts method, the email address is set at the extension level. If you send SMS to a ring group, each extension will get an email. Using the SMS/Destination method, the email address is set at the phone number level, regardless of extension/ring group routing. I actually prefer this method, because it avoids duplicate emails if multiple extensions in a ring group belong to the same user. And if you need to enter multiple email addresses in SMS/Destination Edit, just separate with a semicolon. And if you want to ONLY deliver to an email, just leave extension/ring group blank.
  • The check for target registration isn't foolproof, but should help avoid some issues with looping in clustered environments. I'm looking into similar changes for the chatplan to take care of this for internal messaging.

As I said, I’ve had to run down several bugs today, but we are actively using this in production, so hopefully this will be stable by sometime next week.
Stay tuned--I'll post when the changes are committed.
 

MTR

Member
Oct 25, 2017
181
9
18
45
For point 4. Using library instead of SQL, is that for default settings?

Also the carrier that someone add it . This was me. Shuold I send you the code I did on a PM?
 
Jan 9, 2018
140
12
18
54
For point 4. Using library instead of SQL, is that for default settings?

Also the carrier that someone add it . This was me. Shuold I send you the code I did on a PM?
On point 4. Yes, that’s the default settings that we discussed before.

If you have committed all your code, I can get it from github (since I need to do a diff check there anyway). If not, feel free to send it over.
 
Jan 9, 2018
140
12
18
54
I've hit a bit of a snag with loading default settings in PHP. I can, of course, load the specific settings I'm looking for directly from SQL, but I'd rather use a library function, similar to what I've referenced in Lua (based on comments above). I'm having a bit of trouble finding them in the code base. Can anyone suggest where I might look for this?
Just FYI, this is without any user logging in. I'm guessing that there is some function that is called upon login that loads the global defaults, along with any domain-level settings. However, with webhooks, that login doesn't happen.
 
Status
Not open for further replies.