NEW CallerID Superfecta 2.2.4: THE MODULE

Caller ID Superfecta: THE MODULE v 2.2.4 Release Announcement

NOTE: For the latest 2.2.4 version and release notes, jump to here. (Brand NEW as of 12/23/10)

The rest of this article is obsolete, but is maintained here for future reference. See the link ABOVE for the Latest and Greatest Superfecta Version 2.2.4

I am delighted to announce the release and immediate availability of Caller ID Superfecta: THE MODULE Version 2.

Version 2 represents the largest change to the inner workings of the Superfecta since the module version was created. Version 1.X very strictly adhered to the original Nerd Vittles version of the program in both structure and approach. This served us well, but made the module harder than necessary to maintain -- especially considering the regular desire to add new data sources and the regular need to update the parsing routines due to provider changes.

Jeremy Jacobs, PBXIAF forum member and principle developer for Superfecta module version 2.0, has addressed several important trouble tickets, and feature requests, as well as initiating significant architectural changes designed to make future maintenance of the module easier.

Here's a brief list of the upgrades and changes that come with version 2.0 of caller ID Superfecta: The Module.

• No longer returns a "," for the white pages data provider. (Ticket #: 3)

• User selected provider search order. (Ticket #: 7, and 37)
You now have complete control over the order in which providers are searched for the caller ID information.

• Now includes the Asterisk phonebook as a lookup source. (Ticket #: 15)

• Prefix code hooks. (Ticket #: 10)
The caller ID Superfecta now supports the ability to trigger an additional outside process when the caller ID function is run. For example, this feature might be used in a call center to allow the system to automatically bring up a customer record number for the customer service representatives use automatically.

• Provider Google lookups now work again. (Ticket #: 20)

• Provide operating credentials to Superfecta from inside the module Web user interface. (Ticket #: 30, and 17)

• New data provider: Addresses (Ticket #: 19)

• New data provider: Phone Spam Filter (Ticket #: 19)

• New data provider: Yellow Pages (Ticket #: 19)

• New timeout parameter: prevents long timeouts from unavailable data sources. (Ticket #: 19)

• White Page now returns a business name, where one is present, instead of a persons name. (Ticket #: 41)

• New Debug function (Ticket #: 36)
In version 2.0 of the module, the debug function is built into the Web user interface. By entering the target telephone number into the debug field and pressing the debug button on the form -- you are provided with the results, and latency for each enabled data source. This new debug function should greatly enhance and speed up any possible troubleshooting. But the debug function is also a great tool to help you fine tune your provider selection and sort order. Providers who can’t be reached, or who perform too slowly can be turned off.

Under the Hood
An entirely new architecture has been deployed where the data providers are concerned. In the past, all data providers were defined in one single file, along with all of the data collection and parsing routines. In addition, the module user interface required manual coding intervention any time data sources were added, or removed.

In version 2.0, things have changed dramatically for the better. Now, each data source is provided via its own small customized script. Updating a specific data provider’s script no longer means wading through the data and code for all of the providers. Even the web ui and database data are automatically updated All this means faster, easier maintenance releases in the future.

Developing good software is hard work and takes a lot of time. Doing that for free, when you already have a full-time job, as well as a "real life", can only be the result of having suffered repeated head injuries as a child. Of course I'm only kidding! In fact, doing that for free can only be the result of possessing the most generous of spirits, and a genuine desire to contribute to the creation of open source software.

I’m sure you will agree Jeremy deserves a lot more than only our thanks and appreciation for these great upgrades. I hope Karma rewards him richly! But, until Karma gets around to it, please join me in a round of applause and sincere thanks for forum member Jeremy Jacobs (FitnessPlus) for taking on the re-engineering and upgrade of the CallerID Superfecta: THE MODULE.

Caller ID Superfecta: THE MODULE Version 2.0 May Be Downloaded from the Development Support Site at This Location.

-tshif

Caller ID Superfecta: THE MODULE – Version 2.0.0 Installation and Use

Caller ID Superfecta: THE MODULE V 2

The Caller ID Superfecta has proven to be the most often downloaded module in the Development support site – and for good reasons. Everyone expects caller ID services with their telephone service. And WE ALL KNOW there’s only a few choices for getting it. We can pay our TSP’s for it – if available, we can pay an outside source for it, or we can use something like The Caller ID Superfecta for it.

In the beginning, our own NerdUno developed the Caller ID trifecta, and released it into the wild. As a testimony to open software – his original creation went on to be maintained and updated by the members of this forum. Renamed by Ward in one of it’s previous upgrades to “Caller ID Superfecta”, and in line with his original vision, it has been extended to include more sources of information than ever before.

Then, we released Superfecta: THE MODULE to take advantage of the FreePBX module format for ease of install and upgrade – to make it readily usable by a larger number of users. But – as we know, nothing is as consistent as change – and change is exactly what several of the data providers regularly undergo. This of course requires updates to the Superfecta code to keep everything working.

Open Software Should Be Open
It’s a rare and beautiful thing to see folks freely contributing to a coding project. But have a look around here at the PBXIAF forums and you see it; A lot of it. It may not be unique – buts its rare enough, and worthy of all our support. Toward that end, a Development Support Site and SVN has recently been provided to host the Superfecta (and several other Open Source PBX projects’) development effort and provide a place for the distributed community development effort to thrive. (http://projects.colsolgrp.net) This new release of The Caller ID Superfecta is the third exciting piece of software to be released through the Dev site. Every single bit of work done on the module has been accomplished by members of this very forum – on a completely volunteer basis.

Additional Documentation Available
User and technical documentation is available that discusses the inner workings of the new version of the Caller ID Superfecta. (http://projects.colsolgrp.net/projects/superfecta/documents)

Install and Upgrade
This module is a direct upgrade of the previous Caller ID Superfecta. It can co-reside with the non-module version of Superfecta, or any other Caller ID solution, without difficulty. It will take less than 1 minute to complete the install. Upgrading will not wipe out your settings. (Settings from version 1.x of the module may be lost during upgrade – so make a note of anything you want to remember before the upgrade.)



Installation:
Step 1: Download the module to your desktop From here.
(Generally, you'll want to select the most recent released of the Version 2.X Series.)
Don’t decompress the archive!
Step 2: Use a web browser to access FreePBX on your PBX. Choose Tools, Module Admin, Upload Module.
Now, browse to the superfecta-2.X.X.tgz file on your Desktop and click the Upload button. Now click local module administration, scroll down and click on CID Superfecta and then the Install radio button. Now click Process, and complete the usual FreePBX install and reload process.
Step 3: Use a web browser to access FreePBX on your PBX. Under Setup, Select CID Superfecta.
You will be presented with configuration screen. Establish settings that make sense for your PBX, and Save your changes.

The module installer has created the entry for Caller ID Superfecta in the FreePBX Caller ID Lookup Sources. The ID and Password Fields for the Caller ID Superfecta entry in the FreePBX Caller ID Lookup Sources area will be filled in by controls on the CallerID Superfecta Module Web UI.

Configuring Services
As you know, the current version of FreePBX can only assign ONE CallerID lookup source per inbound route. CallerID Superfecta gets around this limitation quite nicely by allowing you to configure from a total of 11 different sources, to be used in any combination you see fit.

Using the ENABLE and DISABLE radio buttons, select those services you wish to use for your Caller ID. Please note that SugarCRM and WhoCalled require additional configuration items (down low on the UI) to function.

The much requested and utterly COOL feature of being able to choose the order in which the sources are used is fully deployed in this version of the Superfecta Module. Use the Up and Down arrows to change the order to one that makes sense for your circumstances.

General Options
The ID and password in General Options is required when your PBX runs on Apache Security (Such as PBXIAF 1.1-1.3/4, and Generally TrixboxCE). This will generally be your MAINT/maintpassword combination.

If you run a distro when you use the FreePBX ACL (credentials) method instead of Apache security, these two fields may be left blank. (PBXIAF 1.0 in ACL mode, Foncordiax, Fonica PABX distributions.)

Lookup Timeout allows you to define how long Superfecta waits on a result from any data source before giving up and moving on.

Spam Count Threshold allows you to define the number of times a phone number is reported as a SPAM source before Labeling it as such. For now, only the source PhoneSpamFilter uses this value. As other sources that use this type of threshold are added, they will use this value as their threshold also.

CID Prefix URL allows the caller ID Superfecta to trigger an additional outside process when the caller ID function is run. For example, this feature might be used in a call center to allow the system to automatically bring up a customer ID number for the customer service representatives' use. The syntax looks like this example:

Code:

[URL="http://nowhere.example/"]http://special.someonescallcenter.org/get_customer record.php?phone_number=[thenumber][/URL]

In this example, the Superfecta would send this URL as it begins the lookup process. (This function is provided for advanced users with advanced needs.)

Debug Functions
The Debug functions are now built into the application. The debug feature has lots of uses, and it’s easier to use than ever. The debug system takes the number you enter, and runs it through all the sources you have selected. It displays the results for each source, as well as the time it took to get the result. Using this information, you can now make more informed choices about which sources you want to use, and in what order you want to use them.

The last bit of info in the debug window shows you which data source would have been used, and how long it would have taken to do so.

Find out which sources give the best data for your use, which sources are fastest and slowest, and use this information to determine exactly which service you want to use, and in what order.

Conditions/Prerequisites:
This module is compatible with the security models used in, and has been tested with the following distributions:
Fonicatec PABX
Foncordiax
PBX In A Flash
TrixBOX CE
The Module’s wiki Page is here.
The Module’s Development Support Site is here.


Asterisk Version Compatibility
This module has been tested and found to work correctly with Asterisk versions 1.4 and 1.6.

Credits:
The original CallerID Superfecta was written originally by Ward Mundy, as the Caller ID Trifecta. It has evolved to support more sources, and is now maintained by the community of forum users at PBX In a Flash. The Superfecta was converted to FreePBX Module format by Tony Shiffer & Jerry Swordsteel.

The principle and lead developer of Caller ID Superfecta: THE MODULE v 2.0.0 is Jeremy Jacobs.

Until next time
-tshif

Updated: 5/24/09 The version of Caller ID Superfecta shown here has been superseded by version 2.1.X, which you can read about here.

Great work, but would you guys consider adding the Google search like there used to be in one of the older scripts ?

It did a search and if x amount of the results were in a preconfigured list it marked it as spam.

We'll consider just about anything - but your description was a little thin of details. I don't remember a preconfigured Google list - Tell me more about it, or direct us to a link with more info -

It was in the last version of the ultimate cnam lookup

http://pbxinaflash.com/forum/showthread.php?t=65&highlight=cnam&page=20

Code:

# www.google.com general search
##(set in enable/disable section)## Googlesearch = 1
# Will insert GS_CNAM as a prefix into the CNAM string
# if www.google.com lookup scanned for phone spam web sites
# yields a number of hits above GS_threshold.
# GS_CNAM may be in sprintf format, with %u standing for
# spam score. 
# E.g.:
#  GS_CNAM = "GSPAM%u "
#  GS_CNAM = "GS%u "
#  GS_CNAM = GSPAM%u
#  GS_CNAM = GS%u
#  GS_CNAM = "SPAM score %u "
GS_CNAM = GS%u
# GS_sites is a space-separated list of phone spam web sites.  Can use
# backslash \ to continue to next line.  Each Google hit mentioning any
# of these adds 1 to the score.  Site names are case-insensitive.
# E.g.:
#  GS_sites = 800notes.com whocalled.us whocallsme.com callercomplaints.com
# Note: some of these sites seem to be owned by the same people. We need to
# make a list of independent sites.
GS_sites = 800notes.com whocalled.us whocallsme.com callercomplaints.com yourcallerid.com
GS_threshold = 3

Being outside of North America non of the lookup sources are any real use, it would be useful to do a Google search on the phone number and compare the first page of results to the list of urls which could be edited to suit, if x amount appear then mark it spam.

I have added your feature request to the Dev support site:
http://projects.colsolgrp.net/issues/show/53

I should also point out that with the new version of Superfecta, adding most new sources is extremely easy.

We already support one of the sources on your list, WhoCalled.us.

If you decide to roll up your sleeves and build a new source for any of these you have suggested (800notes.com whocalled.us whocallsme.com callercomplaints.com yourcallerid.com) - let us know - and well open a new subproject at the dev site for you to work in. Contributions are always welcome!

I think I can safely say its well beyond me, I can see adding a new source similar to whocalled.us would be simple (even for me) but I can't begin to understand how to make the cnam Google search fit into it.

The 1 site that seems to always have my nuisance callers is whocallsme.com, I'm not sure why so many people in the UK are using that 1 site, sadly it has no way to get a easy result or spam score.

Myself I would edit that list to just whocallsme but others could change it to whatever sites work for them depending on where in the world they are, I assume theres hundreds of these sites in various languages.

Actually - one of the things I hope to see come of the new architecture is the addition of sources for numbers outside the US.

I know one of the reasons we don't have such sources today is what we have perceived to be a lack of sites from which to get the data. I had not seen a couple of the sources you provided before today - so that's good to have them.

Also - in previous versions of the Superfecta (and maybe the current one - I haven't checked) we deal with telephone numbers formatted as 10 digits only. It seems to me there may be some additional programming concerns to support phone numbers with more digits.

In any case - its worthy of exploration. Toward that end I have created the following two tickets in the Superfecta dev support site.

Provider script for whocallsme
Size task for non US Phone number support

Don't be to anxious for results on any of the three tickets I opened tonight. We just finished a fairly major release and the guys who volunteer their time for these projects will probably not be ready to jump into anything major in the way of immediate updates. But your thoughts are appreciated - and stay tuned. You never can tell what might happen or how fast until it does!
-tshif

I did test the module with a number and noticed it took my UK number and mangled it into a US type before getting a wrong result from whocalled.us.

I'll keep looking at it, sometimes these things become clearer if I look at it long enough , thanks.

Awesome update, keep up the good work!!!

Andy

Great job. Thanks a lot.

Wow it works great! Thanks, the update was easy to do! I love the sandbox you added to debug!

A description of each service would be good too ... You could specify, this source is good for USA/CAnada, this one is only good to check residential, etc...

I just created my first new source for this new module... It works from canpages.ca and right now only works for residential numbers, I might change the script to get commercial listing out of it, but their HTML is really messed up. I actually simply extract the name from the <title></title> but the doesn't work for commercial. It's also way faster than WhitePages, I get sub 1 sec result most of the time, while White Pages results are more than 3 seconds.

Where should I submit it?

Now its my turn to say it - Wow - Thats GREAT!

I have sent you a PM to get this underway -
-tshif

Note: We will have an actual procedure ready very soon for anyone to contribute sources like this. The truth is - you guys "got the spirit" of giving just a little faster than we got the submission process together. I'll get that fixed right away, but until then, don't let that stop you from contributing. Send me a PM, or message right here, and Ill get back to you with submission instructions ASAP. And again, thanks for contributing!

Update: You code has been checked in and is now undergoing QA testng. Early results are very good. THANK YOU for your contribution. Ill keep you posted as for its progress.

-tshif

Please help my understanding

It seems to me that this module is now doing two separate but semi-related functions: CallerID lookup and spam filtering. Here are my questions. Let's suppose I have a provider that NORMALLY sends me "good" caller ID data. So I don't need CallerID lookup most of the time (and in such cases, I don't want what I've already received from my provider changed), but I do want to do "spam" call filtering. Is that possible, and if so, how do I set it up to do that?

But really, there are two exceptions to the above rule: I do want it to check my internal directory (Asterisk Phonebook) and if the number is in there, override what the provider sent (but if there is no matching entry in the Asterisk Phonebook, to not touch what the provider has sent). Is that possible?

And also, if the provider has sent "useless" information (e.g. blank data, "unknown", or any of the words/phrases that commonly indicate that no valid data is available) then I'd also like it to try and pull a name, or at least a city/state, from one of the other sources, but still while not affecting "good" data. Is it possible to do that with the existing module?

If it's possible to do the first and second items but not the third, then I might suggest that for each lookup source, perhaps you really ought to have three options: Disabled (never use), Conditionally Enabled, and Enabled (always use). So in my scenario, I might "Enable" the Asterisk Phonebook and a spam filter because I always want that used, but only conditionally enable the other lookup sources. Then you could have a text box for entering Caller ID "names" that would trigger conditional use - a blank name (or all whitespace) would always trigger, but you could also put in words like Unknown or Private, or whatever your provider(s) send instead of real data, in the textbox. When "Good" data is received, it would only check the Enabled sources (probably spam filetering and/or Asterisk Phonebook), but when useless data is received it would also try the "conditional" sources. Just a thought.

Also I do wish there were documentation (or maybe a video) that more explicitly explains how, when and why each option is used. The mouseovers are of some help but for example, I'm still mystified as to how or why you'd use a "CID Prefix URL."

Use of Everycall.us?

Forgot to mention, the use of Everycall.us as a "junk" call lookup source (that does not require registration for use) is documented in How to automatically reject calls from telemarketers and other "junk" callers, but I don't know how well the method shown there would translate into something that could be added to your module.

We would love volunteers to create documentation for users - it could be posted to the dev support site or here quite easily.

One of the things I have learned about free software over the years is that documentation is sacrificed for the low low price. If there are any docs, they were written by good, hard working folks like yourself - experimenting with the software, writing down the results, and formulating it into a users document of sorts.

If your ready to roll up your sleeves, and start that process - I promise well take the docs you produce and post them for everyone's use - with plenty of credit for your diligence and example of volunteerism.

Shall I sign you up?

Catch 22

It's the chicken and egg problem - in order to write good documentation you need to understand how the module works in the first place, and in order to understand how the module works you need some documentation (or at least answers to the questions you ask) to get you started.

But your point is well taken, as is Ward's - this is a fantastic piece of software, especially for those whose providers don't send good data. The only point I was really getting at was that if your provider is sending good data most of the time (that it's pulling from official telco sources), you more than likely don't want that overwritten, except in the "outlying" cases where bad/no data is sent. But you might still want to do the spam lookup/filtering. If I knew in advance it would not trash the "good" data I'd be a lot more willing to experiment with it (and who knows, maybe write some better docs after I become more comfortable with it).