1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. If you had a PIAF Forum account in the vBulletin days, log in with your old credentials. Otherwise, sign up again and we'll get you back in business as soon as we can.
  3. A serious FreePBX vulnerability has been reported. Update your Framework Module immediately. Click here for details.
  4. Critical FreePBX vulnerability! Update your server immediately. Details here.

XO Communications Outage - Vitelity

Discussion in 'Providers' started by mruge, May 20, 2010.

  1. mruge Guru

    It appears that there is an outage with XO Communications and this is affecting their network of DIDs. Here is the latest that is being reported by Vitelity:

  2. Fortel Guru

    Vitelity DIDs / XO Communications down

    Right...

    It's been a bit over 4 hours now... all of our Vitelity DIDs are failing with "Your call cannot be completed..." or some such...

    If VoIP is a bit slow to catch on, these are the kinds of problems that don't help.

    Ugg!
  3. NorCal Internet New Member

    Same here... Luckily, I guess, my number is down too, so none of my customer's can call and complain...
  4. The Deacon Guru

    I just called XO and spoke with a rep named Heidi. She reports that XO is not having an outage, nor is she aware of any outage affecting any DIDs. She also said that I'm the second or third call that she has received today.

    FWIW: I noticed the same "...XO is experiencing an outage..." message posted on the voip.ms site as well.

    The weird thing is that the failover numbers for Vitelity aren't failing over; it's telling people that the number is no longer in service. :shocked:
  5. cjkeeme Guru

    This is the longest downtime I have ever seen. What is the deal?

    This is because Vitelity provides voip.ms with their DID's.

    EDIT: Still down...I have purchased new inbound DIDs from another provider and am forwarding all calls to these new numbers. At least Vitelity's failover and forwarding system seem to be working... This is a very expensive solution. I just can't see how an outage can last this long. I hate to speculate, but could this be a contract dispute?
  6. Fortel Guru

    Vitelity's failover / forwarding isn't working for most of our DIDs...

    (Oh, and I called XO earlier...tech was not aware of any issues affecting any DIDs.) My guess is some service that XO provides to Vitelity is affecting Vitelity DIDs, not XO DIDs...

    Big problem here...
  7. phinphan Member

    Same problem here with Vitelity. Calls to all my numbers get the not in service message and I cannot forward the calls. This is a big no-no to have a 10 hour and counting outage.
  8. cjkeeme Guru

    I have encountered a few instances where forwarding isn't working, but failover is working in those cases. It's worth a shot.

    Does anyone find it impossible for a Tier 1 provider like XO Communications to have a 12 hours outage. I have never heard of anything like this.
  9. cjkeeme Guru

    All is back up again. That was not fun...need sleep now.
  10. SpaethCo Member

    Sorry to say this, but then you haven't been paying attention.

    http://www.datacenterknowledge.com/archives/category/downtime/

    There is no shortage of stories related to outages at major hosting and carrier hotel facilities.

    Downtime sucks, but you can only design so much redundancy into your system and account for so many failures. Sometimes you just can't plan for events like an explosion in your electrical room taking out 3 walls and causing the Fire Department to force a complete electrical blackout of the building for 5+ days.

    Major outages are rare, but not impossible.
  11. wardmundy Nerd Uno

    One nice advantage of adding the Google Voice DID layer is that it makes it incredibly easy to switch inbound calls to another DID when there is a failure like this... at least until the Google Voice DIDs croak. :wink5:
  12. phinphan Member

    If only Google Voice would let you get more than 1 call per account that would be perfect but that would not make much sense for them.
  13. NorCal Internet New Member

    It would seem as though they are partially back...

    I'm still seeing issues when calls are made from AT&T wireless at least, and possibly all of AT&T, getting "Disconnected" messages.

    Calls from Verizon, Surewest and others seem to be working to the same numbers...
  14. If this really hosed you, you should ask for a root cause analysis (RCA) or request for outage (RFO) document from them. Something like this would be that the mgmt or executive level and would have some type of document to explain the issue. They generally have a public one. If they do not have one, or they will not give it to you, that says something. That they do not take availability really seriously.

    If they blame a partner, they would have already asked for this from them and should provide regardless.

    It is pretty standard.
  15. NorCal Internet New Member

    The issue with calls completing from AT&T wireless seems to be resolved now... So, as far as I can tell, everything is working again...
  16. parker New Member

    XO DID outage

    Scuttlebutt is that the outage was XO, and affected other ITSP's, not just Vitelity.
  17. The Deacon Guru

    I just received an email from Vitelity regarding the outage. It appears to be a hardware issue on the XO side, but from my vantage point, Vitelity had a part to play in this by dropping the ball regarding notification, failover and updates to its customers.

    So here's to having been on both sides of this issue and knowing what a lesson this can be. :cheers2:

    Here's a copy of the email:

  18. sfr33man2 New Member

    Ongoing and increaing Vitelity issues

    We have been a customer of Vitelity for years now. We actually signed up when it was EXGN (who was acquired or merged with Vitelity)

    For the most part, our Vitelity service has been good and has gotten better over the past few years with outages happening less frequently, but in 2010 it seems that we are having nearly constant issues with DIDs and clients complaining to us that people come in or call a cell and say that they cannot get through.

    Vitelity, understandably, wants alot of info to investigate. Unfortunately because the issues are intermittent and the calls never hit their switch, I cant pull CDRs to give them or be very descriptive. Clients dont track every call, nor would I expect them to. Their service should just work and issues should be an exception. I know that before we open a ticket we investigate the heck out of our softswitches, borders, and internet connectivity to ensure its not an issue with us.

    The problem is that if one person makes the effort to get hold of my client through alternate means and inform them of an issue, its probably the tip of an iceberg. People calling a business to shop around or get a quote aren't going to keep trying to call my clients back, they are going to go down the list in the phonebook or online and someone else is going to get the business my client could have gotten if their phone service worked properly.

    As resellers, our clients dot know who Vitelity is nor should they. Our clients pay us for phone service and when there is an issue, the issue is ours. Not someone elses.

    I have had a couple MAJOR issues when porting from XO to Vitelity where XO does not unbuild the number from the PRI/Tandem platform it was on when they move to SONUS for the port to Vitelity. That makes it so that they get incoming calls through us from most callers, but anyone else with XO who tries to call that client gets a disconnect message as XO tries to route that call on-net, but theres nothing at the other end as that equipment has been removed and replaced with our equipment. We usually dont find out about this until a week later, at which point its a MAJOR issue and the client is extremely upset. In every case that this has happened, Vitelity didn't get the results to fix it. I had to personally make phone calls to their NOC and sit on conference calls and even go to an XO regional office to get it fixed. This is difficult as I am not XOs customer. Vitelity is. Understanding that they are big and cant treat every client as their only client as I have the luxury of doing is hard at times. And I have broken my back fixing that kind of issue, nearly getting sued twice over it. Had I let Vitelity handle those to completion, I would likely be out of business, or had a judgment against me.

    Anyhow, is anyone else experiencing these issues with intermittent DID issues increasing as of late? I really do like Vitelity, but for the sake of protecting my brand I am considering porting new DIDs to other carriers who use level 3 or 360 networks. If this continues and I continue to get canned answers of "we aren't experiencing any issues" and the like, I may have no choice but to start transitioning to these other carriers where I have not had any issue whatsoever with incoming DIDs.
  19. wardmundy Nerd Uno

    Coming tomorrow to Nerd Vittles. :biggrin5:
  20. Bitnetix Guru

    The CIO and VP for Technology of XO sit in an office near our office. Want me to go yell at them for you? :)

Share This Page