Monday, February 24, 2014

RE-UPDATED: New and different FIOS issues in League City TX

February 26, 2014:  The post below describes the apparent failure of a Verizon FIOS "backbone server" called ALTER.NET, a problem that Verizon's CSRs report is currently affecting subscribers all over America.  The resulting outage is not limited to Galveston County or the upper Texas coast.

No information is reported to be available on when this will be corrected.  Performance issues attributable to alter.net had reportedly been experienced by some FIOS users for about a year prior to this current incident, the impacts of which are detailed in the post below.

March 24, 2014:  We are still having intermittent failures within the FIOS network, enough to bring the normal flow of my work to a complete halt because of internet inaccessibility (I work largely from home).  The failures originate with ALTER.NET and VERIZON-GNI.NET.

April 4. 2014:  I still have an open service ticket on this problem.  I'm at the point now where I'm trying to get Verizon to tell me *exactly* what it's doing to rectify it.  I did learn yesterday that Verizon automatically closes service tickets unless they hear back from a customer within a set period of time (I forgot to ask what those limits are).  They closed my March 24 ticket after taking no action for that reason (I don't recall a contact attempt, but I lead a busy life so maybe I missed it).

July 20, 2014:  Not only do the problems continue, it now seems to be increasingly difficult to obtain partial refunds from Verizon for service not rendered.  See this post.

***

Original post:

I alerted on January 23, 2014 about FIOS speeds falling to intolerable lows.
This was a typical result from last month.  I was paying for ten times this amount of download capacity.  
A month later, the speed issue appears largely fixed, but now it seems we have a new and different problem involving time-outs of certain website calls but not others.  Some websites I can load reliably, others not at all, even though I can see from my cell phone that those same websites are fully functional.

I don't know how many local customers are impacted, so I'm posting this just in case there are others in my area for whom this info might be useful.  It took me an hour on the phone with Verizon to get to the bottom of this latest problem, so if you are affected, I'd like to save you from having to reinvent this wheel.

The effects are bizarre - for instance, I can load the homepage of Galveston County Daily News, but numerous of the sub-pages hang and time out.  When I dial into the FIOS helpline, one of the first questions asked is, "Do you have the ability to connect to the internet?"  That's usually an I/O question, but not in this case.  It's hit or miss.

My technical skills are not very advanced, but a FIOS technician walked me through the use of the Tracert command to find out what was wrong.
It means "trace route".  Explanation screengrabbed from Google.  I'd add more links to this post but my access to the internet is severely restricted at this point.  
The process goes like this.
  1. You use CMD in the main search box for Windows and it brings up a DOS box.  
  2. Then you can enter the tracert command followed by the name of the website that refuses to load.  Here's an example of the results.

Here's the basic identification of the hops for a site that I cannot access, as those hops were related by the FIOS technician (this may not be completely accurate but you should get the idea).  With respect to the numbers in the left-hand column:
1 is my internal network
2 is the equipment installed in my house by Verizon
3 is the line terminal in my street
4 is the fiber terminal in an office located in my city - essentially, their router before the signal goes out to the open internet
5 is the first external location through which Verizon's signal is routed
6 -11 are additional external routing nodes before it finally gets to the site I originally wanted to access
You can see a 103 millisecond result in Step 5.  That tells Verizon that there's a serious problem with this part of the transmission process because it should be much faster than that.  With those kinds of speeds, nothing will load because it will time out before it has a chance to do so.

Compare that DOS screengrab above with this one, which is for one of the websites that I can access:
No such huge millisecond number appears anywhere.  
Verizon can't fix this kind of problem quickly.  After spending an hour with them on the phone, they told me that they basically had to set their technician to tracking down this mysterious bottleneck in order to figure out what's going on.  Various Whois queries placed the offending interim ISP (which is called alter.net) variably in New York state or (gulp) Beijing, China.  I only get access to half the internet while Verizon's internal investigation drags on.

Anyway, just another day in consumer paradise.  If you have a hang problem, you can try that method above to see if that's the source of the issue, or simply call your Verizon FIOS rep. Good luck and remember that every service failure warrants a corresponding offset on your bill (in my opinion, anyway).

Update February 25, 2014:  I'm including this additional example for Verizon's reference because the CSR I worked with yesterday told me that, for security purposes, they don't have the ability to receive emails.  So they can look at this on the internet instead.

Additionally, I am noting for the record that there are also multiple other references to Verizon's problems with alter.net on the internet (e.g., here and here), including this one from Verizon's own FIOS forum which I will reproduce below because content like this tends to disappear from the internet and I want future searchers to be able to see this for reference.

When I called the FIOS help line yesterday, their own CSR knew nothing about this existing issue with alter.net, an issue that has reportedly persisted for close to a year now.  I do believe from the CSR's demeanor that he was truthful about that.  When he saw how my trace was progressing, he was extremely surprised.  What this suggests to me is that Verizon is not even informing its own CSRs about known existing problems.  What is the sense in that??  Not only does the resulting inefficiency waste my time, it also wastes the time of their own employees.

This forum poster's speculation about Verizon's use of alter.net to throttle bandwidth is provocative.  It's pure speculation, but look at that same question within the context of what's happening to at least some of us FIOS subscribers in League City Texas:  We had shockingly bad problems with download speeds last month, as evidenced by my screengrab at the top of this post.  My FIOS service was running at 5 Mbps; for comparison, my cellular modem this morning runs at 11 Mbps (I just checked it).  FIOS was only half the speed of a cellular modem, if you can imagine that.  And then after weeks and weeks of screwing around with no results, Verizon solved our speed issue - but immediately after they did, this alter.net problem began to manifest in its place.

So this begs the obvious question:  Was the use of alter.net the method by which Verizon solved or partially solved its bandwidth problem in our area?  By routing some of us through a de facto throttling site?  We have no proof, but it sure is one hell of a coincidence, isn't it??

Anyway, here's a partial reproduction from the FIOS forum for future reference:

 alter.net issues continue for months - verizon doesn't care
I have had multiple reoccurring issues with Fios and alter.net here in NYC for the past 6 months. Upon searching the web I see it's a reported problem by other users. I have 150/65 and going to multiple sites that pass traffic through alter can be extremely slow.
Talked with Verizon and they won't do anything about it. They will ask you to run a speed test through the Verizon Fios site. Of course everything checks out fine since it is one hop directly to their servers. They will say that anything else is third party and won't help.
My suspicion, though I have no facts to back it up, is they know about it but perhaps use it as a way of limiting your bandwidth. Primarily this happens on the weekends and from 11am to 1am week days. Anything that doesn't route through alter.net works at full speed.
Perhaps it is also that other ISP's pay for only a limited amount of bandwidth which jams up the highway.
I don't have enough technical knowledge to know but my traceroutes show the problem clearly.
For those users thinking about switching to Fios you should do traceroutes to the sites you most often use. If they hop through alter.net…. find another provider…….or stay where you are if it works well……

1 comment:

  1. Alter.net is Verizon's Business network backbone. All the new Fios connections they've been putting in, in NYC have caused a huge bottleneck because they're overselling and not expanding the network.

    I'm currently in MA and a ping to a server in NJ is 120+ms, unacceptable!

    ReplyDelete

I'm forced to moderate comments because the spammers have become too much for me to keep up with. If you have a legitimate comment, I will post it promptly. Sorry for the inconvenience.