Leased Line Testing

[gadfly56]

I'm hoping someone here can help me out. We have a customer with a leased line connection (Verizon) connecting locations approximately 3 miles apart. The connection is for computer stations that monitor fire alarm equipment at each end. They are having communication issues such that the nodes are losing and regaining connection at about 20 minute intervals. Then they have to reset the panels/stations every time the troubles swing in and out. Although we are looking at potential equipment related issues, my question here is how can I test the leased line for end-to-end quality? I've poked around the web, and it looks like the equipment to do the testing is available for the modest (NOT) sum of $3,000 or so. This excludes, of course, the experties to use it. Are there relatively easy tests that can be done, or a way to monitor the leased line quality if only on a temporary basis? I'm not keen on taking Verizon's word that their infrastructure is in tip-top shape.

 

[zbang]

(Semi-random late-night mutterings- I haven't done fire panels or BA systems in 20+ years, so I my knowledge is old. OTOH I do know telecom.)

What kind of line is it? That is, DC loop, modem/audio signal, or digital? Are both ends served by the same wire center (central office)? Most important, is there any digital equipment in the way . At only 3 miles, it -could- be a simple pair of wires between endpoints, which would work for almost any signaling. You can test that with a battery and a light bulb. Once you're into audio signals, then you need the real test sets to get qualitative info. OTOH, plug a walkman in to one end and the other into a small amp and see how bad the music sounds.

It's going out every 20 minutes? If it's that predictable, get the telco out to test and make them test for at least 30 minutes.

A shot in the dark- does the fire alarm equipment use 1200bps modems? They're especially susceptible to 'sync-slip' on digital spans. There's almost no way to test for that without $$$$$ equipment.

 

[Sierrasparky]

Won't Verizon provide a line quality test and provide the results?

 

[GoldDigger]

If the connection includes some kind of router at each end it would be simple to inject some test packets at each end and monitoring them at the other end. That will tell you whether there is an underlying data loss problem, and check the timing characteristics of it.
Ultimately it would then be Verizon's problem to figure out where the problem is occurring.
If there is no way to access the data link, then you would have to disconnect the normal equipment to do the test.
But perhaps equally useful would be passive, putting a network packet analyzer on either end and looking at error rates and packet distortion. Again, the ultimate resolution will be up to ,Verizon.
And it may turn out that the problem is with the configuration of the end equipment rather than the line itself. A network analyzer trace would show that too.

 

[tom baker]

I have a few leased lines left, they are analog audio tone. I would test these with a signal generator, use a 1,000 hz signal generator, set for 0db and use a scope or frequency meter at the other end, you should have about 16-20 dB loss.
Its a dedicated pair one end to the other or does it go thru the central office?
However, Verizon should be able to test this circuit, but I have had many end users tell me and my experience is the techs that used to work on these once very common circuits are retired. Doesn't matter what equipment goes on the line, the line should have no more than 16 dB loss at 1000 hz

 

[fmtjfw]

If they are T-1 lines, also might have sync-slip. Ask Verizon to get their oldest tech, or the retiree they call in when the young whippersnappers are stumped.

If you are using modems that can be "managed" look at the error logs, there may be clues there.

 

[hbiss]

I too wouldn't believe what Verizon says for one second. If I were the OP I would sub out the testing to a telecom company and let them handle it and fight with Verizon. The only way to get satisfaction these days is to be there with the Verizon tech (vendor meet), know more than they do ( that's easy ) and have the test equipment to prove they are wrong. If the OP can't even tell what kind of line or service it is, any discussion about testing is pointless.

-Hal

 

[gadfly56]

Thanks for all the replies. In answer to some questions:

What kind of line is it?

Not sure. It's copper to the demarc at each end, but beyond that I don't know. The buildings are in different exchanges, but I don't know if they might still be located at the same central office. It's supposed to be a dedicated dry pair, but we're pretty sure it isn't. With the tip and ring open at each end, there is about 100 ohms at each end. If you short either end, the opposite still reads 100 ohms. Not looking good for copper end-to-end, although according to my techs you can push a tone from one end to the other. I don't believe it's a T1 line either. There's no need for that bandwidth. The modems operate at 19.2K baud.

Won't Verizon test?

Sure they will, and have several times. According to them, all is unicorns and rainbows. I've never seen a report so I don't even know what they are claiming, exactly.

Routers, network packet analyzers, etc.

This is a strictly analog set up. Tip and ring connected at each end and that's it. The short haul to the computer at each end is 20 - 50 feet.

Next Steps

I've spoken to one of Siemens top trouble shooters, and he said he's seen this sort of thing before. In all likelihood the signals are going digital at some point, which means (if I understand correctly) multiplexing and switches. Now for an important bit of info. The modems are set up to adjust their speed if they detect connection issues. So if the switch gets jammed, they'll drop the connection speed. When traffic lightens up, the modems don't always come back up properly, and this is where zbang's observation regarding slip sync may come into play, especially if the modems are driven down to 1,200 baud by an overstuffed switch. The Siemens engineer said what he has done in the past is tweaked the modems to connect at 9600 baud, period. This has solved the problem for at least one customer in Union county that has a massive number of leased line connections to their central site. Any time they've had an issue after that it's been traced to a Verizon issue.

So Tuesday we have a party planned for the customer's sites. We have a contractor who is very familiar with this gig and has done this sort of work for the Siemens branch. He'll bring in a couple Siemens panels with the tweaked modems and check if that clears the issue. For now we wait with bated breath.

 

[GoldDigger]

Is the digital information being sent through the modems synchronous or async?
If it is a sync, it may not survive a change in modem data rate without proper flow control.
If it is sync, the end equipment may be trying to send data faster than the line can support and leading to a buffer overflow and loss of protocol communication.
I do like the idea of substituting fixed speed modems, since it will both check whether the line can handle the lower data rate (no intermittent problems) or that the equipment cannot tolerate the lower data rate (immediate problems, not with hour(s) between failures.)

 

[gadfly56]

Well here we are, a little further into our saga. After much waving of rattles and reading of entrails, it looks like Verizon has finally 'fessed up that the lines were not properly configured for the intended purpose. It leaves open the question of whether they were ever configured properly in the past 5 years, but a new ticket has been opened and we hope to see additional progress this Wednesday. Stay tuned, same bat time, same bat channel.

 

[Sierrasparky]

Well here we are, a little further into our saga. After much waving of rattles and reading of entrails, it looks like Verizon has finally 'fessed up that the lines were not properly configured for the intended purpose. It leaves open the question of whether they were ever configured properly in the past 5 years, but a new ticket has been opened and we hope to see additional progress this Wednesday. Stay tuned, same bat time, same bat channel.

Ya know I just don't get companies like Verizon. Why they refuse to look at their problems and make proper assesments.

Well I have the same issues with Verzion in my neck of the woods. Wether from leased lines or from DSL circuits. Verizon never claims there is a issue. "every thing tests ok"

We'll then you have their DSL support. Your speed is great, we just ckecked your connection. My speed is slow at my end I say. I ask what speed do you see I ask of the Tech. I see you are getting 3-7 . Which is it I say I have never seen a speed test with such variable speed. The tech responds you are getting the speed for your plan. I ask so what is that as you have just performed a test.
you are getting 5433 down says the Tech. I reply that is the Sync speed of my modem not the true speed out on the net. After much back and forth the tech responds you are correct that is the sync speed and not a speed test. I go on further to tell them that my usual sync is about 8550 and my speeds are about 7.3 down and I ask why did you lie to me. The tech reply was i am sorry I will send a tech out. I told him I had checked the line from my end and all is well. I am only 4k feet from the mini CO.

They field tech called while at the CO and said I am going to change a connection and you should be up very soon. So in about 20 min I was back up to full speed and better and they never came to the office. All fixed at their end. This happens about 1 time a year. Each and every time the problem is at the Verizon DSLAM.

you might ask why do you not change to cable. Well Cable is not available on my street. I could push for it as the speed would be much greater. However the Cable is always going down in a storm and the Telephone and DSL is always up but sometimes slow.

Anyway just ranting over the poor tech service from many companies lately.