subscribe

Path: sun.sirius.com!usenet
From: Don Hurter
Newsgroups: sirius.tech
Subject: 14.4k modem busy signals - mystery solved (hopefully)
Date: 5 Oct 1995 03:17:00 GMT
Organization: Sirius Connections
Lines: 97
Message-ID: <44vinc$oek@sun.sirius.com>

Up until last night we continued to get reports of busy signals on the 14.4 modems, even though there was an entire bank of 24 that rarely got hit. We test our phone line hunting by calling a number in the pool, which causes a modem to pick up to see who's there. Then we immediately call the same number again with a second modem to confirm that the call hunts to the next available line. If the hunting is broken from PacBell (a not entirely rare occurrence), we get a busy signal if the call doesn't roll over. If everything works, we reach the next sequential line.

Then we call the next line following the one we just tested, and repeat the procedure. If all the hunting works properly we can repeat this for every modem line until we wrap around to the first line. This shows us that there are no dead ends in the hunt groups which block any further hunting somewhere.

However (there's always a 'however' in these things), we have ten broken 14.4 modems that won't issue a login prompt unless the user issues an extra carriage return, and we needed to pull those modems offline. There is a Centrex feature which allows us to forward any given line in the hunt group to another line other than the one following it in sequence. (Centrex is the phone company software that gives our telephone service some intelligence, including the hunting itself.) We use the forwarding feature to jump incoming calls over the ten bad modems, otherwise people would get ten long ringing sessions as the call passed from one line to another if we simply unplugged the wires. BTW, we cannot simply re-arrange the hunting order at will, as this is something PacBell 'hardwires' into the phone service, and each change costs us $25 per line, and usually takes two to three days to implement. You can see that once we order a block of phone lines and add them to the end of the hunt group, they are there to stay unless we incur a lot of expensive service changes.

On top of this, there are always a few lines (say two to three out of over two hundred fifty) which have excessive noise, or broken hunting features, or any other of a variety of general telco maladies, which we constantly need to have repaired by PacBell. Lately we haven't had time to set aside the day or two it usually requires to report the trouble, have a service tech show up to repair the line (sometimes they can't because of problems in the Central Office which require equipment changes), test the repairs, and then close the trouble reports. Instead we simply forwarded calls over those bad lines and have them on a to-do list when things settle down.

The result of all this is a number of forwarded calls hopping around in the hunt group, which are a pain to keep track of and even more of a pain to test around. Nonetheless, we have called every single modem line and verified proper call hunting multiple times, yet never figured out why people get busy signals when there were 20 idle modems patiently waiting in the racks.

Last night I discovered what had happened. There is a limit built into Centrex service which only allows so many call forwards before the switch at the CO loses track of the call. On AT&T equipment that limit is something like 15 or 20 forwards, far more than we have temporarily programmed. However, our switch in the South of Market CO is a Northern Telecom DMS100, which only allows 5 forwards maximum. Now as long as the incoming calls simply hunt along for an available modem, there are no forwards involved. But as more of the pool gets used, calls will eventually hit forwards that we used to jump over dead lines. Finally, calls will fill up the pool until they hit the 10 bad modems at the tail end, at which point they should forward to a new block of 24 modems. But as luck would have it, that particular forward is number 5 in the list, which means that the switch overloads and people get busy signals rather than arriving at the new modems. Note, however, that this behavior never manifested itself when we tested the hunting, as we would perform that test during slow periods and we'd always reach intermediate lines before encountering all five forwards. The only time the system would break down was when we needed it most, namely when all the modems were in use!

As a customer you shouldn't need to concern yourself with these problems, nor even need to be able to understand the above. But the five-forward limitation is not included in any documentation we received from pacbell, and the only way we discovered it was the hard way (and at your inconvenience.) There's a lot of talk in the ISP newsgroups about growing pains that all the services suffer, and we've been bitten by our share of bugs. We never considered that the service we receive from PacBell would have these sorts of scalability hurdles, yet the only way to learn it is through experience.

In the meantime, we have new phone lines on order to supplement our existing number, which are due to arrive on Friday. As a side note, we ordered these lines back in May, and it has taken PacBell this long to deliver them, primarily because they had to dig up the streets to run the new cables. Up until now we have been riding it out on a surplus of 100 lines that we ordered at the last minute last spring before all the neighborhood's phone wires got used up. (At $80 install per line, plus our existing service, you should have seen our phone bill that month!) Simply managing the telephone service here or at any ISP is a full-time job, and it is all too easy to fall behind when juggling new orders, existing outages, and new services such as ISDN and Frame Relay. I've been trying to catch up on a lot of this and more over the past week, and had the workload compounded by the previous weeks' 28.8k modem/portmaster fiascos. I frankly haven't been able to even launch my newsreader to check these groups until most of this gets squared away. We have placed ads for assistants who can deal with some of this as well to help with the ongoing tasks. The ISP's job is never done.

-- Don