subscribe

Path: sun.sirius.com!usenet
From: Don Hurter
Newsgroups: sirius.tech
Subject: Service update - reports from the front lines
Date: 22 Oct 1995 04:55:03 GMT
Organization: Sirius Connections
Lines: 133
Message-ID: <46cir7$c45@sun.sirius.com>

This past month we've been extremely busy taking care of the nuts and bolts behind the scenes, as well as building a foundation for expansion that we started planning as far back as February. For the past two weeks we've been performing a major clean-up of equipment and system configurations including the following; troubleshooting older modem problems, troubleshooting phone line problems, installing new backbone routers, separating Ethernet segments to better balance traffic, obtaining new IP addresses from the Internic, re-assigning addresses to older equipment, re-cabling equipment, installing new phone lines, installing new modems and Portmasters, testing new modems and Portmasters, troubleshooting new Portmasters which don't work, replacing new Portmasters with yet newer units, testing the newer units, revising the user's manual, updating Web services, ordering a redundant microwave backup link, updating dedicated clients' routing configurations, troubleshooting Ascend ISDN ethernet problems, shipping out defective equipment to vendors, waiting for replacement units to arrive, re-arranging modems and Portmasters to their final configurations when the new equipment finally arrives, de-commissioning the flawed USR modems, updating the phone line hunting to its final configuration, re-testing all the equipment, and updating all the system monitoring software to reflect all the changes.

In addition to that, we have designed a Bay-area wide network plan that will allow us to expand to new areas without falling into the traps that we've all experienced at our San Francisco location, while simultaneously improving the S.F. operations to take advantage of the grander set-up. Implementing the plan has included the following; evaluating limitations in our original service design, determining our future growth expectations (and then tripling it), investigating larger-scale network and back-bone provider options, forecasting financial figures to determine the feasibility of the various options, negotiating with numerous telecommunications carriers for long-term service contracts, searching for real-estate for remote POP equipment rooms, negotiating leases for same, designing layouts for the equipment and communications services, ordering secondary MPOEs from Pac Bell for large phone service cables and termination blocks, contracting out electrical work for high power capacity, ordering data circuits to tie the various locations together, ordering routers and other equipment, planning future IP assignments for proper routing and potential portability to other service carriers, installing the circuits and equipment, testing circuits and routers, installing modems and Portmasters, cabling equipment together, installing phone lines, testing phone lines, correcting phone line problems as delivered, testing modems, testing remote routing and authentication, and again updating all the system monitoring software to reflect all the changes.

There are countless other logistic details and snafus that entered into it as well, plus the ever-present hand of Murphy who likes to throw whole toolboxes, not just monkey-wrenches, into the mix. The items mentioned above are not intended as a boast or to seek sympathy for the amount of work involved; instead I just wanted to give an example of the complexity of the ISP business. We have put in a lot of hard hours over the past few months, and suffered a two-week setback in September when multiple pieces of equipment failed intermittently and simultaneously. You all suffered the frustration of unreliable connections or access to services, and we wore ourselves out trying to troubleshoot variables that combined beyond our ability to keep up. We also needed to make headway on the San Mateo POP that was already in the works at that time, and keep contractor's deadlines in check lest the costs soar out of control.

We were aware of the existing modem problems at the time, but since the replacement units were on three-week back order we decided to focus on the San Francisco network/IP/phone-line/router clean-up while we could. During that time I avoided any distractions and spent many early-morning sessions reconfiguring the existing modems and Portmasters while Arman implemented the final IP settings. I have even been without a computer on my desk for the past two weeks (a liberating experience, btw), and only replied to vital email when I had the chance.

The good news is that the worst is over, we've tracked down most of the problematic equipment and eliminated altogether the defective USR rackmount modems which cost us dearly both in terms of cash flow and customer goodwill. During the process of swapping out equipment we needed to keep the older units on standby in case the new ones didn't work (as happened multiple times), and as a result the physical and logical configurations in our equipment room got out of sync with the original intended plans. This entailed additional changes and service disruptions when we later re-arranged everything to its final settings, but that's done now, and the phone hunting order, IP assignments, ROM revision sequence, and cabling are finally back on track with what we tried to do months ago.

We've learned an awful lot during this growth phase about project planning, expectations of new equipment, system scalability, error recovery, and assumptions about what works and what doesn't. Our San Francisco operations were a testing ground for equipment deployment, and after one year we know that nothing short of exhaustive testing and over-designing everything will yield good results for future expansion. As a result, our San Mateo POP (and pending San Jose location) are fairly bullet-proof installations with heavy-duty power circuits, plenty of phone lines, virtually unlimited T-1/T-3 potential on a direct fiber backbone, and room to grow. (I'll describe the whole POP deployment saga at a later time - there are a few good stories in there.) Essentially we'd like to think that we have learned from our mistakes, and I have a personal inclination to over-engineer the results. As I mentioned earlier we have just now been applying those lessons to San Francisco, and to our knowledge most of the complaints over the past few weeks have been addressed.

I apologize for the information blackout in these newsgroups during this period, but we really had little choice. One can never be sure that any fix that is planned will work the first time around, and in the case of the 28.8k modems this rang true in spades. I've tried to provide status about what we're doing to correct various problems, but after a while simply saying 'We're working on it' sounds hollow. The truth was that we never really knew when we could declare a problem solved, especially when new equipment came through out of the box with intermittent defects. Instead we focused all of our efforts, every day and all day, on chipping away at the problems, and for the first time in a month I feel confident enough to say that the modem/login problems should be under control.

I'd like to ask your indulgence in light of this to once more report repeatable connection problems here in this newsgroup. I appreciate those who have tried to help out in the past by ferreting out port numbers using terminal logins, but unfortunately much of that data is now irrelevant since we have replaced a lot of equipment. The final configuration changes were performed last Friday afternoon. Any connection drops or login errors before that were most likely attributed to either known buggy equipment or early morning modem resets we needed to do during the problem diagnosis. We tried our damndest to perform most of the work without further disrupting people's connections, but in a few instances we had to power cycle equipment after waiting at least three hours for most of the users to complete their sessions and clear off the machines.

If you still experience modem trouble please note the time of the problem (very important), and if using MacPPP when you encounter repeatable errors try again using the 'Terminal Window' option, which reveals the Portmaster and port assignment for that session. [I'm not familiar enough with the various Windows dialers to describe their methods.] I'd like to start becoming more active again in these groups, and will read whatever anyone posts, but can't guarantee I'll be able to respond to everything for another week or so while we finish up our network installations. I thank everyone for their enduring patience during all this, and look forward to reporting some very good news regarding our future network plans when we get everything in place.

-- Don