Jason Mader

NCAC network


The VA campus has one unreliable DS-3 microwave link and one T-1 to the Foggy Bottom campus, and each LAN has a 100Mbps full-duplex interface to the firewall I manage. DS-3 statistics are provided by ISS Technology Engineering on montag.nit.gwu.edu (to only people with passwords) but still nothing for the T-1.

Links to our own network maps and statistics can be found here.

Network outages

2001-06-22: Electrical storm knocked out power to the Virginia Campus taking down the WAN equipment. Outage lasted for 14 hours.

2001-10-17: All GWU.edu authoritative nameservers were missing glue for the cisr.gwu.edu delegation preventing CISR names from being resolved. Reported to be caused by a syntax error in the db file.

2002-02-21: Network 24 was withdrawn from the routing table, isolating it from all but the local networks. Outage lasted for 16 hours.

2002-05-27: Thunderstorm briefly knocked out power to the Virginia Campus. Interruption of external resolver and cache for the 24 subnet and on the NCAC ESX-2400 ports which are not on UPS.

2002-06-13: Disconnection of network 24 from the router for 22 minutes during a network diagnostic event.

2002-08-01: After a brief WAN failure at 9:20am, another for 1 hour 20 minutes. Interruption blamed on Verizon plugging a T1 interface back into the wrong circuit.

2002-09-08: A hardware failure in NCAC's ESX-2400 caused ATM hosts and some Ethernet hosts to be disconnected. Outage lasted for 23 hours.

2002-12-16: Midnight to 1:30am both NCAC and Virginia Campus mail servers were disabled to transfer the accounts from NCAC to mail.va.gwu.edu.

2003-01-25: Aprox. 10:00am to 2:53pm the nva2424M-2 switch failed. A reset corrected the outage for 2nd floor devices.

2003-03-27: 7:33am the 24 subnet interface was accidentally disconnected by a telecomm technician. Outage lasted for 1 hour 7 minutes.

2003-05-23: 8:29am the mail server unexplicably rebooted. Back up in 9 minutes after checks on the mail spool disks. Starting at noon the mail server began to fail repeatedly; the system was taken offline and the mail spool was copied to new hardware. Mail services were restored at 6:20pm.

2003-07-10: Aprox. 2:00pm to 4:00pm the whole area around the Virginia Campus lost power and everything not on EM power had to be shutdown, including the mail server.

2003-10-31: 9:56am Jeremy Blum accidentally disconnected the cable for the 24-subnet while moving furniture into 330A. The firewall will be moved expeditiously to avoid a recurrance. Outage lasted for 5 minutes.

2003-12-09: Mail spool problems start at 12:43am. 4:18am an unexpected filesystem problem took down the CGatePro process until the mail server could be restarted and the filesystem repaired. Outage lasted for 2 hours 27 minutes. I traced this to a recursive file duplication of a large message folder during the daily AVP scan.

2004-02-03: NCAC Desktops VLAN was down while I changed routes to speed up inter-VLAN routing. I typo on an addressed created a routing loop that took me some time to find.

Network malfunctions

2001-10-19: ns.gwu.edu stopped returning glue for blazer.va.gwu.edu. Fixed within a couple hours of the origin.ncac.gwu.edu glue expiring, then declared by NIT to not be a problem. djbdns mailing list post on the matter.

2002-05-02: Traffic ceased to be sent over the DS-3 during a severe storm. Reported to help desk, case 30045. The link was down for eleven days.

2002-05-21: An OC-12 trunk on the NCAC network become misconfigured after a switch reboot making a loop which interefered with normal network operations, including failure to reach the external resolver and cache.

2002-12-02: IP renumbering of the VML at 8:30am didn't go smoothly because ISS did not update access-list's with the routing on the Virginia Campus border router. Fixed at 1:45pm.

2003-03-12: Slammer worm on a host at the Virginia Campus opened enough simultaneous connections to fill the firewall active table and prevent most incoming and outgoing connections from about 3am to 10am.

2003-03-15 to 2003-03-17: 2nd Floor 2424M switch experienced system: Out of pkt buffers error causing intermittant network failure on Saturday, and complete failure on Monday. The network topology is going to be changed to bypass this switch.

2003-04-23: ISS engineer admits traffic shaping is to blame for performance problems to the Building 2 network.

2003-07-12 to 2003-11-10: ISS engineers lengthed the building 2 uplink Ethernet cable to 157m, exceeding the maximum segment length of Ethernet (100m).

2003-12-02: Building 2 networks cannot reach Internet hosts; fixed after new GE links were failed over to the DS-3. ISS doesn't know why this happened. Outage lasted 3 hours.

VA and SEASVA networks

The backbone of these networks is made of HP Procurve 5308xl, 2424M and 4108gl Ethernet switches. 802.1Q VLANs are in use on the network.

When connecting to the VA campus or SEASVA LANs use DHCP to obtain network configuration. SEASVA IP addresses must be requested prior to connecting.

Network Operations personnel for The George Washington University Virginia Campus can be reached by e-mail at noc@va.gwu.edu. There is a web based Web Help Desk available for technical support.

Virginia Campus Internet services group

Infrastructure-grade network services provided to the VA campus are DNS, DHCP, Internet messaging, network time, NetBIOS name service, printer spooling and PPP dial-up access.

The services are monitored and the group is part of the security response.

National Crash Analysis Center

I maintain a Gigabit Ethernet network for the FHWA/NHTSA National Crash Analysis Center. Ethernet edge connections are provided by the HP 4308gl and HP 2650. Our high performance network core is the Force10 E300.

Network Operations personnel for the FHWA/NHTSA National Crash Analysis Center can be reached by e-mail at noc@ncac.gwu.edu.

All IP addresses used on the NCAC network must be requested in person by providing the Ethernet address and hostname. No access is allowed until confirmation is sent by e-mail.

Some network ports might require registration and 802.1X supplicants to access.

Use your mail server username & password. Authentication is provided by Radius using EAP-MD5. Comments about EAP-MD5: Similar to CHAP and prone to identity exposure, dictionary attacks, session hijacking and man-in-the middle attacks.

In general, EAP-TLS is stronger.

Wireless Networks

Currently I have wireless access points providing limited coverage for students in Classrooms 232 and 305 and in the Virginia Campus Library, and maintain instructions for wireless access at the Virginia Campus.

A University notice about wireless networks can be found here.

Channel information:
Classroom 232 and the Virginia Campus Library access points use channels 1-5. The AOL lab also uses channels 1-5. Classroom 305 uses channels 11-14. The NCAC offices and NCAC film library use channels 6-10.

A local wireless broadband point-to-point service uses channels 2 and 11, and 6 for their backbone. (6 will be going away soon).

Press

AOL, GWU Create Wireless Research Lab, April 2001.

Other resources

MAX | Mid-Atlantic Crossroads
Top Ten Blocking Recommendations Using Cisco ACLs
Top 20 Vulnerabilities