Computer Services’ next few maintenance periods will focus on taking steps to prevent a potential collapse of the university’s internet network, like what happened during the summer break.
The most recent maintenance period began at 10 p.m. on Sept. 21 and concluded at 2 a.m. the next morning. The maintenance impacted services like Blackboard and restricted access to computers in the TECH Center.
On Aug. 8 around 12:06 p.m., many Temple services lost contact with the university’s computer network. Everything was restored to normal hours later at 3:46 p.m.
While the network was down, TUportal, TUmail, and Self Service Banner were unreachable. Voice over IP phones in Morgan Hall and University Communications’ offices above the TECH Center were down as well, since those phones make and receive calls through an internet provider.
August’s spontaneous network failure was the fault of two manufacturer bugs in the system which hit in quick succession.
Information perpetually runs through the network in segments called packets, but on that day an unrecognizable, or “poison” packet affected the core router, which facilitates internet requests from users on the university’s network.
“The system didn’t recognize [the packet] and it caused a bigger problem,” said Timothy O’Rourke, vice president of Computer Services and chief information officer.
When Computer Services rebooted the system to troubleshoot, a second bug hit. This occurrence is rare, officials said.
“Because of the size of Temple and the amount of work pushed through [the system], a one-in-a-million bug hit us,” said Larry Brandolph, associate vice president of Computer Services.
There were an estimated 5,000 people on Main Campus at the time, a far cry of the roughly 32,000 students on campus during the fall and spring semesters.
A system status report was updated within about a minute of the network going down to inform the community what was happening. The Computer Services’ website was also updated immediately.
No TU Alert was sent out because there was no danger to anyone on campus, Brandolph said. TU Alerts are not connected to the affected network and are operated independently.
“We weren’t asking anyone to change any behaviors,” Brandolph said.
The network is also used by Temple University Hospital. The hospital was impacted by the network loss, which was considered a serious problem because the hospital is fully running at all times.
“We considered it an emergency,” O’Rourke said. “And treated it as such.”
A team of 10 people from Computer Services worked on fixing the network for nearly four hours. The troubleshooting process involved “all hands on deck, including the provider,” O’Rourke said.
“The response-time process grabs key resources that are necessary to start troubleshooting,” Brandolph said. “[There is a] crisis call to as many people as possible. The goal is always to restore services as soon as possible.”
Avaya Services, the provider for all the university’s networking equipment, has been fixing bugs in its system and will be implementing these changes in the future.
“We are very cautious of student activities, so we try not to make any big changes,” Brandolph said.
In the future, if the system ever recognizes the same bug or “poison packet” again, the system will discard it. Whenever there’s a technological problem, a system status report is instantly posted to TU Portal to inform people of the issue. The system status report history can also be viewed.
“We have state-of the-art technology here at Temple,” O’Rourke said. “Every piece of technology has issues and we deal with it as it comes.”
Lian Parsons can be reached at lian.parsons@temple.edu
“a system status report is instantly posted to TU Portal to inform people of the issue”
In reality, the System Status was not updated until much later. I was on campus at the time, you were in fact able to still access some websites (System Status included) while most things hosted under the Temple domain were completely inaccessible. They were not accessible from outside the college’s local wired connection, though.
There was no System Status update, and if there was, no one could have accessed it unless they were at a Temple-owned computer.