[TowardEX NOTIFY-ALL] Incident Update MTC67012 (Power failure 1 Summer St)

[ip-admin at twdx.net]

Hello,

As a number of customers are already aware, this morning at about 10:43 AM, we had
experienced a failure on one of the power circuits feeding a shared colocation
cabinet. Some customers were affected by this failure until the problem was fully 
resolved.

During the power failure on this particular feeder circuit, the circuit breaker
that is manning this circuit did not trip, however the PDU located downstream had
lost input power.  While the issue was being investigated, we had activated a new
feeder circuit from the distribution panel to the affected cabinet PDU, restoring
power to all affected equipments.

After our continued investigation, we had found the cause of the problem located
inside the junction box "P2CX274", which was installed by an outside electrical
contracting company back in 2007.  Neutral wires were not securely spliced with 
twisted-wires, but instead a wire-nut was simply screwed on top of the two wires.
The wire-nut had come lose and splice holding the neutral wiring had failed, 
thereby opening the circuit and causing loss of power to PDU #DS4.

The loose splice had been corrected and we are working with building management to
schedule a maintenance window to visually check all junction box splices that are
feeding TowardEX datacenter power circuits.


This incident may also be a good opportunity to make sure that all of your critical
server equipments have redundant power service installed -- you will need to buy a
redundant power supply (unless you have one already) to order redundant power 
service.  Affected customers with redundant power service did not go down as a result
of today's outage.


Again, we sincerely apologize for any inconvenience this incident had caused. While
the cause of today's problem is determined to be a faulty workmanship by a licensed
contractor, it is indefensible and we apologize sincerely for customers that have
been affected by this outage.

If you have any questions or concerns relating to this issue, please do not hesitate
to contact us at 617.849.7278 option 3 or via email to support at towardex.com.

---------

TowardEX Technologies, Inc.
Network Operations Center
Phone: (617) 849-7278 option 3
Fax: (432) 225-3784
Email: ip-admin at twdx.net
INOC-DBA: 27552