digital.forest Technical Support
Breaking News

******Non-Impacting Facility Testing******

On Saturday, March 13th between 07:00 and 12:00 hrs. we will perform a number of tests on our emergency power system. These tests will not have any direct impact to the operation of the facility.

We have coordinated with our vendor partners, and will have a full compliment of df staff on-hand to investigate the issues we've experienced during Seattle City Light's attempts to upgrade the high-voltage transformer that supplies our facility.

We will cycle our automatic transfer switch (ATS) a couple of times during this window in order to provide facility load to our roll-up generator. Throughout this process, we will be monitoring our HVAC equipment and performing extremely detailed power quality monitoring on the generator power to help isolate the root cause of the unusual behavior we observed last Saturday.

We will provide an update here following the completion of these tests.

************ digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

posted by Shawn Hammer at 02:52 PM on Tuesday, March 9, 2010
Categories: Facility Maintenance

******Update 3/6/2010 14:10******

This maintenance has again been postponed due to rental generator related issues. We will reschedule and provide an update with the new date once it has been determined.

As a result of the generator related issues, some of our HVAC equipment experienced a power quality issue (client equipment would not be impacted as all of the power to the data center floor is scrubbed in our UPS systems). The result was some increased temperatures in some of our data center spaces. digital.forest operations staff was on-hand to mitigate and we resolved the issue quickly. Some clients may have noticed high temperature alerts on their monitoring systems but no supply temperatures should have exceeded 80° F. We are continuing to monitor our HVAC systems to ensure proper operation.

posted by Shawn Hammer at 02:18 PM on Saturday, March 6, 2010
Categories: Facility Maintenance

******Update 3/6/2010 06:15******

We have postponed the start of this maintenance until today at 12:00 noon due to issues with one of the rental generators for the building. The duration of the work will remain the same once we begin.

******Update******

The correct date for this maintenance is Saturday, March 6th.

******NON-SERVICE IMPACTING MAINTENANCE******

On Saturday, March 6th Seattle City Light will perrform an upgrade on the transformer that provides power to the digital.forest data center. During this maintenance our facility will operate on generator power and will have a second generator in place for redundancy. This maintenance will provide additional power to the building and our facility. Our facilities management team has planned the work with representatives from Seattle City Light, Veca Electric, Sabey Corporation, and our neighbor tenants. We have planned contingencies for staff shortages, foul weather, time over runs, and broken or damaged parts. As always, we are taking all necessary steps to ensure continuous service to our customers and life-safety for everyone working on this project. The process for this work is very similar to the process used during the September 4th, 2009 work which you can read about here and here.

****** The schedule for work is as follows:

04:00 to 04:45 digital.forest staff will pre-flight and warm up our generator

04:45 digital.forest staff will transfer critical load from utility power to generator power

05:00 Seattle City Light will de-energize the main transformers feeding our building

05:00 to 14:00 Seattle City Light will replace on of the main transformers feeding our building

16:00 Seattle City Light will re-energize the main transformers

16:00 to 16:30 Seattle City Light and Veca Electric will validate the voltage, frequency, and rotation of the power from the new transformer

16:30 digital.forest staff will transfer critical load from generator to utility

16:30 to 16:45 digital.forest staff will monitor power conditions from the new transformer and cool down our generator

16:45 to 18:30 digital.forest staff and Veca Electric will remove the wire connections from the second generator and clean up the operations area

************ digital.forest remains committed to providing our customers with the highest level of service, the greatest degree of protection, and the most transparent communications. If you have any questions or concerns about the above maintenance, please contact your account manager. Our account management staff is available Monday through Friday from 08:00 hrs PST until 17:00 hrs PST at 877-720-0483 Option 2.

posted by Shawn Hammer at 06:20 AM on Saturday, March 6, 2010
Categories: Facility Maintenance

During the scheduled maintenance window on Friday, February 26th, we experienced two intervals of network instability, each resulting in significantly reduced connectivity. Our network experienced increased latency and in many cases servers being unreachable from certain external networks. While technically not a network outage, we are aware that many of our clients were seriously impacted from approximately 01:00 until 01:10, and again from 02:05 until 02:35.

The root causes of both network events were directly related to configuration conflicts within our new core routers. These conflicts were very different in nature, and manifested themselves in very different ways. Following each of the impacted intervals we successfully isolated and mitigated the specific configuration conflict. Although unrelated to each other, both of the configurations that caused these issues are essential to the safe and stable operation of our network.

Our ongoing mission is to maintain, and strive to improve, the reliability and performance of the critical systems that are the hallmark of digital.forest's service. As stated in our support blog post of February 24th:

"In February of 2009, we commenced a comprehensive analysis of our current network infrastructure, comparing it to what are now the most trusted and efficient network technologies available. Appraising our current and projected future requirements alongside available mature technologies, we initiated the process of testing numerous network hardware and configurations. "

"Utilizing current models of Cisco carrier-grade network equipment, our new core infrastructure is designed to be fault-tolerant, easily expandable, fast and flexible."

We are constantly evaluating our processes and protocols, and as a direct result of client feedback we are implementing a new notification system to provide significantly improved advance notification of upgrades and maintenance. digital.forest is also working rapidly to provide a truly fault-tolerant support website and helpdesk. We will provide a direct announcement, including any necessary instructions, to all of our clients as soon as these services have been enhanced.

The entire staff at digital.forest wishes to express our sincerest regret for the inconvenience to our clients and their customers during our network upgrade. Be assured that we will continue to work tirelessly to improve both the systems and processes we employ to deliver the service and support that you have come to expect.

posted by Shawn Hammer at 03:33 PM on Tuesday, March 2, 2010
Categories: Network