One of our physical servers experienced a low-level networking failure related to a very rare expression of a bug in the kernels network modules. This error, unfortunately, took down several critical subsystems that are required for proper functioning.

As a result of this failure, we are working toward re-evaluating the distribution and redundancy of the impacted systems so that this rare, but ultimately quite large, failure mode will not be encountered in the future.