90% of the power in our academic data center goes 13.8kV 3-phase -> 400v 3-phase, and then the machines run directly from one leg to neutral (230v). One transformer step, no UPS losses, and the server power supplies are more efficient at EU voltages.
But what about availability? If you ask most of our users whether they’d prefer 4 9s of availability or 10% more money to spend on CPUs, they choose the CPUs. We asked them.
There are a lot of availability-insensitive workloads in the commercial world, as well, like AI training. What matters in those cases is how much computing you get done by the end of the month, and for a fixed budget a UPS reduces this number.
> and then the machines run directly from one leg to neutral (230v)
And then every machine has a switching power supply to convert this to low-voltage DC, and then probably random point-of-load converters in various places (DC -> AC -> DC again) for stuff like the CPU / GPU core, RAM, etc. Each of these stages may be ~95% efficient with optimal load, but the losses add up, and get a lot worse outside a narrow envelope.
Yes, but it's not like any other layouts avoid those issues.
You could feed your servers off fat 12/24/48 volt supplies but with how much power a modern server can pull you're already converting in bulk even if you don't do that, limiting the potential advantages. For running CPU/GPU/RAM, there is no other option. When you need hundreds of amps at 1-2 volts, you convert that centimeters away if at all possible.
A datacenter using DC distribution is still using high voltages and stepping them down in layers. The hassle it avoids is in other aspects of power delivery.
I was part of an electrician team renovating a three-letter agency's main data center (adding UPS to an archaic government renovation). As part of this process, we had to isolate the Emergency Power Off system/buttons.
Unfortunately, the EPO ended up having been installed differently than the plans indicated [0] — there was definitely more than an hour of unexpected downtime figuring out how to "unrig" its decades of faithful misconfiguration.
During this same months-long modernization, a separate facility decided to install a temporary tap off our already-temporary taps (no engineering consultation attempted) — with resulting Megawatts of melting disaster. More hours downtime.
tl;dr: shit happens
[0] EPO's main neutral was tied into a general lighting circuit's j-box, some bullshit jerryrigging from an 80s sparkie; our apprentice was tasked with taking out old fluorescents... ended up taking out entire facility (through no fault of his own).
But what about availability? If you ask most of our users whether they’d prefer 4 9s of availability or 10% more money to spend on CPUs, they choose the CPUs. We asked them.
There are a lot of availability-insensitive workloads in the commercial world, as well, like AI training. What matters in those cases is how much computing you get done by the end of the month, and for a fixed budget a UPS reduces this number.