Where Power8/9 really shines is memory bandwidth - it's orders of magnitude faster than leading edge Intel stuff currently. But if you're not memory constrained, Intel machines can still have a bit of an edge.
That makes Power sound very appealing. I've never met a real life scientific simulation that wasn't often constrained by memory bandwidth. I'm sure they exist, but they aren't all that plentiful. I'd love to try running the code that matters to me on Power. (For one thing, I don't even know if it would pass basic regression tests; numerical code is touchy. Are gcc/gfortran reliably adequate on Power like they are on x64?) Unfortunately, as everyone else is commenting, entry level x64 is dirt cheap and entry level Power isn't.
gcc is perfectly cromulent on Power (speaking from experience with my own PowerPC and POWER6 systems, and when I had time on one of Raptor's POWER8 systems). The speed differential between that and IBM xlc is pretty small IME nowadays, too.
I'm more hesitant to say the same about clang/llvm but it's been awhile and I'm probably not current.
It's true that raw FP performance isn't the be-all and end-all, but things aren't necessarily memory-bound either. HPC codes at scale are often dominated by communication.
I'm surprised if numerical issues are that much of a worry. x86 has now caught up in doing FMA, in particular. I don't know what the build systems actually are, but most of Fedora (and Debian?) is available for POWER, and will typically have at least minimal tests in the package. It's not as if POWER is new for HPC; the Daresbury HPC-X machine was quite high in the top500 15 years or so ago, as an example I was physically close to.
That makes Power sound very appealing. I've never met a real life scientific simulation that wasn't often constrained by memory bandwidth. I'm sure they exist, but they aren't all that plentiful. I'd love to try running the code that matters to me on Power. (For one thing, I don't even know if it would pass basic regression tests; numerical code is touchy. Are gcc/gfortran reliably adequate on Power like they are on x64?) Unfortunately, as everyone else is commenting, entry level x64 is dirt cheap and entry level Power isn't.