We were using Xvfb with PhantomJS. Moving to a headless browser gave us the advantage of making sure everything is rendering just as we see in the GUI mode of that browser. This had a lot to do with moving away from PhantomJS. But it also removed the unkowns, Xvfb and other dependencies that came with it, from our stack. The lesser the unkowns the better I guess.
We have found PhantomJS to be very buggy and hard to debug. We generate millions of PDFs a month. Whenever we upgrade PhantomJS, something breaks with no way to debug easily. We end up reverting to the older version. After trying 3 times, we gave up and had settled for the older version.
In the middle of moving a massive codebase off PhantomJS, it is indeed a nightmare. Old versions leaked memory, debugging/logging is really weird because you get trapped in the context of the browser, and pooling...one of the old phantom pooling packages on GH is part of someone's grad thesis.
Does not play well with React Fiber either, in my experience. Aaaand isn't the brain behind PhantomJS stepping down or something? To any potential users: just go with Chrome
Chrome's headless mode uses significantly less memory than using xvfb. I assume FF would have the same benefit. Plus one less moving part, dependency, etc.