For all of you old fogies working on angular 1 apps and are running browser tests with karma you can run your tests in firefox with headless mode with a customer launcher. Add the snippet below to your karma conf
It's becoming more and more popular not to use a browser for unit tests.
You do "pure" unit tests, which test most of the logic, computations and intention in nodejs, then rely on (usually just a few) E2E tests in selenium or whatever to make sure stuff actually work in the browser.
For example, in a standard React/Redux app with low amounts of legacy code, you can test almost all of the app's constructs in node and be very confident that everything works.
There's still a few places where Karma + Browser shine. Namely when something requires a unit test-style environment but is integration heavy. Think libraries abstracting browser details such as a rich text editor or a library like jQuery.
We were using Xvfb with PhantomJS. Moving to a headless browser gave us the advantage of making sure everything is rendering just as we see in the GUI mode of that browser. This had a lot to do with moving away from PhantomJS. But it also removed the unkowns, Xvfb and other dependencies that came with it, from our stack. The lesser the unkowns the better I guess.
We have found PhantomJS to be very buggy and hard to debug. We generate millions of PDFs a month. Whenever we upgrade PhantomJS, something breaks with no way to debug easily. We end up reverting to the older version. After trying 3 times, we gave up and had settled for the older version.
In the middle of moving a massive codebase off PhantomJS, it is indeed a nightmare. Old versions leaked memory, debugging/logging is really weird because you get trapped in the context of the browser, and pooling...one of the old phantom pooling packages on GH is part of someone's grad thesis.
Does not play well with React Fiber either, in my experience. Aaaand isn't the brain behind PhantomJS stepping down or something? To any potential users: just go with Chrome
Chrome's headless mode uses significantly less memory than using xvfb. I assume FF would have the same benefit. Plus one less moving part, dependency, etc.
Well why do you want to do that, if you are finding the tests are too slow it maybe worth writing some tests with jasmine that can be run in node without a browser.
That's what we currently do, karma and browser testing are useful but sometimes you can get away with not using them.
One thing we are trying is using JSDom to run browser based tests in node
Tools like headless browsers might turn into the "perl" of this generation. :)
Years ago I could always find a way to make Perl do whatever I wanted. It was usually "wrong" to fire up an entire interpreter compared to using faster, less flexible tools but it got the job done. Now I imagine people thinking, "the browser will do this", especially if they can grab screenshots or other local data files.
It would be cool if mozilla and google would standatalize the headless API and command line flags as llvm and gcc has done, so you would perform headless cross browser testing.
The onus of who actually automates the browser is moving from Selenium (the open source project) to the browser vendors. This will help with reliability and is also exactly what you're asking for:
I would be very interested in tracking the details you're hinting at to know when this is done and stable to use reliably. Do you have specific examples/issues related to what your tests today require that is unreliable / not currently possible using WebDriver with Safari and/or headless Chrome?
Any idea when exactly the PDF capture is performed? So many pages now render a minimal initial load, while the actual content is greatly delayed (usually < 100ms, but sometimes 3+ seconds) due to being loaded from one or more AJAX sources.
Capturing the PDF at even the onload event (let alone the useless DOMContentLoaded event) is going to capture an incomplete picture in a huge number of situations. Frankly, a PDF capture at onload event is going to be utterly unusable. Is there a way to set a delay before capturing, ie: 10 seconds?
In an ideal world, a true "finished loading" event would trigger only after all DOMContentLoaded and onload handlers are executed, only once any AJAX requests launched by such handlers are completed (including requests fired off in their handlers, recursively). Of course, websockets and long polling ajax would still interfere with detecting a true "finished" event, but it'd be better than setting an artificially large delay of 10+ seconds.
I don't think this feature is "utterly unusable". You can run your JS, wait for whatever events you want, and then pipe the resulting HTML to a --print-to-pdf command.
The primary reason to print a pdf would be to do so to any random 3rd party page. Sadly, you have no control over how 3rd party sites render/load their content. And --print-to-pdf likely captures the render of the page at some predetermined point, my guess being the "onload" event. Which, again, is not going to result in a fully rendered page for many sites that load in their primary content using AJAX.
Yeah, D3 is what we're working with as well; also need to be able to embed the charts in Powerpoint, but that'll be a different story. All fun and games.
True - but you can get them to output to PDF, then composite/stamp this pdf onto a PDF with headers/footers/whatever on it. This is very easy to do with pdftk.
I built a simple REST API that wraps this call for my other projects and it's been working great the past few months. If this would be useful to you my contact info is in my profile.
Remember that if you pass through any data from an outside source, you're running their code on your browser when you thought you were just printing a document.
The security issues don't apply in all circumstances, of course, but if you feel like taking a shortcut to PDF generation and printing through a headless browser, you need to keep them in mind.
I highly recommend that anyone using a headless browser that may access outside sources employ something like this to run the headless browser in an external environment, in this case AWS Lambda: https://github.com/justengland/phantom-lambda-template
I used this approach for a project at a previous company and not only did it keep the potentially unsafe external code execution isolated from the rest of the stack, but it also proved to be fantastically scalable because of the ability to have AWS Lambda running many headless browsers in parallel compared to trying to scale something like this out on your own hardware.
Did you find any library that implements this? I mean, it should not be hard to implement, just slap together some templating language like thymeleaf or velocity and shell it to the headless browser. But still, it would be nice to have all this already implemented.
Last time I had to generate PDF's from scratch rather than inheriting a mess I built a custom heavily modified version of bootstrap (really ripped it apart) just for print use and fed it to wkhtmltopdf via snappy, it worked great and was very fast to develop since I could output to the browser as HTML to debug.
Simply put, it's designed to make PDF output and provides a lot of quality-of-life tooling around that. Better support for CSS paged media and non-HTML XML input, better handling of page-spanning tables, shims for things missing from paged media (like hyphenation and ligature controls, PDF-specific objects to modify output with Javascript, Javascript embeds into PDF output), better out-of-the-box styles, better documentation and support, and more predictable output and better debugging tools.
Browsers are catching up on paged media options, but they're not all the way there yet. The rest of the stuff Prince does is not (and arguably shouldn't be) in the scope of headless Chrome.
There was a brief moment where headless Chrome supported flexbox in print before Prince did, but that was fixed a month or so ago in Prince dev and should land in production soon if it hasn't already.
PrinceXML is in a league of it's own. Very, very light weight. Very fast and it has outstanding output. The price tag is steep for sure, but if your business relies on PDF output then nothing comes close.
It has unmatched print CSS support, so it'll take anything you throw at it.
Also it blisteringly fast. You'll process 10 documents before headless Chrome has even finished starting up.
There's a free trial which prints a watermark so you can give it a spin. very easy to get up and running, even has a GUI so that you can throw some test docs at it without any effort at all.
I've used wkhtmltopdf for years and it's really great. However, having a browser (headless Chrome and now Firefox) render the page closes the gap between what we build in HTML and what we want to output in a PDF, without having to deal with nuances of the wkhtmltopdf rendering engine.
We're developing a library[1] for this use-case. The goal is to have the same simple API for multiple browsers. Right now it supports both Headless Chrome and PhantomJS.
We think we'll begin work on Firefox headless soon. PRs welcome :)
I'm currently evaluating testcafe which is pretty nice, and headless chrome was easy to setup. I'm interested to see what people recommend for browser testing, I'm also using preact.
Writing end-to-end tests is always a sucky and flaky experience, and it always ends up slowing down CI. With that said, I've used testcafe without any big complaints.
My general rule of thumb is to try having as few end-to-end tests as possible. It's fine for an e2e test to cover multiple aspects of your application. Trying to write e2e tests as if they were unit tests just leads to sadness and infinite CI builds.
I don't think there's any value to running all your tests on the browser. It's much faster and simpler to run the bulk of your tests on node, with a fake DOM environment.
If you're interacting directly with the DOM, you might want to consider breaking that off as a separate lib. From the lib's repo you can have all of its tests run on a real browser.
Totally agree. I am a fan of Test Pyramid model, where at the top you have a very small number of long running, complex tests (usually e2e), and at the very bottom you have a lot of small, isolated and very fast unit tests.
Testing everything in a browser/e2e provides a rather poor cost/benefit ratio, especially as the project grows and more features and tests are added, so it should be reserved for cases where it is really important.
I like having e2e tests that cover the very core functionality and involve multiple parts of the stack. For example, if you have your rather typical app with a login modal, it's probably good to have a test that clicks on the button, makes sure that the modal is visible, and the user is redirected to the expected page after logging in. But testing if error message is displayed if email address is invalid? That's too much.
I currently have setup two CI jobs running Webdriver.IO with different configurations (different in what browser is used). Locally I just run it on all installed browsers using selenium-assistant - not headless, because I want to see what's going on when running locally.
(And yeah, cross-platform Edge would be nice for being able to test in CI without needing a third party.)
The article mentions screenshots. I am interested in running the browser as a rendering tool, but ouputing not just a bitmap, but something closer to PDF. The goal here would be to crawl a bunch of pages and spin up the CPU to run all the needed JavaScript to render things, but to obtain an identical version of the page that runs about as fast viewing a screenshot, but also includes text. An offline "web OCR", if you will.
NeWS, Display PostScript, Quartz all come to mind. I know you can save webpages as PDF documents, but I am thinking of something that is like PDF but closer to HTML-just more closely coupled to what is being displayed, and not whatever arbitrary style or JavaScript abuse the developer decided on.
Even just outputing some neat, canonical HTML based on the state of the page once everything is loaded would be helpful, so that the bitmap could later be combined in some kind of new document format.
I'm pretty sure you could write some WebDriver + JavaScript that captures the DOM/text in the page. You might even be able to capture layout information for later reconstruction.
That is actually exactly what I want (why didn't I think of it...)! Thank you.
In fact, wkhtmltoimage supports svg and does a great job of rendering, say, GitHub or techmeme. It falls over on formatting mathoverflow.net, but I think the same technique could be changed a bit to more closely resemble what actually gets layed out on the page in a running browser instance.
Can someone do an ELI5 on why headless browsers are useful?
I've use them before to make screenshots of webpages, and I have noticed that many NPM packages come (or came) with PhantomJS as a dependency, but I have no idea why one would need that.
Automated integration/regression tests is a pretty big use case.
You spin up a copy of your server, point the headless browser at it, run some JavaScript to simulate some user interactions, and verify the page contains the right strings / screenshot doesn't deviate too much from a golden image.
I guess the ELI5 way of saying that would be: "Making sure your website still works."
People also headless browsers for content generation inside larger systems. (I saw one team make videos by creating CSS animations, and then capturing screenshots from the headless browser at 30 fps.)
Putting the website through user actions in a firefox browser and recording/capturing/documenting the results headlessly. Headless version allows you to go through hundreds of results in the background.
Maybe. I got stuck trying to imagine whether a five-year-old would understand what "headless" means in a software context (as opposed to, "headless dragon" or something).
Scraping for data is another big use case. Like a price comparison web site. Or the company LinkedIn is unhappy about because they are scraping profile data.
Crawling the Web. Although you might think that you can crawl a web site by poking through the HTML looking for <a> tags and opening the URLs they contain; that stopped working well a long time ago. Modern sites are essentially complex GUI applications that run inside...a browser. You need either a browser, or some browser-equivalent thing therefore in order to run them and properly crawl them.
Isn’t this why these ‘headless’ solution have (or are actually built on) JavaScrict engines like V8 with the ability to also simulate any user interaction?
They're a hassle-free way of getting the data. No need to worry about CORS, sessions, cookies, CSRF and other modern web stuff. Just simulate a human and you’re in.
Yeah, I used to work for a company doing similar things. Was more expensive that way, but there were tons of sites you could only get the data being in a proper browser.
In the interim I've been using standard C# web requests and using the cookies from the ChromeDriver. Doesn't work for downloads where it is not a direct link unfortunately.
Headless Chrome/Firefox render the website into a framebuffer in memory (instead of on the screen), but websites still look the same. You can extract an image file from this framebuffer and it will look as if the browser had been running in normal GUI mode.
Lynx on the other hand renders onto a text terminal, which is a completely different output device compared to a pixel-based framebuffer.
Does anyone have any metrics on headless vs headed browser testing? Can you say headless is 25% faster and 20% more reliable? What is the sales pitch for headless browsers for UI automation?
I don't have numbers handy, but I didn't notice any significant performance difference between headless and headed chrome. I think the main benefit of headless is ease of test server setup. We use a CI service, so that's not a factor.
We are sticking with phantomjs for now because it is significantly faster than chrome for our tests.
What about the "app" mode?
Chrome have introduced the --app command line switch long time ago and similar functionality would be very useful also in Firefox. I suppose that implementation would be also very easy, i don't understand why there is still no such functionality in Firefox.
Firefox does have their own, documented and seemingly stable remote debugging protocol. I haven't used it recently, but there is the Valence project (https://github.com/mozilla/valence) which provides adapters for their debugging tools for other protocols, namely Chrome's remote debugger. In the past, I have used this to debug Chrome with firefox's devtools. Maybe it works for your needs?
Does anyone know if there is a way to control the window size? If there is does it also modify screen.availHeight properties and other give aways that you have modified the window size.
2008 for Sauce Labs, to be precise. Before that, BrowserCam (founded in 2002, acquired by Gomez in 2007) was part of my inspiration for Sauce Labs. BrowserCam was one of the first screenshots-as-a-service sites. The only thing it was missing was Selenium support for end-to-end test automation. :-)
There totally is a market for it. Sauce Labs makes $25MM/year doing it. One way to secure the service is by using pristine virtual machines for each session and discarding them afterwards.
Perhaps I'm spoiled, but I'm wondering why this wasn't a thing many years ago. Am I wrong in saying this was brought about in reaction to Chrome adding the same feature, or has this been there for a while? Clearly people wanted this for ages, hence the work on things like PhantomJS. But it seems like it was only added once they realized they needed to play catch-up with Chrome.
Hi, I'm the Product Manager for Headless Firefox at Mozilla.
No, this wasn't a reaction to Chrome. We decided to implement headless mode last fall after research last summer indicated that it would increase website testing in Firefox and thus improve web compatibility.
Of course, I was aware at the time of Chrome's own efforts to implement a headless mode, and I've continued to pay close attention to their work.
I've also made similar decisions to Chrome at times (f.e. to use the same --headless command-line argument to enable the mode), while making different decisions at other times (like deciding to prioritize support for the WebDriver API, whereas Chrome has focused on support for the Chrome DevTools Protocol).
But my focus has been primarily the direct benefits to web developers of being able to run tests against headless Firefox; and the indirect benefits to Firefox web compatibility of more web developers testing on Firefox.
I remember using this with xvfb to achieve headless Firefox. The rendered text could then be accessed via localhost with a text-only browser or tcp client e.g. netcat.
Possibly contributing factor: the old selenium plugin for Firefox is broken by the webextensions move. While working for that use case, they cleaned up other issues for it?
This seems way too fast to be a response to chrome
This isn't another use case for it; you can still use Selenium with Firefox 57+. It's only the extension that allows you to generate Selenium scripts by performing the tasks you want it to execute manually that will no longer work, and no effort is invested in that because there are good enough alternatives.
Both. Ideally, you shouldn't run website tests on a single browser. We shouldn't want to return to the days where websites only worked well on a single browser.
Exactly. I already think it's a shame that I can only run Firefox and Chrome in Docker (i.e. CI), but I hope that their differences will compensate enough to cover most potential problems in Edge and Safari.
You don't need to set all your environment variables in your bashrc, if that's not obvious. They're not global state. You can change them on a per-terminal basis, and that only effects the environment of that one terminal.
Firefox does some weird stuff with multiprocessing, where if you run the firefox command twice you only get one process. I don't know how that would effect things, but I imagine just setting the env variable and running as a different firefox profile would do it.
There's always firejail/firewarden, if you need a quick and easy sandbox for a program you already have installed, and that will solve that problem, if it is a problem.
As a UI designer, I dream of strikeheadlessstrike chromeless browsers, where the UI appears when you need it, intuitively, even when NOT in full screen. The top bar simply takes too much unnecessary space. If there's any dev that wants to collaborate on this, let me know :)
A small nomenclature clarification seems to be in order:
"Headless" means without output. It is often used to describe servers without any attached monitors (the monitor would be the head that it is missing), but can similarly be used to describe browsers that do not even create a window on your desktop. It simply runs as a service that can be scripted.
What you are looking for, I would rather call "chromeless". The browser is there, but it has no visible UI-components outside of the webpage.
> What you are looking for, I would rather call "chromeless". The browser is there, but it has no visible UI-components outside of the webpage.
Why "chromless" and not just "Full screen"? The latter is what that mode has been known by for decades? Or is it a subtle distinction that "Full screen" mode can show the location bar by moving the mouse to that edge of the screen (at least in Firefox 55), while "chromless" never lets you see the location bar?
Those are two orthogonal concerns. A (theoretical) browser in full screen mode might or might not display UI elements/browser chrome. A windowed browser might or might not display browser chrome.
A chromeless windowed browser would look something like QuickTime in macOS, if that helps you; the window consists entirely of the playing video until you mouse over it.
I really hate this trend in UI design. I have to scroll back half way up the page in iOS Safari to get the menu bar to display again, for example. I've watched a lot of non-technical users struggle to figure out what magic they have to perform to get an app's controls to even appear.
Your job is to make things usable & functional, not pretty.
Anyway, IE11 on Windows 8 (tablet mode) had that, and it was amazing (assuming you used the touch screen, that is - the UX wasn't so fantastic with a mouse).
It's the only thing from Windows 8 I miss in Windows 10.
Or popups? Just imagine how realistic I could make a popup box seem know people will just think that it is from the operating system which I can also detect in another headless window along with another transparent window for bitcoin mining...I think you get the idea. UX yeah this is neat for appliances it will make for more seamless experiences, lets just hope you all are sanitizing inputs and building in secure methods of auth to make the experience worth the risk.
No it isn't, the URL is unreliable source of truth with Unicode spoofing attacks and things like that. Safari only shows the domain at this point; the URL is an unimportant implementation detail.
You mean, before the 90% of the global population that doesn’t speak english as first language goes online, right?
Because the #1 use case for this isn’t emoji, but sites such as bücher.de (I originally had a link here to the site, but HN punycodes that: http://xn--bcher-kva.de/ ) (which nowadays had to redirect, because the URL displayed in punycode made people believe it was a phishing attempt)
It would be cool if browsers could highlight URLs that contain the kind of characters which are commonly used to spoof, though I suppose it would lead people to a false sense of security.
Your claims are false for the majority of the world's population (which lives in non-English speaking countries and may well want to visit sites whose names make sense in their own language).
So sites such as bücher.de (I originally had a link here to the site, but HN punycodes that: http://xn--bcher-kva.de/ ) shouldn’t exist?
That’s a very america-centric world, it’s like enforcing only US-ASCII on all websites. Most of the world doesn’t speak English, and browsers showing domains punycoded leads to mistrust, especially if it’s a legitimate retailer (the one mentioned above actually added a redirect to a romanized version of the URL due to that)
Firefox [0] and Chrome [1] have policies for when to show Unicode and when to show raw punycode. You can also set flags to force them to show punycode all the time.
Cyrillic Er looks exactly like Latin P, so writing apple with Cyrillic "Er"s would likely be spoofing. But if you highlight all instances of Cyrillic Er, you also have to highlight every P, since the URL of a Russian bank spelt with Cyrillic Er could be spoofed with Latin P.
In the end, you would just highlight nearly everything. A more useful approach would be to highlight when you switch script inside a domain name. That seems to be what firefox does for non-whitelisted domains (with some more rules to allow eg www.stマイクロ.jp (ST Microelectronics in japanese))
I used to achieve this by using a tiling window manager (i3).
In i3, if you run firefox normally it opens up in a pane, if you choose full screen (f11) inside firefox, it fills up the entire pane. In the other panes you can still keep your other windows, such as emacs, file browser, terminal etc.
F11 is full screen, not headless. Headless means you can't see the browser or the website content, it's built for scripting purposes (automated web testing and web scraping). The only time you can see the content is if your script requests a screen capture.
Sounds like a great idea. My skills aren't quite up to UI design collaboration standards but I guess I am curious as to how you would build and implement such a browser?
``` browsers: ['FirefoxHeadless'], customLaunchers: { FirefoxHeadless: { base: 'Firefox', flags: [ '-headless', ], }, }, ```
Note you need to be running the beta version of firefox, I needed to download it from here https://www.mozilla.org/en-US/firefox/channel/desktop/