As performance is important for both users and SEO reasons, we’re taking some steps for Zoover.nl to improve those metrics. This is part two of a series where we discuss some of the improvements we are making.

In an effort to increase awareness of — and encourage collective responsibility for — performance, we added breaking tests for performance regressions on all pull requests. Three months later, we look back at not just the changes in performance, but also how the tests impacts our decision making.

Testing for performance

For every pull request, we deploy a test environment that contains the changes. As part of our CI pipeline to build that environment, we run a test with Lighthouse on several pages, and compare those results to the staging environment. We have added a bunch of hand-rolled gatherers so we can run audits on custom metrics as well. If those metrics have decreased compared to the reference environment, or stay below the limit, the tests pass. In all other cases, the build is flagged as failed, and the change author is notified.

Measuring the right things

When we first explored this option a while ago, one of the issues we ran into was that Lighthouse’s default metrics (First Paint, Time To Interactive) were not particularly useful for comparing changes. Not only are they dependent on response times (which is very unpredictable in our slower test environments), we ship not just our own code, but also a tag manager that executes third-party scripts on our page. The variation in these factors make it impossible to reliably assess the impact of any proposed change. We decided to zoom in on our own code, and compare the following metrics:

• Critical JavaScript Size: This includes the size of all our bundled JavaScript. Keeping this down is important, because it impacts how long it takes for our site to become “interactive” — meaning that search works, drop-down menus work, display advertisements are loaded, etc. Before that is possible, our JavaScript needs to be downloaded, parsed, compiled, and finally executed. The more JavaScript you ship, the longer this will take, essentially blocking the user from interacting with your website.

• Critical CSS Size: We also measure the total size of CSS shipped to the browser. Before CSS is downloaded and applied to the page, the browser will show a blank screen. The longer this takes, the more likely it is that the user just gives up, and leave.
• Inline Styles: When this project was started a few years back, inline styles was the go-to solution for styling in React land. That however, has changed, as it is pretty bad for performance. Because it’s not supported by AMP either, and we want to keep that option open, we are moving away from inline styles. This metric measures the number of elements on a page that have inline styles.

Additionally, for posterity and debugging purposes, we also record the amount of elements on a page, the response size of the HTML, and the size of the images on the page.

The developer experience

While our initial implementation was pretty rough around the edges, we made various improvements over time that helps us in running accurate and consistent tests, and detailed logs so anyone can dive into a failed performance test and figure out why it’s failing:

• We output the metrics to the console, with lighthearted emojis for the feelgood factor. We’re all humans, so seeing a nice little green heart when you actually make things better is a pretty nice bonus.

• We report uncompressed sizes. This is more of a practical choice; our staging environment (and production) is behind a CDN which applies its own compression, and the compression rate is different than the compression on our test environments. This makes a comparison unreliable. We wrote our own Lighthouse gatherer to make this work.
• We disable external scripts. There’s too much variation in how external scripts impact page speed, so we disable it in order to focus on the things that we can control. This comes with a disclaimer though: in our case, external scripts have an even bigger impact on page speed (50–70%), so it’s definitely important to keep that in check as well, but because managing this requires a more holistic approach, with many stakeholders, we had to remove it from the equation for now.
• There’s a small threshold of 0.2%. If the change is within that threshold, the tests will still pass. If we don’t have a little wiggle room, even things like very small bug fixes will fail the performance tests, and it will just block us from getting critical things out the door.

Additionally, we store some artifacts that we can download from our CI tool:

• A Webpack Bundle Analyzer report: For a great visual overview of what actually ends in your bundle, you can create a report with webpack-bundle-analyzer. It shows a treemap of included modules, sized by its share in the minified asset. This helps us in quickly seeing which modules got bigger and smaller, and why they did. It’s especially helpful in analyzing duplicate modules (like React DOM ending up twice in the bundle, or various Lodash instances).

• Diffs for the HTML, CSS and JavaScript. Our custom resource gatherer not only stores the uncompressed size of the response, but also the response’s body. We store this in the Lighthouse report, and then use it to generate diffs when comparing to the reference branch. Its output is similar to git diff, and it’s especially helpful for CSS and HTML to figure out what exactly has changed.

• The entire Lighthouse report as a JSON file. In this 1mb+ file, Lighthouse stores a bunch of information about the run. All audits and metrics, but also things like screenshots and runtime errors. As a nice side-effect, we were able to use these files to retroactively upload the performance test results to Elasticsearch so we can analyze the changes over time.
• Network errors and console messages. In some cases, we saw some weird, inconsistent results. To gather more debugging data, we’ve added custom gatherers for console messages and network errors so we can figure out retrospectively what happened during a CI run. This for instance was particularly useful when we need to figure out why a lazily loaded image was sometimes unexpectedly included in the reported size.

Assessing the impact

To see what metrics we moved, we downloaded all the results from our CI tool, uploaded the scores to Elasticsearch, and created a dashboard that shows how our metrics progressed. Results are pretty good: JavaScript is down by almost 30% on average. A similar change can be seen in bootup time and main thread work. CSS is up by 5%, but still under 50kb. This is pretty great considering the fact that elements with inline styles dropped from almost 300 to 75.

On the flip side, productivity took a bit of a hit, especially in the beginning. It can sometimes be really hard to figure out how to ship a new feature without running into performance regressions. For some this is a challenge, but for all of us it can be pretty frustrating. But we’d like to think things are definitely getting better: we now see it as a warning, an incentive for us to to ship features economically, and fix any lingering performance issues; it’s there to start a conversation, not as a final decision on whether a change is accepted or not. It has increased awareness of performance among not just us as engineers, but stakeholders too, and it has exposed some fundamental flaws in how we built things that can now be addressed. Hopefully the downward trend continues as we keep getting better at fixing our performance issues.

Holding the performance line by regression testing bundle size was originally published in Zoover Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

1. This was a great conference

Expectations were high because of the stellar speaker list — and expectations were met. A wide range of topics were covered, from internet protocols, to low level JavaScript hacks, to performance monitoring. Single track works well — it’s pretty chill to not have to worry about talks that happen at the same time, or rushing from one area to another just to get in. The hosting by Tim Kadlec and Steve Souders was thoroughly enjoyable. Pretty much all the talks were not just interesting, but also fun: Zach Leatherman’s talk about Webfont was a particular highlight. And the food was pretty great, too 😋. The only thing that was a bit off was that some of the talks had quite some overlaps, but all speakers gave their own spin to it. (and thanks to the organization for making some space for us at the last minute 😊)

2. JavaScript is the main culprit for slow web pages

JavaScript makes website slow. No news here, but this phrase was so often repeated during the two days at the Compagnietheater that it will be stuck there until next year’s event. Here’s the problem: we’re shipping more and more JavaScript to the browser, and low end devices (like cheap laptops, or anything that’s not an iPhone) cannot keep up compiling and parsing said JavaScript. For the 95th percentile of researched web sites, over 60% of time is spent parsing, compiling and executing JavaScript. For mobile websites, that means 10 seconds of JavaScript execution on start up. To keep users happy, especially those on low end devices and/or in emerging markets, we have to do better. How? Well, ship less JavaScript. What will also help: using preload headers, utilizing service workers and curtailing third party scripts.

3. Improving performance requires buy-in from all sides

After Tim Kadlec’s opening talk, Harry Roberts came out swinging against third party scripts and tag managers. Armed with a bunch of data and anecdotal experience, Harry pointed out that third party scripts (and their third party scripts) often make a website unbearably slow. Then, Harry switched to people: the only way to fix this issue is to talk to stakeholders that use the third party scripts. Are they really necessary? Can we solve this issue in a more user-friendly manner? Performance needs a holistic approach: engineers need to care about performance, and evangelize its importance — but stakeholders and management need to see its value too. Michelle Vu is an engineer on the performance team at Pinterest; she expanded on this by pointing that engineers sometimes need to be convinced, too: the performance team tries to give product engineers the data and the tools needed to detect and fix performance regressions.

4. Be data-informed.

I’ve never seen so many charts! Line charts, bar charts, pie charts, flame charts. Some curvy, some straight. Recurring advice: make sure you are measuring performance. Developers work on beefy devices, on lightning fast environments, whilst our users are stuck with the production website, riddled with marketing tags, on a low-end device. Ideally, we need two layers of data. Firstly, we need to capture data from real users on our production environments (Real User Monitoring). SpeedCurve is an example of RUM as a service. Secondly, to catch regressions before they get into production, we can use synthetic tests, with Lighthouse or WebPageTest. Michelle Vu and Katie Sylor-Miller both mentioned the difficulty and importance of A/B testing your performance improvements. And lastly, if you’re going to monitor metrics, make sure you pick a metric that makes sense for you. For some websites (like blogs), it will be First Meaningful Paint (or any of its siblings). For other, more interactive websites, something like First CPU Idle will make more sense.

5. Where was the React/Vue/Angular perspective?

Lots of performance evangelists point towards usage of modern frameworks (like React or Angular, and Vue to a lesser extent) as one of the reasons for performance degradation. The idea is that framework culture (anything that can be built with React, has to be built with React) leads to impossibly bloated JavaScript bundles. There’s no denying this as far as I’m concerned: for instance, our platform is built with React, and we’re pulling in around 1mb of uncompressed JavaScript per page, which seems way too much for a content-driven website.

Which makes it that much weirder that this part of the community was seemingly missing from performance.now(). It was only fleetingly mentioned in one or two talks as far as I can remember. It does feel like two separate worlds at this point — and I think everyone would benefit if those worlds can be brought closer together. Hopefully we’ll see that next year when we gear up for the second edition of a great conference.

And lastly — some of my favorite talks from day one.

Notes from performance.now() was originally published in Zoover Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the (supporting) metrics that we are trying to improve for this trimester is the amount of CSS we ship to the browser. CSS is absolutely critical for the (perceived) performance of your website: any render is delayed until your stylesheets have been fully loaded. We’ve given ourselves a budget of 50 kilobytes of uncompressed CSS: not entirely coincidentally the same limit that AMP imposes on stylesheets.

Now, given your perspective, that might be a highly ambitious goal, or one that is easily achieved. For us, it is the former. Here’s why:

• We use CSS Modules, which doesn’t encourage re-use;
• We do not have a lot of design system discipline, so we have a bunch of small variations of seemingly similar components & styles;
• Our pages have lots and lots of content;
• We continuously have A/B tests running which adds more bloat to the page;
• We have to support a myriad of devices and browsers.

To keep our CSS small, we already have a few optimization steps in place:

• Code-split CSS with mini-extract-css-plugin
• Optimize & minify CSS with optimize-css-assets-webpack-plugin
• Use browserslist with autoprefixer to prevent unnecessary vendor prefixes

Unfortunately, that’s not enough to keep it under 50kb on some of our pages. There are some things that we can do in the long term: double down on a design system and encourage re-use, and combat feature bloat. These options require however a significant investment, and buy-in from stakeholders. As a short-term solution, we decided to explore the possibility of generating different variations of our stylesheets, with only styles that are applicable for the current device and viewport. That means removing any styles inside of irrelevant media queries.

Exploring the options

First things first: a link element has a media attribute, right? You can use this attribute to tell the browser to only apply a given stylesheet when the viewport matches the specified media query. Great, we can probably use that to download only the stylesheets we ne —

Nope. Wrong. The browser will still download stylesheets with a media query that doesn’t match, even though it doesn’t plan to use it. Here’s a test from Scott Jehl that demonstrates this behaviour: http://scottjehl.github.io/CSS-Download-Tests/

So, what else do we have left? Well, data. At Zoover we use screen classes: extra small, small, medium, and large. Roughly, the first two apply to phones, the third one to tablet, and the last one to desktop. That leaves the possibility open of using the User-Agent header to detect what kind of device is requesting the page, and then loading a stylesheet that is stripped of media queries that apply to screen classes not relevant to that device. However, because you have devices in all shapes & sizes, there’s no guarantee that a phone will never have a “medium” viewport, or that a tablet will never have a “small” viewport. Which brings us to step 1:

1. Get the data

As a first step, we needed to get an idea of how closely device types match the expected screen class. Since we are already tracking page loads in the browser, it was easy for us to add some metadata to our metrics about the device type, the viewport dimensions, and the screen class. We can then see the distribution of screen classes for a given device type. Here’s a Kibana visualization of screen classes on desktop:

Now, as you can tell, pretty much all of it is in the lg screen class. About 2% is md and xs and sm are below 1%. That means we don’t need media queries for the latter screen classes in our CSS file. Similarly, for tablet we can drop xs and for mobile we can drop both lg and (surprisingly) md.

2. Pre-process Webpack’s CSS output

Now, we need some way to hook into Webpack’s compilation process. The seemingly obvious way to do this is to write a Webpack plugin. A Webpack plugin allows you to tap into certain phases of the process. The phase that we need is additionalAssets. There might be a more appropriate phase here, but this… works. The catch here is that we need to run after mini-css-extract-plugin runs, but before optimize-css-assets-webpack plugin, as we want to hand our own generated CSS off to the minification process that we already have. Here’s how the plugin looks, roughly:

Pretty easy! Now we can send all CSS to our little tool that will generate even more efficient CSS.

3. Remove unused media queries

Now that we have some CSS, we need to process it a bit. We need a function that accepts named ranges, and returns CSS for every range that is stripped of media queries that are not applicable. The perfect tool for such a processor would be PostCSS, which is basically Babel for CSS. Now, of course someone has already written something which does exactly what we need, but because it’s really easy to do things with PostCSS, and for the sake of my own ego, we’re just going to write our own:

There we are. We used css-mediaquery for parsing and evaluating media queries in Node (in a browser, you can use window.matchMedia). We’ve also made sure the order of the input CSS file is respected. We can then return this output back to the Webpack plugin that we wrote before, and all our device-type specific CSS will be written to disk.

4. Using device type detection to serve the CSS files

Now, as a last step, we need to serve the appropriate CSS file to the browser. At this point we have already created device-type-specific CSS files. Now, when a request comes in, we can use the User-Agent header to detect what kind of device we’re serving. Then, we use this information to serve the trimmed down CSS file for the device type that we created in the previous step:

There’s one odd thing here: we set a data-href attribute on the link element. This is because mini-extract-css-plugin embeds a little runtime that takes care of loading stylesheets as the browser navigates your application. That runtime needs to check whether a stylesheet was already loaded. The way that it does so, is by checking the href attribute of already present stylesheets. Luckily for us, it falls back to data-href, which means that we can trick Webpack into thinking the stylesheet it is looking for is already loaded. If we don’t do this, Webpack’s runtime will immediately inject the original stylesheet into the document, negating any improvements that we were hoping to get out of this change. Here’s how that looks for desktop:

5. Evaluation

Our CSS files are approximately 10% smaller. Here’s a result of one of our performance tests (where we compare feature branches to our master branch):

In the grand scheme of things, 10% is pretty small, and in a lot of cases, not worth it. It will not be the biggest win we can make either. However, it gives us a little more room to breathe as we work on more structural solutions for our resource bloat, it was fun to write and it cost me more time to write this article than to implement this optimization. That might be my writer’s block though 😀.

At the end of this week, we’ll be attending the performance.now() conference; pretty sure we will have enough material then for the last part of the year, so stay tuned!

Optimizing CSS by removing unused media queries was originally published in Zoover Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

After I joined Zoover in March, one of the first things we did was adding hot reloading to our React app, by using Webpack’s Hot Module Reloading and react-hot-loader. Before that, we used gulp-livereload and its full-page refresh to get the updated UI. Initially it was working quite well, but it got more unreliable along the way, and after we added code splitting and react-loadable to our application, it got to the point where my colleagues were just doing a full-page refresh manually.

This is a pretty bad place to be in: not having confidence in the correctness of the hot update basically kills all advantages it might have: first, you’ll wait until you see the screen update; then, you’ll refresh anyway because you need to be sure that what you saw is correct. Frankly, we were better off with the full-page refresh that at least was automated. This had some drawbacks though: our server-side render setup means two Webpack builds, >1s rebuild times, and a server that is down when the full-page refresh occurs, which often meant it would take 5–10s to see changes on your screen.

The Why

The first guess was that code splitting, in combination with react-loadable broke it.

In case you don’t know what those two things are: code splitting is the ability to define split points in your code. A split point is a point where Webpack splits up your bundle into separate files, that can be loaded on-demand. This could help you keep the weight of your critical assets down, and it’s an absolute necessity for bigger applications, especially on mobile (where bundle size hurts the most). react-loadable is a pretty nifty React component that helps you import and render lazily-loaded components, and make it all work on the server as well (which is non-trivial).

That first guess initially seemed wrong: the thing that broke hot reloading in some (or many) cases was one of the React optimization plugins we used (turning those plugins on in development mode was probably not the best choice, but alas). They modified components in a way that made it difficult for react-hot-loader to work. After turning off those plugins, hot reloading certainly worked better, but it still was broken for lazily-loaded components. Initially I thought it had something to do with the module not being required because of the react-loadable works, but it turns out it (also) breaks at Webpack’s level.

For those unfamiliar with hot reloading in a Webpack and React context: you have to “catch” hot updates for a specific module in the same place where you render your application. If you get a hot update, you’ll need to re-render the application. Meanwhile, react-hot-loader adds a small piece of code to every component module that registers that component with react-hot-loader. Then, when you re-render your application, react-hot-loader proxies React.createElement calls, and replaces React components with those from its own registry, which will have the latest versions from the hot update.

Now, if you’d expect updates from split modules to bubble up to the entry point, you would have the same expectations in life as I would have. And we would both be wrong. Turns out those updates only bubble up to the top module in the chunk of the changed module, so we’ll never know when to re-render.

The Fix

There are a couple of options to make it work:

• Explicitly accept updates for all chunks in your entry point, and re-render for those updates.
• Use the presence of module.hot or process.env !== 'production' to add explicit, static require statements for lazily-loaded modules.

However, both options essentially require double administration. I’m lazy, so I don’t want to do things more than once (I need to save time to compensate for all the Blue Screen of Deaths my XPS is giving me). What I want to do is turn off Webpack’s code-splitting in development mode. Sounds pretty easy, right? Well, turns out that’s more difficult than I thought. webpackMode: eager could solve our problems, but we can only set it in a comment, on a per-import basis, not for everything at once. AddingLimitChunkCountPlugin and setting maxChunks: 1 disables the output of multiple files, but doesn’t seem to solve anything. After some hair pulling I figured there might be a compromise: maybe we can add the explicit require statements, automatically?

In come Babel plugins: the perfect place to parse source code and automatically add some statements, preventing the double administration we’re trying to avoid. It’s quite hard to find documentation on how to write a Babel plugin, other than looking at babel-handbook and some example plugins, but it’s really easy to get started with once you start up AST explorer.

Here’s the gist of the plugin I wrote: it simply adds static imports for every dynamic import it encounters, and replaces dynamic import calls with a Promise that immediately resolves with the default export of the imported module. When Webpack finally gets to this code, it doesn’t see dynamic import statements anymore, it doesn’t create any additional chunks, and hot-reloading works again.

Try it out for yourself:

• A repository that reproduces the issue
• The plugin in ASTExplorer

To use it, simply add the plugin to your .babelrc file:

Also, just to be sure: don’t use it in production mode, unless you want to disable code splitting over there as well.

About Zoover

Zoover is a travel website, located in Amsterdam. Our goal is to help our users find the best vacation possible. Our website is built with React, on a Node backend, and we’re always looking for people who are good at what they do.

Hot reloading in React apps with code splitting was originally published in Zoover Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.