Replace sprockets/browserify with Webpack #2617

Gargron · 2017-04-29T11:20:26Z

Fix #1712, #1742

What this toolchain upgrade provides:

Live reloads during development
Friendlier asset compilation output (e.g. progress, final bundle size)
Code splitting, async chunk loading, smarter minifying
Automated management of JS translations

Untested potential benefits:

Smaller bundle size?

TODO:

No more best_in_place for admin site settings - needs replacement
No more local_time JS - needs integration or replacement
Make custom.scss work again

Development implications:

Run ./bin/webpack-dev-server at the same time as rails s
Run yarn run manage:translations to get a list of all missing/redundant translations in the web UI

ykzts · 2017-04-29T11:46:10Z

@Gargron Great change!

But, application.js is depends on jQuery. see #2465

…ckground cache updates

Gargron · 2017-04-29T12:27:56Z

@ykzts Unfortunately, jQuery is still a dependency because of jquery-ujs which makes some non-GET links work. There is rails-ujs which is jQuery-free but it didn't work with webpack. jQuery is also used in public.js. On the upside, webpack splits the code into vendor.js and application.js/public.js bundles, so dependencies are not loaded twice.

ykzts · 2017-04-29T13:03:58Z

@Gargron Uh... As far as I see, jquery-ujs is not used in the part where React is used. Even if you delete it, application.js will run with the exception of the notification.

Gargron · 2017-04-29T13:05:11Z

The web UI has one logout button which needs jquery-ujs

ykzts · 2017-04-29T13:08:00Z

Oh... I see. Certainly. I'm sorry.

Gargron · 2017-04-29T22:40:24Z

I don't understand why view specs are failing with this cryptic error. I changed nothing about them...

nolanlawson · 2017-04-30T18:10:25Z

config/webpack/production.js

+const webpack = require('webpack')
+const merge = require('webpack-merge')
+const CompressionPlugin = require('compression-webpack-plugin')
+const BabiliPlugin = require('babili-webpack-plugin')


I'm not sure it's worthwhile to use both Babili and Uglify; AFAIK Uglify still produces smaller code than Babili, although the downside is that it only works on ES5, so e.g. if we wanted a pure ES6 build then that's the point where Babili would be very useful.

This article claimed that babili was preferable to uglify because it could understand the new syntaxes better.

Yeah so that doesn't really apply to tree-shaking; tree-shaking is about bundling ES modules, which is handled by Webpack 2 before the rest of the ES6-to-ES5 transpilation. You can test it, but I'm fairly certain Uglify alone (with mangle+compress) will still beat Babili, assuming you're outputting ES5 code. (In fact it looks like Babili's own benchmarks still show Uglify winning, although I know they're making progress.)

nolanlawson · 2017-04-30T18:11:22Z

config/webpack/production.js

+        cascade: true,
+        keep_fargs: false,
+        warnings: true
+      },


Just setting compress to true should be fine, unless there' is something in particular you wanted to tweak here?

Was implementing suggestions from this comment: #1712 (comment)

I dunno, most every JS project I know of just uses uglify with mangle and compress set to true. You can tweak the compress settings in rare cases, but in my mind "mangle/compress" (aka uglifyjs -mc) is basically Uglify's "default" setting. :)

nolanlawson · 2017-04-30T18:12:05Z

config/webpack/production.js

+        warnings: true
+      },
+
+      mangle: false,


I don't often see mangle set to false, would this break something to turn on?

Afaik it would make trying to debug production errors way harder..?

A good solution for that is sourcemaps. :)

if mastodon published sourcmaps in prod I would be all about that. I personally like having mangle off otherwise though.

nolanlawson · 2017-04-30T18:13:09Z

config/webpack/production.js

+      algorithm: 'gzip',
+      test: /\.(js|css|svg|eot|ttf|woff|woff2)$/
+    })
+  ]


It'd be a good idea to also use the DefinePlugin to ensure that NODE_ENV is set to production which cuts out a lot of development-only code in React as well as other libraries (React docs on the subject). I also opened up a PR for doing the equivalent thing in Browserify FWIW: #2635

rails/webpacker sets NODE_ENV=production in bin/webpack

Huh that's odd, React's docs still say to add DefinePlugin, but maybe it's no longer necessary for Webpack 2? Just checked the output bundles after asset:precompile and I'm not seeing any NODE_ENV references, so LGTM 👍

nolanlawson · 2017-04-30T18:47:05Z

FWIW, just tested the bundle size, and before this PR application.js is 2.97 MB, afterwards application.js is 563 kB and vendor.js is 1.57 MB, so combined they're 2.14 MB which is a huge reduction. Very nice! 🎉

Gargron · 2017-04-30T20:13:05Z

Travis keeps breaking because node-sass needs some binaries that apparently don't get cached by Travis?

Gargron · 2017-04-30T20:29:58Z

Here are upgrade notes:

Ensure you have Node 6.x and not less
Install bundle/yarn dependencies
If it complains about node-sass, npm rebuild node-sass --force
Make sure CDN_HOST contains protocol part, if you are using it
Precompile, restart

kyzh · 2017-04-30T20:44:25Z

I'm not sure if you are happy for people to provide feedback here.
Feel free to tell me off if you don't think it is appropriate.

This branch is live on glitch.social (thanks Bea).
I've been trying for a while and there is noticeable improvement.
It definitely helps with recent issues I had on mobile #2648

(Edit: typo)

…ected

mjankowski · 2017-05-01T18:09:35Z

Gemfile

@@ -58,18 +55,18 @@ gem 'sprockets-rails', require: 'sprockets/railtie'
 gem 'statsd-instrument'
 gem 'twitter-text'
 gem 'tzinfo-data'
+gem 'webpacker', github: 'rails/webpacker'


Using the github: style here produces a security warning on bundle install -- using https://.... will fix it.

But also, I think this is on rubygems now: https://rubygems.org/gems/webpacker/versions/1.2

Okay. Just did what the webpacker readme recommended. Will gladly pin the version.

Yeah ... I think this was only possible once Rails 5.1 came out, and before that you DID need to pull from GH repo

mjankowski · 2017-05-01T18:15:52Z

app/javascript/styles/basics.scss

@@ -1,6 +1,6 @@
 body {
  font-family: 'Roboto', sans-serif;
-  background: $color1 image-url('background-photo.jpg');
+  background: $color1 url('../images/background-photo.jpg');


It seems weird that the stylesheets move into app/javascript/... -- is that normal for webpack?

mjankowski · 2017-05-01T18:16:17Z

app/javascript/styles/fonts/montserrat.scss

@@ -0,0 +1,11 @@
+@font-face {


Same question here ... it seems weird that the fonts move into app/javascript/... -- is that normal?

It's not normal but if I leave anything in assets it will get picked up by sprockets, I think, and then it would get processed twice.

I guess I would have assumed that the JS moves to app/javascript but anything which really is still just a static asset either a) stays in app/assets (and is made available to the JS, where needed? I think webpack does this...?), or b) moves to like - app/fonts or something?

The thing I'm asking about here isn't the move away from sprockets and to webpack, but about the naming convention of having fonts and images in an app/javascript dir...

It isn't too weird really, as some React practices involve bundling CSS modules directly with components, e.g. CSS and JS side by side, with any relevant images included. We don't use that but it's still all close together.

mjankowski · 2017-05-01T18:18:16Z

config/initializers/assets.rb

@@ -8,6 +8,6 @@

 # Precompile additional assets.
 # application.js, application.css, and all non-JS/CSS in app/assets folder are already added.
-Rails.application.config.assets.precompile += %w(application_public.js custom.css)
+# Rails.application.config.assets.precompile += %w(application_public.js custom.css)


is this left commented out on purpose, or should be a delete?

It's commented out by default so I left it commented out instead of deleting

mjankowski · 2017-05-01T18:20:32Z

Should we add a Procfile.dev (for people who use foreman in dev) or bin/server or some other way to include the running of the webpack-dev-server for people to get all in one process alongside the rails server and workers?

…ormance by using ImmutablePureComponent instead of React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in <UI />

…same time

mjankowski

I'm approving this on the assumption that we continue to find edge cases and work out best practices once the bulk of it is in.

nolanlawson · 2017-05-05T05:45:44Z

On second analysis, it looks like this didn't shrink the bundle size much. 😕

796K	application-6d6d82d4e4776ad9b07e.js
8.0K	public-39f8a254bfe861a34311.js
2.0M	vendor-5e5fbf294d06423aa79e.js

Looks like we're still at around 2.8MB of JavaScript total. I've run webpack-bundle-analyzer on the latest master branch (4d22d03); here's a live view of the data. A few things jump out at me:

emojione takes up a ton, hopefully Add support for ES modules (work in progress) joypixels/emojione#489 will get merged soon
all the various language JSON files take up a lot inside of Mastodon itself
buffer is unnecessary; it's required by an unused API in http-link-header
Lodash takes up a bit, worth looking into lodash-webpack-plugin and babel-plugin-lodash
Using Rollup, Mastodon's code could be one big index.js instead of many small modules

Mostly I don't see any easy wins here; just need to assiduously go through dependencies and try to remove what's not needed, replace where possible, etc. Code-splitting would also be good. Keep in mind that Sean Larkin from the Webpack team recommends <300KB (without gzip) for each chunk, whereas Addy Osmani on the Chrome team recommends <100KB to hit <3 seconds on 3G.

Since Webpack used from mastodon/mastodon#2617, it is necessary to start `webpack-dev-server`.

hinaloe · 2017-05-31T15:45:55Z

Dockerfile

@@ -32,14 +30,14 @@ RUN echo "@edge https://nl.alpinelinux.org/alpine/edge/main" >> /etc/apk/reposit
    imagemagick@edge \
    ca-certificates \
 && npm install -g npm@3 && npm install -g yarn \
- && bundle install --deployment --without test development \
- && yarn --ignore-optional --pure-lockfile \
- && yarn cache clean \


Why remove yarn cache clean task? for rails/webpacker#405 ?

Gargron added 2 commits April 29, 2017 12:28

Replace browserify with webpack

b43bb79

Add react-intl-translations-manager

57b4a3d

Do not minify in development, add offline-plugin for ServiceWorker ba…

3bb06c8

…ckground cache updates

ykzts mentioned this pull request Apr 29, 2017

Upgrade react-rails to v2.x #2465

Closed

Gargron added 3 commits April 29, 2017 15:38

Adjust tests and dependencies

7b323c5

Merge branch 'master' into feature-webpack

50561fe

Fix production deployments

ae6bdc9

Gargron mentioned this pull request Apr 30, 2017

Remove Intl polyfill #2633

Closed

Merge branch 'master' into feature-webpack

13a644f

Gargron force-pushed the feature-webpack branch from 0fbd693 to 13a644f Compare April 30, 2017 16:31

Gargron added 5 commits April 30, 2017 18:59

Fix tests

ea7681b

More optimizations

4e1e2ab

Improve travis cache for npm stuff

b531fd0

Re-run travis

d20bacf

Add back support for custom.scss as before

1aa7c10

nolanlawson approved these changes Apr 30, 2017

View reviewed changes

Remove offline-plugin and babili

56ece91

nolanlawson mentioned this pull request Apr 30, 2017

Use envify + NODE_ENV=production for smaller JS bundle #2635

Closed

mjankowski mentioned this pull request May 1, 2017

CLI utility to check coverage of a locale #1742

Closed

1 task

Fix issue with Immutable.List().unshift(...values) not working as exp…

949fccf

…ected

Gargron changed the title ~~[WIP] Replace sprockets/browserify with Webpack~~ Replace sprockets/browserify with Webpack May 1, 2017

Gargron added 2 commits May 1, 2017 18:17

Merge branch 'master' into feature-webpack

db5db35

Make travis load schema instead of running all migrations in sequence

2018526

mjankowski reviewed May 1, 2017

View reviewed changes

Gargron added 4 commits May 2, 2017 05:05

Fix missing React import in WarningContainer. Optimize rendering perf…

eca4e6d

…ormance by using ImmutablePureComponent instead of React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in <UI />

Merge branch 'master' into feature-webpack

c4c82c6

Merge branch 'master' into feature-webpack

d057a93

Add react definitions to places that use JSX

9e3e179

Gargron mentioned this pull request May 2, 2017

Increase React component spec coverage #2722

Closed

2 tasks

Gargron added 2 commits May 2, 2017 23:30

Merge branch 'master' into feature-webpack

d326aad

Add Procfile.dev for running rails, webpack and streaming API at the …

0008a2b

…same time

mjankowski approved these changes May 3, 2017

View reviewed changes

Gargron merged commit f5bf5eb into master May 3, 2017

Gargron deleted the feature-webpack branch May 3, 2017 00:04

yhirano55 mentioned this pull request May 3, 2017

update gems #2754

Merged

This was referenced May 7, 2017

Fix Node.js installation recommendation, also use 6.x mastodon/documentation#210

Merged

Use the production build of react #2763

Closed

Add a Service Worker for offline support #2917

Closed

toryalsip mentioned this pull request May 11, 2017

Assets fail to load #2973

Closed

y-yagi added a commit to y-yagi/documentation that referenced this pull request May 13, 2017

Add doc about webpack-dev-server

bb1e1a4

Since Webpack used from mastodon/mastodon#2617, it is necessary to start `webpack-dev-server`.

y-yagi mentioned this pull request May 13, 2017

Add doc about webpack-dev-server mastodon/documentation#243

Closed

hondash mentioned this pull request May 29, 2017

V1.4.1 custom hondash/taketodon#2

Closed

hinaloe reviewed May 31, 2017

View reviewed changes

olof-nord mentioned this pull request Nov 22, 2022

Update project to use Node.js v18.x #20968

Closed

Replace sprockets/browserify with Webpack #2617

Replace sprockets/browserify with Webpack #2617

Conversation

Gargron commented Apr 29, 2017 • edited Loading

ykzts commented Apr 29, 2017

Gargron commented Apr 29, 2017

ykzts commented Apr 29, 2017

Gargron commented Apr 29, 2017

ykzts commented Apr 29, 2017

Gargron commented Apr 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolanlawson Apr 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolanlawson commented Apr 30, 2017

Gargron commented Apr 30, 2017

Gargron commented Apr 30, 2017

kyzh commented Apr 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjankowski commented May 1, 2017

mjankowski left a comment

Choose a reason for hiding this comment

nolanlawson commented May 5, 2017 • edited Loading

Choose a reason for hiding this comment

Gargron commented Apr 29, 2017 •

edited

Loading

nolanlawson Apr 30, 2017 •

edited

Loading

kyzh commented Apr 30, 2017 •

edited

Loading

nolanlawson commented May 5, 2017 •

edited

Loading