TojiCode

WebGPU/glTF case study

2022-07-20T14:34:00.004-06:00

Over the last year or so I've been working on Chrome's implementation of the new WebGPU API, a successor to WebGL based on more modern graphics API patterns. While working on the API I've also been busy building a lot of demos and other community resources with it, and one topic has been brought up by people I've talked to probably more than anything else: How do you keep the API state manageable when loading something like a glTF model?

So I wrote about it!

If you're interested in WebGPU, glTF, or simply graphics on the web, check it out!

Efficently rendering glTF models: A WebGPU case study

WebXR Scene Optimization

2020-03-18T16:09:00.001-06:00

I posted a walkthrough of some basic steps for optimizing scenes for WebXR/WebGL on GitHub. Take a look!

https://toji.github.io/webxr-scene-optimization/

Early access to the WebXR Device API in Chrome Canary

2018-02-12T17:16:00.000-07:00

Hey, are you a web developer that wants to play with bleeding-edge browser VR APIs and influence their development? If so read on!

The WebXR Device API is the next iteration of WebVR, and the API that we intend to release in stable browsers at some point and support going forward. It’s still a work in progress, but the first (mostly) working implementation is now available in Chrome Canary for Android and Windows behind a flag, and so it’s the perfect time for motivated developers to start playing with the API and giving the Immersive Web Community Group feedback!

Getting started

The WebXR Device API is now available on Chrome Canary for Android with Daydream and Cardboard and Windows for OpenVR-compatible devices. (Basically anything that supported the WebVR API previously.) IMPORTANT: To enable the API on either platform you’ll need navigate to about:flags and switch the “WebXR Device API” flag to “Enabled”.

Additionally, on Windows you’ll also need to set the “OpenVR hardware support” flag to “Enabled” or the API won’t return any devices.

If you have an Oculus Rift or Windows Mixed Reality headset you can still use them with Chrome Canary’s WebXR implementation right now, but you’ll have to do so through OpenVR. That means you’ll need to have SteamVR installed.

Testing resources

While things are definitely early, there’s a few resources that you can use to get started.

WebXR Device API Samples (And the GitHub repo)
The API Explainer, which is the most complete document at time of posting
The WebXR Device API spec, which is a work in progress but may contain more detail on some specific items.
The spec issue tracker, for bugs related to the specification like missing functionality or questions about how something works (or is intended to work).
crbug.com for issues with Chrome’s implementation (Use the Blink>WebVR component)

What we’re looking for

We’ve spent the past year or so working on this new API, based heavily on lessons learned from what was and wasn’t working well with WebVR, so the Community Group is feeling pretty good about what we’ve created. But we also know that it’s impossible for us to account for everyone’s needs, so getting feedback from developers early and often is invaluable. We’re especially interested in hearing if there’s anything that you want to do with the API but can’t, especially if it was possible with the WebVR API previously. Beyond that, though, if there’s anything that the API allows you to do but makes unreasonably difficult or is simply difficult to understand how or why it works the way it does, we’d love to know!

Of course we can’t act on every feature request, but there’s still a lot of opportunity to help tune the API to better suit the needs of the development community before it’s final release.

What’s working and what’s not

Currently Chrome Canary’s implementation of WebXR should be able to handle pretty much everything described in the API explainer, which largely covers functionality related to drawing to the headset and magic windows. The main exception is that mirroring as described in the explainer is not working just yet.

The biggest overall omission is a new input API, since we’re still discussing how it should work. In the meantime, however, you can continue to use the Gamepad API extensions defined for WebVR. Also, this release does not yet cover any AR functionality because it to is still being defined.

It should also be noted that the API is still subject to change. (Otherwise your feedback wouldn’t do much good!)

History (A.K.A: Wait, another API? Why?!? What’s happened to WebVR?)

WebVR has been kicking around in one form or another for a while, and some really interesting content has been built with it, so it may come as a surprise to see that there’s a new API that covers the same capabilities in a different (backwards incompatible) way. To shed some light on that it’s worth doing a brief run through of the history of this new API. (This is the part that you can skip if all you care about is pushing pixels with the latest shiny thing.)

About a year ago what was then the WebVR Community Group was gearing up for what we hoped would be our first stable release of the WebVR API. Microsoft had joined the effort and pointed out some incompatibilities with Windows Mixed Reality that we were able to address (which is where WebVR “1.1” came from), and while we knew of some other potential issues we didn’t want to rock the boat too much because it felt like we were almost at the finish line.

It was at this point that we started to get some late feedback from various people that are more deeply familiar with the web’s conventions as a platform, who pointed out various aspects of the API we intended to ship and said, in effect, “This is not how The Web does things.” This was a little deflating initially but, crucially, it was also correct. WebVR was doing certain things in a way that went against the grain of how we collectively would like the web platform to behave. Certainly the web is not known as bastion of API uniformity, but given the opportunity we wanted to make things better, not worse. Especially for an API that may end up serving as a foundation for how the web interacts with a new computing platform. So we took the feedback seriously and went back to the drawing board to address the issues, even though we knew that it would break compatibility with existing content. (Better now before the API had officially been released than several years from now when the amount of content was exponentially more.)

And then, of course, the conversation turned into “Well, while we’re breaking things… can we fix this thing over here too?”

And while it was frustrating to realize that the finish line was a lot further away than we had thought, the opportunity to re-evaluate the entire API with fresh eyes and with modern VR and AR hardware in mind has proven to be extremely positive. The result is an API that is more secure, has the opportunity to be more performant, addresses a larger range of hardware, and is hopefully easier for developers to use as well.

As one example amongst many: the way that WebVR interacted with WebGL canvases was initially designed in an environment where the Oculus Rift DK1/DK2 used OS screen mirroring to show content on the headset, and Cardboard had just been announced. In that environment rendering to an on-screen, fullscreen canvas made a lot of sense. But the hardware APIs quickly moved past that to more direct presentation methods that solved a lot of problems, leaving WebVR with a bunch of awkward side effects to support a mode that nobody actually used anymore. With WebXR we refactored the rendering and presentation pipeline to more closely match what the underlying hardware is doing, giving the browser more control over frame submission and the developer more flexibility regarding how they produce and display rendered content while enabling far more API extensibility in the future. It’s a potential win for performance, usability, and API longevity, and we never would have been able to do it if we had stayed rigidly focused on compatibility with existing WebVR apps.

Speaking of which, WebVR isn’t going to simply disappear now that WebXR is becoming available. In Chrome we’ll have a transition period where both APIs are available for a few releases before we remove WebVR entirely, giving developers a chance to migrate existing content. Additionally, we’re also working on a polyfill that implements the WebVR API on top of WebXR (and visa versa) to make the transition easier. But if possible we would encourage developers to start looking at migrating content to the new API to ensure the best forwards compatibility. (It’s worth noting that it’s possible for a single web app to support both APIs, so adding support for WebXR does not necessarily mean abandoning support for WebVR just yet.)

Thank you!

Thanks to the awesome web developer community for helping support the development of the Immersive Web by being willing to put up with the API churn over the years. We know it’s not always easy to keep up with, but we’re heavily invested in making sure that we deliver a strong foundation for VR and AR on the web for years to come.

New experimental WebVR builds for Android

2016-12-13T23:23:00.001-07:00

It's an exciting time for WebVR on mobile right now! Today the Chrome team announced that WebVR is available as an Origin Trial in the Chrome 56 Beta for Android. Oculus has also recently launched their Carmel browser preview, and Samsung is continuing to improve their implementation. On the desktop side Mozilla is continuing to make great progress, and the spec is making huge strides as all of the interested parties (like Microsoft, whose input has been invaluable!) identify the rough points and flesh out the edge cases. The future of VR on the web looks bright!

And because we're so excited about that future, a few of us on the Chrome VR team wanted to give developers a peek at what's coming for WebVR performance beyond the Chrome 56 release. Browser release schedules being what they are we weren't able to get all of the optimizations for WebVR into Chrome 56 that we wanted and still hit the release date. We've already got some exciting proof-of-concepts for significant performance enhancements that we intend to make part of future builds though! So in the tradition of the experimental desktop builds that have bootstrapped the WebVR desktop development community we're making an experimental Android Chromium build available today on https://webvr.info/get-chrome (Look for ChromePublic-webvr-native-WIP-201*****.apk)

This build carries with it all the caveats of the previous experimental Chrome builds for Android. It's not intended for normal browsing, it's definitely a bit unstable, it doesn't have various patent-encumbered media decoders baked in, and it comes with exactly zero support. What it DOES have is our latest and greatest rendering pipeline research that we've found to yield pretty impressive performance wins.

I should take a moment and call out my colleague on the Chrome VR team Klaus Weidner for his excellent work in this area. I may have the public reputation for being the WebVR guy, but Klaus is responsible for a great deal of the WebVR rendering optimization on Android. He gave a great lightning talk about the work he's being doing at the W3C VR workshop this October, which you should check out if you're interested in that kind of thing!

There's some known issues outlined in the Release Notes you should know about, the primary one being that this is currently only for Daydream Ready devices (though a Daydream View is not required.) But if you're a WebVR developer with the right hardware take a peek and give us some feedback!

[EDIT: I also wanted to clarify the Origin Trial thing, since it appears to be a point of confusion. Origin Trials are for sites that want to make a feature available to all of their visitors by default. It's intended for use on pages like the ones linked below that are public facing and don't want to make users flip obscure flags to see the VR content.

If you're just doing development or testing of WebVR pages, though, you do NOT need an Origin Trial token! You can access the exact same features by turning on the "WebVR" and "Gamepad Extensions" flags in about:flags. (The Origin Trial covers both features).]

A couple of pages I recommend trying out with the new build

https://webvr.info/samples (of course)
https://www.clicktorelease.com/code/cruciform/vr/
https://threejs.org/examples/?q=webvr
https://media.tojicode.com/q3bsp/ (If you've got the stomach for it the touchpad on the Daydream controller will let you run around the level w/ click to jump!)

Happy WebVR-ing!

Update on WebVR spec, Chrome, and HTTPS

2016-09-09T11:29:00.001-06:00

(This post was originally sent to the public-webvr mailing list, which you should totally follow if you're interested in the latest and greatest WebVR news!)

Hello WebVR community!

The Chrome team has been a little quiet on the WebVR front recently, and I apologize for that and thank you for your patience. We’ve been heads down and working hard to deliver the best experience we can for developers as soon as we can. Several things have happened recently that will affect what we’re delivering and how. As such, we wanted to give developers an update.

Spec changes

First off, if you haven’t heard already, Microsoft has announced that they are working on WebVR support in Edge. This means that even wider support for WebVR apps across new and exciting hardware! It’s a huge deal for web developers, and reiterates the fact that the best way to reach the widest number of customers in VR will be via the web.

Microsoft also approached the WebVR community group with some very legitimate concerns about how their VR/AR platform would interact with the API as it exists today. Chief among these concerns was the fact that some hardware and APIs provide very carefully calibrated projection matrices for each eye per-frame that should be used for rendering. If those projection matrices are not used the scene may appear distorted or not align correctly with the real world, leading to user discomfort of varying degrees. Computing the projection matrix from the field of view as WebGL has done to date was simply not sufficient.

To address this concern and some others that were raised, as well as ensuring the API can evolve to encompass even more hardware features in the future, we’re making some changes to the WebVR spec prior to launch. I’ve been affectionately referring to this new version of the spec as “WebVR 1.1”, but in reality it’s just going to launch as “WebVR”. You can see the changes in the updated spec, but for those of you who would rather not analyse source diffs the TL;DR is:

VRDisplay.getFrameData() has been added. It populates a VRFrameData object with a pose and left and right eye’s projection and view matrices. To ensure that content renders correctly on all WebVR-enabled platforms the projection matrices must be used directly and using the view matrices or transform based off of them is highly recommended.
Your application will be responsible for allocating and providing the VRFrameData object. This allows you to more intelligently handle the object lifetime and avoid creating unnecessary garbage to collect.
VRDisplay.getPose() and VREyeParameters.fieldOfView are now deprecated. They will stick around in Chrome’s implementation for a while after we ship so that existing content doesn’t break immediately, but they may also be left off of other browser’s implementations at their discretion. Apps should use getFrameData() (which includes the same pose that getPose returns) and the projection matrices it provides instead. If your app still needs the field of view it’s recommended that you derive it from the projection matrix. We’ll make sure the algorithms to do so are in an easy to reference location.
VRDisplay.depthNear and VRDisplay.depthFar will now be used to compute the projection matrices returned by getFrameData(). Previously these field’s uses were vaguely defined (they were originally meant to support positional timewarp) but with this change they’ll become a much more important part of the API.
VRDisplay.submitFrame() will no longer accept a pose. This was more of an aspirational feature than one that had a practical current use and caused a fair amount of confusion amongst developers. Since not all platforms can support it it’s best to not suggest that it’s available. WebVR will assume that all submitted frames were rendered with the data supplied in the last call to getFrameData(). (Which is how basically every WebVR app works already anyway.)
VRDisplay.getImmediatePose() has been removed. I don’t think anyone actually used this outside of unit tests. In fact, I’ve already taken it out of the desktop experimental build and nobody seems to have noticed.

We realize that making any code updates for compatibility is a chore, and apologize for the inconvenience. I hope that you’ll agree, though, that wider hardware compatibility in the long term is definitely worth a bit of pain now, before the API has officially launched. We encourage developers to update your code and give us any feedback you have on the new structure of the API.

(I’m also personally curious to know how important compatibility with the previous API is to the community. Let me know if you have strong opinions on the subject!)

There’s actually a lot more than we want to do to expose even more features of the various VR platforms, but we’ve settled on the above changes to maximize compatibility while minimizing API churn. Rest assured we’ve got no shortage of ideas for an eventual WebVR 2.0!

Code updates

We’ve got a whole pile of sample code and library updates to go along with the new spec changes.

A migration guide is available to walk you through the basic steps needed to update your code.
All of the WebVR Samples have been updated to demonstrate proper use of the new API.
A shim library is available that “upgrades” implementations of the previous WebVR API to the new one, which should be handy while the browser catch up. (The WebVR samples are using it now.)
Three.js has been updated with support for the new spec.
The WebVR polyfill will be updated very soon to reflect the new API as well.

For what it’s worth, I found that I was able to update the WebVR samples in a few minutes apiece, and in many cases the resulting code was notably cleaner!

Schedule changes

I definitely view these changes as a positive thing for the WebVR ecosystem, but they are also coming fairly late in the process, and as a result we’re seeing the need to change our plans regarding launching in Chrome to allow for the new API to “bake” for a bit. In short, we plan to make WebVR available first as an Origin Trial. If you’re not familiar with the concept, it’s a way for Chrome to make experimental APIs available by default on domains that include an experimental feature token in the header. The tokens are issued freely and automatically from a short sign up form, and all that’s expected from the origins that use it is to provide some feedback on the feature (bug reports, feature requests, etc.)

(If you’re not planning on putting WebVR content in front of users you’ll still be able to turn on WebVR without a token via about:flags for development or local testing.)

We feel this is the right call for WebVR in Chrome at this point because the new API changes have yet to be tested in the wild and we want to ensure the community has ample time to alert us to any shortcomings or oversights. At the same time we know that the community has been chomping at the bit to get WebVR more widely available. Starting as an Origin Trial gives us the best of both worlds by allowing early adopters to make WebVR content available by default on their sites without locking us into any spec guffaws that we might miss in a rush to push through API changes at the last second.

I’m still in the process of bringing up experimental builds with the new API, but I’ll be sure to let everyone know once they’re available!

Secure Origins

One final thing that I’m eager to update the community about is Chrome’s policy when it comes to using WebVR on non-HTTPS domains. I need to apologize for this one, because after some initial miscommunication behind the scenes and a public, “Oh, hold on, we spoke too soon,” we left things in a state of limbo for over a month. I definitely could have done a better job of communicating while we worked out the details with Chrome’s security team. I simply wanted to ensure that whatever we said next on the subject was clear, accurate, and reliable. I’m sorry to leave you all hanging!

The good news is that we are going to stick with the policy that we announced last time: WebVR will be allowed on all HTTP domains with a persistent overlay to alert users to the non-secure origin. That includes HTTPS content embedded on HTTP domains. This policy is the result of careful consideration on the part of Chrome’s security team, and I’d like to thank them for their collaboration and patience while we forge paths into this new and uncharted territory for the Web.

From the last round of emails I know that there were several questions about the behavior and appearance of the security overlay. I’ve put together a simple demo to show the general direction that we’re leaning at this point. Try it with a WebVR experimental build on desktop or with any mobile device and Cardboard. It’s not meant to represent the final UX, we’ve still got some usability testing and refinement to do, but it should give you a decent idea of where we’re headed.

It’s worth noting that this policy does not mean that you don’t need to bother with HTTPS for your WebVR apps. Hosting your content on a secure origin should be considered a baseline requirement for anything that you intend to share with the web at large. What this policy enables is a simpler development cycle, learning curve, and troubleshooting process. Security is still critical, it can just be step 2 now instead of step 0 for new projects. Looking for resources on getting started with HTTPS? Check out Google Developer’s Security and Identity docs.

As HTTPS becomes more prevalent and frictionless, and as the development process on secure origins becomes even easier, the Chrome team does intend to continue to deprecate powerful APIs in non-secure contexts, and eventually may lean toward requiring HTTPS for all new APIs. We don’t have a timeline for deprecating WebVR specifically (might be years down the road) but please be aware that eventually APIs such as WebVR and Fullscreen may only be available over secure contexts.

Thank you!

So, as you can see we’ve been incredibly busy and aren’t slowing down any time soon! Thank you for being such an awesome community and for throwing so much of your incredible talent at an API that’s still undergoing some growing pains. Bringing VR to the web is just a teensy bit of a crazy endeavor, but the passion, skill, and creativity that we see from the community assures us that it will be well worth the effort!

Oculus Rift and HTC Vive review

2016-04-08T17:45:00.000-06:00

By virtue of the WebVR work I've been doing I've been lucky enough to have a fair amount of access to VR hardware, and at this point feel like I've got a pretty good handle on where each of the newly released devices stand in relation to one another. Since it seems to be a popular topic lately I figured I'd give a brief overview of my thoughts on the Vive and Rift.

To start with, they're both fantastic devices. I doubt that you'll be disappointed in either of them unless you're looking for some very specific things. I've seen so much obsessing over field of view and pixel density and other crap recently. News flash: you won't care. You'll put on the headset of your choice and say "Aw... this field of view isn't what I was hoping for! And look, I can totally see pixels if I stare hard enough!" and then the content will actually start up and you will probably never think about it again. Do I want these things to improve in subsequent hardware generations? Sure! Is it a problem for this generation? Not in the slightest.

More meaningful is the difference in the industrial design and ergonomics. The Vive is a overall a comfortable device to use, but it takes a bit of fiddling with the straps to get it so it's not sitting on your face uncomfortably. Even once it's on it feels a little front heavy. It's not bad, really, and the only reason I bring it up at all is because the Rift is fantastically comfortable. The rigid straps allow you to easily get it on in exactly the right way so that the weight is primarily on the top strap, the back of your head, and your forehead. It feels like wearing a large hat, rather than goggles. The Rift is also undeniably a sexier piece of hardware. I think the Vive actually looks fine, but next to the Rift it seems like a really polished dev kit.

The exception on the ergonomics front is glasses: I can use my glasses with both headsets and while it's not perfect it's not annoyingly uncomfortable either. When I take off both headsets my glasses usually come off with them, so I have to pull them out from the facial interface. On the Vive it's no big deal but the Rift holds onto my frames for dear life. It's seriously hard to extract them from the facial interface without feeling like I'm damaging something. :( My glasses are pretty wide (146mm), so your mileage may vary, but it does make me wonder why Oculus didn't follow through on the idea of including multiple facial interfaces.

The headphones also make a big difference. I have nice headphones. I like to use them whenever possible. But I'm super glad that the Rift has integrated headphones because it's one less thing to worry about when moving in and out of VR. It's a small thing, but fumbling around for headphones after you've got the headset on, managing the extra wire coming off the back of your head, and remembering to take things off in the right order is just enough of a pain to make integrated audio extremely welcome. (And it doesn't hurt that they sound pretty good too!) The camera on the Vive does make up for it a bit, and it's clearly a feature that would improve the Rift. It's honestly a bit odd that they left a camera out after having the pass-through was such a well regarded feature for the GearVR.

Oh, and I can't forget to mention how much easier the Rift is to set up. The Vive set up process is nothing terrible, there's just a fair amount of stuff to deal with. Base stations, controllers, breakout boxes, wires everywhere... I've taken mine back and forth from my home to my office multiple times and each time setup or teardown is about 20-30 min. And that's with experience doing it! If you're not planning on moving your VR setup around like I do, though, it's a complete non-issue. The Rift, by contrast, is super fast to set up and tear down and has a really nice carrying case to boot! I suspect that you'll see a lot more Rift's than Vive's at various meetups, conferences, and hackathons simply because it's easier to transport and set up.

This level of hardware polish extends to the Rift's upcoming hand tracking controllers as well. The Vive controllers are great! No real complaints to be had. They feel good in the hand, are super responsive, and extremely intuitive. In the brief chance I had to play with Oculus Touch, however, I found it to be more comfortable to hold, lighter, and the triggers to be easier to use (especially the grip trigger.) And that was just a prototype! I'm sure the final consumer release will be even better... whenever that actually happens.

(Fanboy trigger warning: This is the part where things stop being so glowingly positive for the Rift.)

See, here's the problem: I've shown a lot of people the Vive and the Rift. Every single one of them has been blown away by the Vive. People who were complete VR skeptics have walked away going "Wow, I need to find some space in my house." On the flip side, when I show people the Rift (usually not the same people I've shown the Vive to) I place the Oculus Remote in their hands to navigate and every. single. one. of them immediately waves it in front of their face and frowns in disappointment. They watch some demos or play a game and walk away saying how impressive it is, but almost everyone makes a remark about how they wish they had hands. Similarly everyone that tries the Vive is delighted that they can walk around, while everyone trying the Rift comments about how they're scared to move for fear of smacking into something.

As it exists today, VR on the Rift is "really impressive" and VR on the Vive is "OMG this is incredible!" and it comes down to tracked hands and Chaperone. When you feel safe walking around a space and can actually reach into it with your hands it's an entirely different level of immersion. That's not to say the Rift isn't good at what it does. Lucky's Tale is great fun, and anything that sticks you in a cockpit is a wonderful fit for the device. But when I think about my favorite experiences in VR to date it's shooting down drones in Space Pirate Trainer, building crazy vehicles in Fantastic Contraption, painting with light in Tilt brush, or doing whatever you do in Audioshield. (Rhythm boxing? Musical defense? Audio deflection?) None of that is possible on the Rift today, and I feel it's absence intensely.

The good news is that it's hopefully a temporary problem. Touch is coming, and I have every expectation that it will be fantastic. And if Oculus doesn't provide some sort of Chaperone system when Touch is launched they've screwed up big time. I can't imagine that they won't. Even then, though, some problems will remain: It's harder to set up the constellation sensors in a configuration that minimizes occlusion and it's not clear what percentage of the user base will pick up Touch, which may make it a less attractive development target. I really hope that people do, because it makes a world of difference, and I would like nothing better than for VR with tracked hands at room scale to be the effective baseline for desktop VR.

Looking beyond the hardware, I feel that Valve's software is simply more mature than Oculus'. Sure, Oculus Home is a gorgeous environment, much nicer than SteamVR's photosphere. But why does the entire store need to launch every time I start a VR app? And why does the Oculus Store window pop up over the top of whatever I'm doing if something happens to trigger the proximity sensor? Features like friends lists are pretty sparse in their functionality, the lack of any text input in the Home environment is frustrating, and it has weird bugs like rendering that beautiful home in the background even when you're not using the headset. On the other hand SteamVR on the other hand is, well, Steam! While the VR variant still has some quirks it's basically the same system you know and love with some new tricks. The theatre that I can play 2D games in is great, even if it's not a pretty environment, and having basic desktop usage that you can call up at any time is a life saver. Especially for development!

(And I have bigger complaints on the development front, but that's probably material for another post.)

Again, this is all fixable on Oculus' part. There's no fundamentally unsolvable issues here, and I'm hoping that in 6 months time most of this post is obsolete. But it as of today the Vive feels like a more complete hardware and software package, even if Oculus wins on aesthetics.

The most important takeaway, in the end, is that the current gen of VR is compelling no matter which hardware you experience it with. Nobody is poisoning any wells. Both devices have their pros and cons, but neither are bad by any stretch of the imagination. If I could put together a dream system it would probably be a Rift+Touch, tracked by Lighthouse sensors and using SteamVR. Back here in the real world, though, I think the Vive is going to stay plugged into my PC a lot more than the Rift for the time being.

Moving towards WebVR 1.0

2016-02-29T09:44:00.000-07:00

Consumer VR is at our doorsteps, and it’s sparking the imagination of developers and content creators everywhere. As such it’s no surprise that interest in WebVR is booming. Publications like the LA Times have used WebVR to explore the landscape of Mars, and a doctor was able to save the life of a little girl by taking advantage of Sketchfab’s VR features. The creativity and the passion of the WebVR community has been incredible!

Those of us who have been defining and implementing the API feel a responsibility to make sure that it keeps pace with the current state of VR hardware and software. That’s not always easy with the breakneck speed with which the field has been evolving, and as we look at the API as it exists today there’s some significant disconnects from the realities of modern VR.

A quick refresher about how we arrived at the point that we're at now: When WebVR was first conceived by Vlad Vukicevic (April 2014) Oculus had just barely announced the DK2 and the only VR headset most people could get was the DK1. The Vive, 6DoF controllers, Hololens, and GearVR were still behind closed doors at this point. Cardboard had only just been announced when we first started making builds available. The APIs used to interact with the hardware that was available looked very different than it does today. That’s why in my initial blog post about the API I said “Keep in mind that these interfaces absolutely, without question, WILL change”.

We’re taking that sentiment to heart, and in the interest of keeping WebVR relevant and (hopefully) a bit more future proof we’re proposing some major, backwards-compatibility-breaking changes. You can see the new proposed spec here, but I wanted to cover some of the changes in a bit more detail and go into the rational behind them.

Proposed changes

Renewed focus on WebGL content for WebVR 1.0

The idea of displaying DOM elements in VR is an attractive one, and on the surface it feels like it should work fairly naturally with existing features like 3D CSS transforms. The devil is in the details, though, and there’s a lot of subtle issues that make the jump from a 2D screen to true 3D space difficult. We would still like to see DOM content in VR someday, but in the interest of delivering a stable, useful API as soon as possible we’re putting that functionality on hold and focusing exclusively on presenting WebGL content for WebVR 1.0. This will allow us to rally our efforts around supporting one type of content really well rather than multiple (and completely different) types of content poorly.

If you are more comfortable with declarative syntax instead of raw WebGL or Three.js there are great libraries like A-Frame available to help you build that kind of content.

Combine VRPositionSensorDevice and VRHMDDevice into one object

We always wanted the API to be flexible so that as hardware gained new capabilities we could easily expose them. The original idea was that each VRDevice variant (currently PositionSensor and HMD) would represent a feature of the hardware, and all the features associated with a piece of hardware could be identified by a shared hardwareUnitId. This required users to iterate over the enumerated devices multiple times to figure out what a single piece of physical hardware actually looks like.

In practice it’s exactly as awkward as that sounds.

This is one of the areas that we’ve most frequently seen confuse newcomers, and even more experienced devs get it wrong sometimes. As such, we’re discarding this loose association approach in favor of getting back a single object for each piece of VR hardware, now referred to as a VRDisplay.

Remove reliance on requestFullscreen for presentation

The way content is presented to HMDs is probably the area that has evolved the most since WebVR started. When we first started releasing WebVR-enabled builds the DK1 was only recognized as an external monitor. In early experiments people would set up their displays in mirror mode in order to present their content. Given that environment the idea of latching VR present to fullscreen made a lot of sense, and that idea was somewhat validated by Cardboard's emergence. So we ran with it.

That approach has since proven to limit what type of content can be displayed and how developers can present it, and so we’re making some significant alterations to the VR presentation method.

With the new API you will now get a compositor object from the VRDisplay. In order to prevent abuse of the API and unexpected transitions on Cardboard-style platforms the compositor will only be able to be retrieved during a user gesture (the same limitation exists for the Fullscreen API) On the compositor you’ll be able to set the source of the VR frames you’ll be rendering (currently limited to either an HTMLCanvasElement or OffscreenCanvas) and some options such as the viewport to be used for each eye.

The compositor will have it’s own requestAnimationFrame function which will fire at the refresh rate of the VRDisplay rather than the normal refresh rate of the system. Developers will call requestPresent to begin displaying on the HMD and must call submitFrame to explicitly indicate to the compositor when the canvas contents are ready to be presented.

This approach allows for some important improvements:

On desktops developers can now control how content is mirrored to external displays. (If you want to mirror it at all.) You can control where it is displayed on the page and how large it is, decide if you want to show both eyes, one eye, or even a completely different view of the scene, depending on your sites needs and performance budget.
Relatedly, it now allows interaction with the page from external displays while the VR experience is being presented, which is great for guided demos.
It allows for more efficient rendering of monoscopic scenes, which can be a great performance enhancement for things like photospheres and 2D videos, especially on mobile devices.
Because the new presentation method isn’t subject to the same security restrictions as Fullscreen we can allow the browser to stay in VR mode between sites, enabling linking from one page to another without leaving VR.

Replace DOMPoint with TypedArrays

WebVR has been using elements of the Geometry Interfaces spec till now to represent vectors and quaternions, but depending on them is probably not wise moving forward. The spec doesn’t have universal support from browsers yet, and areas of it have proven contentious, slowing adoption. Additionally, there’s overhead associated with creating a number of these objects each frame that we’d prefer to avoid in code that will definitely be on performance-sensitive paths. As such, we’re changing WebVR to represent vectors, quaternions, and matrices with Float32Arrays. These offer a number of performance benefits, but the most compelling is that they allow for easy use of upcoming SIMD APIs for hardware accelerated vector math.

This may seem like a pain, but in reality the changes required to account for a it are likely to be minimal. For example, if you were using WebVR with Three.js previously where you used to call Vector3.copy() to get the values from the DOMPoint into a Three.js vector now you’ll call Vector3.fromArray() instead.

Enable “Room Scale” experiences

Some VR devices, like the HTC Vive, are designed to allow for VR experiences that let you walk freely around a large area, commonly known as “Room Scale VR”. One important factor to enabling these types of experiences is to deal with a coordinate system that is centered around the users physical space. By placing the origin at the center of the users room on the floor, you can easily build your virtual space around it (by placing your virtual floors at y=0, for example). This helps the user feel grounded in the virtual world.

Since not all VR devices have the necessary information about their environment to enable this mode, WebVR will report position and orientation centered around the headset’s position when resetPose was last called. (OpenVR refers to this as “Sitting space”) On systems that support room scale VR, though, the VRDisplay’s stageParameters object will provide the information needed to transform those values into “Standing space”, with an origin at the physical center of floor of the users play space. This includes a transform matrix and the play area dimensions, so that content can be scaled accordingly.

Clearly report capabilities

A newly added capabilities property on the VRDisplay indicates whether or not the device can report positional and/or orientation data, and whether or not the VRDisplay has an external display (such as a desktop monitor) attached that allows for content mirroring.

Renaming and reorganizing for clarity

We’re also taking the opportunity to revise naming of various objects and concepts in the API to better match the underlying APIs and conform to more common verbiage. For example: Previously we referred to the orientation/position data returned by the headset as “PositionState”, which is an obvious misnomer because position was only part of the information returned. We now refer to it as “Pose”, a name which describes the data better and is common to most underlying APIs.

Removing lesser-used and poorly supported features

In order to simplify use of the API and make sure that it covers as wide a range of hardware as possible we’ve removed some minor features of the old API. Most notably, we’ve taken out the ability to set the HMD’s field of view, and in conjunction have removed the max/min/recommended/current field of view parameters in favor of a single fieldOfView property per eye. This makes the API simpler to use and implement, and reflects the reality that APIs like OpenVR don’t allow for user specified FoV values.

Controller support

With both the Oculus Rift and HTC Vive coming out with 6DoF controllers interest in accessing them on the web is naturally high. While these controllers are tightly coupled with VR devices, though, we feel strongly that they should be exposed through the existing gamepad API and not as VR devices. As such controller support is, strictly speaking, not part of WebVR 1.0.

That said, we are still very interested in seeing these devices be supported and will be working with the appropriate people on the gamepad spec to try and extend it to provide the functionality needed. We simply won’t be blocking WebVR on getting that support in place.

Trying it out today

I’ve uploaded new Chrome builds (Windows only) that implements the majority of the new API so developers can begin to work with the new API. Support isn’t perfect yet, but I already feel like the new API allows for much better handling of the underlying devices. And, as always, the code for these builds is available in a Chromium experimental branch for those that are interested. The API changes will be making their way into Top-of-Tree Chromium soon and become available on Android Canary and Dev builds shortly after that, but the Oculus and OpenVR backends are still a little ways out from making their way into the Chrome trunk.

(A note on compatibility: At this time I am only going to be supporting Windows and Android through the experimental binaries. I know that this is disappointing to many developers, but if you've been following VR development lately you know this is more of an ecosystem issue than a choice on my part.)

We’ve also been working to ensure that some of the more popular WebVR libraries are updated to work with the new spec. In some cases the only change developers may need to make is to download and deploy the update version of the library. Three.js and a branch of webvr-polyfill have been updated to support the new API, and support from other libraries is incoming. Also worth noting: the updated webvr-polyfill can wrap the new API with the old interface when provided the ‘ENABLE_DEPRECATED_API’ config flag, which may be helpful for backwards compatibility while you update your code to the new API.

To help demonstrate the version 1 API and provide a reference point for developers we’ve also been working on a new set of WebVR samples. (See the live versions here.) These samples are all single page applications that demonstrate a specific aspect of the API. They use raw WebGL instead of a framework or engine, and keep support libraries to a minimum in order to focus on the API calls in question instead of boilerplate. The samples are a work in progress, so expect to see the list of them grow over time. (And if you have suggestions for a topic you’d like to see covered let us know via the issue tracker!)

Seeking community feedback

While we feel confident that these changes will allow WebVR to keep up with the ever changing VR landscape we also can’t anticipate every possible use case. We would love to hear from you, the WebVR community, about how these changes will affect your content and especially if they make the content you hope to produce unreasonably difficult or impossible. We know that these are breaking changes and that existing content will require updates, and we’re very sorry for the inconvenience! We do feel strongly, though, that it’s worth breaking some content that was built while the API was still labeled “experimental” for the sake of future-proofing.

We’re excited to get WebVR to a point that the API is stable and widely accessible, and this is the first step along that path. Thank you for your patience and support, and keep building amazing VR content!

Bringing VR to Chrome

2014-07-02T13:37:00.000-06:00

[Update: WebVR has evolved quite a bit since this post! Read about the latest changes.]

The Good Stuff

Let's get all the links out of the way upfront, because that's what you're really here for, right?

WebVR-enabled Nightly Chromium Builds
Go here for builds

Source
Chromium WebVR experimental branch

Demos (Should work with Chrome and Firefox's WebVR implementation)
Quake 3 level renderer (Source) - Click the "VR" button in the lower right
WebVR test page (Source) - Simple test apps

Related Links
blink-dev "Intent to Implement WebVR" thread
Vladimir Vukićević blog post about Mozilla's WebVR implementation
web-vr-discuss mailing list - If you have feedback about WebVR in general, this is the place for it
Chrome WebVR bug tracker - If you have issues with Chrome's WebVR implementation, log them here.

WebVR: What it is and what it isn't.

To start out, I'm going to highly recommend that you go read Vlad's blog post about Mozilla's WebVR plans first. Chrome's current implementation is very close to Firefox's, so it doesn't make sense to re-state what's already been written. (I'll discuss the differences in implementations in a moment.)

One thing that's worth talking about, based on the reaction I've seen to Vlad's post, is what a "VR-enabled browser" actually entails. There were several people that seemed to expect that this meant you would start up your browser, put on your headset, and immediately be immersed in a fully VR browsing experience. Sorry, but no. That day is a long ways off, if it ever gets here at all. (If you are interested in something like that, check out Janus VR.)

Instead, just like adding WebGL to the browser doesn't magically make everything 3D, adding WebVR doesn't turn everything into a Virtual Reality experience. Instead, it provides an API that allows developers to create VR content in the context of a web page. Picture this: You are browsing Amazon and find a jacket/TV/bike/whatever that you're interested in. If Amazon's developers took advantage of the WebVR API they could add a button that says "View in VR" which let you view the item through a VR headset in 3D at 1:1 scale. In the case of a piece of clothing you could see it on a virtual mannequin, walk around it, lean in and examine the stitching, and so on as if it were actually sitting right in front of you. You could also imagine similar experiences with educational tools, data visualization, mapping, and so on. WebVR gives developers the tools needed to make it happen.

And of course there will be games. That's such a given it's not even worth mentioning.

The State of WebVR in Chrome

The above builds are created from an experimental branch of Chrome. This means that WebVR features won't be added to Chrome proper until the API has matured a bit, and then only if there's sufficient developer and user interest. It's still an open question as to whether or not VR will take off this time around or flop again like the last several attempts. It doesn't make sense to add significant new features to Chrome if only a tiny fraction of enthusiast users will ever be able to make use of them. In addition, there's very few VR devices that are openly accessible to developers at this point (really just the Oculus Rift and, um... Google Cardboard?) so it's not clear what the common functionality will end up being. If we commit to an API too early we risk excluding devices that don't conform to assumptions made from a very small sample set.

If we wait until VR becomes ubiquitous before considering making it part of the platform, however, we'll already be hopelessly behind the curve. Maintaining an experimental branch of the browser for VR development gives us a way to work out the kinks in the API implementation without interfering with the core code base prematurely.

Currently the builds support the Oculus Rift, and I'm going to be looking at adding support for Cardboard's SDK as well, and I'm happy to looks at other VR solutions as well assuming that they have an open source SDK. I should also mention that this is a side project for me. My normal responsibilities of making WebGL even more awesome take precedence, and so I can't promise timely integration of every cool VR gadget that hits the market.

I will periodically update the builds linked on this page to ensure they stay reasonably up-to-date with future Chrome development, but I should also stress that these builds are intended for development and testing only! They have not been vetted for stability or security, so please don't use them as your day-to-day browser.

The "L" word

Ah, yes. Latency. The first and last thing anybody brings up when talking about VR development. By now you've probably all heard that 20ms is the magic number for achieving that mystical "presence" we all want. The problem is that browsers on the whole are, to put it mildly, not known for their spectacularly low latency. So how does Chrome do?

64ms from headset motion to LCD change. For reference on the same machine the Tuscany sample gives me 48ms. This actually isn't quite as bad as it may seem at first. That means that Tuscany takes 3 frames on a 60Hz display (like the DK1) for your head motion to affect the image on screen. Chrome, then, is taking 4 frames. 1 frame of difference vs a native app is definitely not insurmountable. Of course, both of those are a long way from the holy grail of 20ms, but that's a goal that everyone is still struggling to hit.

So how do we reduce the latency in the browser? That's a far more complicated question than it may seem, and there's a lot of people on the Chrome team who are working on that problem in various ways. There may be some shortcuts that VR content can take to get ahead, though. Chrome has a pretty hefty rendering pipeline that's geared around rendering traditional web content (rectangles, images, and lots of text) as smoothly as possible. This involves a lot of layout, rasterization, and compositing, which is great for that type of content but doesn't really do anything for you when your content is a single, full screen WebGL context. As such, there may be some gains to be had by short-circuiting the compositing process in that special case. I haven't had the opportunity to actually try it yet, but it's an example of the types of optimizations that can be attempted in pursuit of that elusive "presence".

Implementation Differences

Chrome's experimental implementation of WebVR is intentionally very similar to Mozilla's prototype, but there's currently some minor differences due to the parallel nature of the ongoing development. It's perfectly possible to build web apps that work with both right now, and I expect the differences will get ironed out as developers give us a better idea of what's important and what's not.

(Keep in mind that these interfaces absolutely, without question, WILL change. This is just a description of the initial versions.)

First off, since it's Blink's policy to no longer add prefixed APIs, none of the functions begin with "webkit" or "blink" or anything like that. Instead, in order to access them you will need to launch Chrome with the --enable-vr flag. (That's not required for the builds linked above, since their sole purpose is to test out WebVR)

getVRDevices in Chrome returns a promise, rather than accepting a callback. This is because:

Many new Web APIs are trending towards Promise use for asynchronous behavior.
Promises have explicit "success" and "failure" paths, which fits the use case well.
Promises are specced to succeed or fail exactly once, which resolves ambiguity about wether or not a passed callback may fire on an ongoing basis (like when new devices are connected).

To write code that works with both browsers is trivial:

function EnumerateVRDevices(devices) {
// Do stuff with the devices
}

if (navigator.getVRDevices) {
navigator.getVRDevices().then(EnumerateVRDevices);
} else if (navigator.mozGetVRDevices) {
navigator.mozGetVRDevices(EnumerateVRDevices);
}

Chrome's PositionSensorVRDevice is almost identical to Mozilla's but includes a resetSensor function. Calling this function resets the HMD sensors to consider the user's current yaw "forward".

Chrome's HMDVRDevice has a few attributes Mozilla's lacks, and is missing a couple that are present in Mozilla's code. ~~Chrome has displaySize and renderTargetSize vectors~~ to indicate the resolution of the HMD display and the recommended render target size respectively. The render target size is expected to be larger than the display size, and is calculated so that the rendered output has a 1:1 pixel ratio with the display at it's most distorted point (near the center of the eye) once the resulting image has been run through the distortion post-process. It's worth noting that the devicePixelRatio should not be taken into account when setting the size of a canvas for VR rendering.

[UPDATE: Chrome no longer support displaySize, since it doesn't have any effect on rendering. renderTargetSize was changed to getRecommendedRenderTargetSize() to account for the fact that it's value changes when you change the Field of View.]

Chrome currently lacks Mozilla's finer control over field of view. You can call getRecommendedFieldOfView, but there's no getCurrentEyeFieldOfView, getMaximumEyeFieldOfView, or setFieldOfView. All rendering in Chrome currently assumes that you are using the recommended FOV.

[UPDATE: Chrome now support setFieldOfView, getMaximumEyeFieldOfView, and getCurrentEyeFieldOfView.]

Finally, more for my debugging purposes than anything else, with Chrome you can pass vrDistortion: false into webkitRequestFullscreen along with vrDisplay: hmd to prevent the distortion post-process from running. This is useful for checking rendering results, but can also be used if you prefer to implement your own distortion pass. The API doesn't currently provide any information about the desired distortion formula, however, so doing so will be a "best guess" affair, possibly based off of the HMD name. It's highly recommended that you let the browser do the distortion for you.

[UPDATE] I did a terrible job of explaining this and it's already causing confusion, so let me explain the Fullscreen API better. There are two functions in Chrome that you can use to go fullscreen: webkitRequestFullScreen and webkitRequestFullscreen (Note the lowercase "s" in screen). The former (capital "S") is deprecated while the latter (lowercase "s") conforms to the W3C spec. I have only set up the conformant version to work with the VR API. If you use the other one you'll go fullscreen but not get the distortion effect. So in short, use the following for cross-browser VR fullscreen:

if (canvas.webkitRequestFullscreen) {
canvas.webkitRequestFullscreen({ vrDisplay: hmdDevice });
} else if (canvas.mozRequestFullScreen) {
canvas.mozRequestFullScreen({ vrDisplay: hmdDevice });
}
[/UPDATE]

I should also point out that Chrome supports the Oculus DK1 latency tester with no effort on the developers part. That may go away in the future, since it's not very generic behavior, but it's useful for the time being.

Let me reiterate that all of this is subject to change based on developers needs.

Known Issues

The Chrome builds linked above definitely have a couple of issues right now that will be fixed in the near future. First and foremost is that when you call webkitRequestFullscreen({ vrDisplay: hmd }) the canvas does go fullscreen but it doesn't automatically show the content on the HMD. You either have to move the window to the HMD display prior to going fullscreen, which is a major pain, or use screen mirroring. Neither option is great, but hopefully that situation is short-lived. The goal is that when you go fullscreen with a VR device the content automatically moves over to the HMD display and then moves back to the main display when you hit escape.

Oh, and passing a HMD into the fullscreen API only works with canvas elements that have a WebGL context right now. This restriction is likely to stay in place for a while while I watch Mozilla's work with using 3D CSS content in VR. To be blunt: I have no idea how that will work out from a UX perspective, and since Mozilla has more man power on WebVR I'm happy to let them feel it out first.

Also, the final output image after the distortion is done is the same size as your render target, not the size of the HMD display. I think this might screw with chromatic aberration correction, but I'm not sure. It's not optimal from a memory usage standpoint, certainly, and should be corrected in the near future.

Finally, since this is built against a nightly build of Chromium and not a stable branch you should expect all the quirks that come with using a nightly build, most of which will not be related to the WebVR implementation.

How can you help?

If you're excited about this technology and want to help push it forward, that's awesome! Probably the single most influential thing you could do to help the cause is build something awesome that uses WebVR! Whether it's hacking support into an existing app or creating something entirely new, the more use cases there are the more compelling the API becomes.

If you want to try your hand at hacking on the Chrome implementation directly, you can do so by following the normal instructions for your platform to checkout and build Chrome (Warning! Long build times ahead!) Then check out the WebVR experimental branch over the top of it by doing the following:

cd <chrome_src>
git remote add -f wip https://chromium.googlesource.com/experimental/chromium/src
git fetch wip "+refs/wip/bajones/webvr_1:refs/heads/webvr_1"
git checkout webvr_1

You should now have a webvr branch in your chrome repository that you can checkout and develop against. See Chromium Experimental Branches for (slightly) more details.
Beyond that, if you find issues in the implementation please be sure to report them to the appropriate places (Chrome, Firefox). If you have questions or suggestions about the API, please join the discussion on the web-vr-discuss mailing list. If you want to voice support for seeing the API land in Chrome's trunk you can comment on the Intent to Implement email, but really this is a case where actions speak louder than words. If there's sufficient developer activity around the API it will find it's way into the official channels naturally.

I'm really excited to see what the development community does with VR on the web. I've been consistently blown away at the creative genius displayed by the WebGL community and can only hope that we see even a fraction of the same ingenuity applied to VR content. There's still so much to discover in terms of VR best practices and conventions, it seems only natural that the Web become a test bed for this up-and-coming tech!

[Update: Blink was recently merged into the Chromium repo, which removed the necessity for two different experimental branches. This article's links/instructions has been updated to reflect that.]

Crowdsourcing Unreal Tournament

2014-05-11T16:29:00.003-06:00

I spent an awful lot of time from my middle school years forward learning everything I could about the mechanics of making video games. The goal being, of course, to make my own. And not some silly small asteroids game or anything like that. No! I wanted to build Shooters and Platformers and RPGs. The epic kind! With lots of story and cut scenes and hours of gameplay in massive open worlds! (Ah, the joys of blissfully and unknowingly throwing oneself at impossible tasks.)

As you might imagine, I failed spectacularly at this effort. Luckily for me I never quite viewed it as a failure. It was just a matter of not yet knowing enough about subject X or software Y or technique Z. So my quixotic quest actually propelled me to learn a tremendous amount about programming and graphics development and all manner of related subjects, a fact which I am enormously grateful for today as it has directly led to my current employment. And even today I still have a tendency to chase after projects that are too big for me to reasonably handle in the free time I have available. But that's okay, because this is one of those cases where shooting for the moon really can land you among the stars, and it gives me a wonderful excuse to try all manner of fun, crazy things that would never come up in the course of my day-to-day job but still prove useful anyway.

Given this background, I couldn't help but feel the tiniest bit sad that Epic didn't announce their latest engine pricing or their new crowdsourced path for the next Unreal Tournament about 15 years earlier. Teenage me would have been all over that! I spent a decent amount of time hacking with the Quake 3 SDK and other similar code bases, mostly in an effort to learn more about how they worked and how "real" programmers did things. The idea that I could have access to the full source of a modern game engine for a few dollars a month would have been a dream come true! And beyond that, the fact that I could have not only watched but participated in, from day 1, the development of a AAA game would have probably killed me with giddiness. That was exactly what I wished I could have back in the day: A perfect little portal through which I could view the professionals at work creating something real, large, and "cool". I think I would have poured many weeks of my life into following the project and trying to contribute in any way I possible. (I shudder to think of 15 year-old me's code actually ending up anywhere public though. I was better than a lot of people gave me credit for at that point but I wasn't nearly as good as I thought I was.)

I'm still contemplating getting a subscription just to peruse through the code and maybe contribute a few fixes here and there, though my practical side reminds me that I know I won't have the time to participate like I want to. I'm still super excited to see how Unreal Tournament comes together (and I'm crossing my fingers that this grand experiment doesn't implode, if only because I want to see it happen more often.) I'm also somewhat selfishly excited because the nature of the project pretty much guarantees a WebGL port at some point. :D

If, however, you are that kid (or adult) still dreaming about getting into the game industry at some point though, take my advice: PARTICIPATE IN THIS PROJECT! Any way you possibly can! Art, level design, programming, sound, whatever. Do it! And for the record: No. Jumping on the forums and saying "These are my ideas to make the game awesome! Now go make them happen, other people!" is not participating. If you think that counts as "game design" you've got a truckload of disappointment headed your way.

I've already heard some cynics griping that Epic is getting free labor out of this (they are), but I'm telling you now that spending some of your time and talents making Epic's game will be worth it to you if this is the field you want to go into. Not only will we hopefully end up with a great game at the end of the day, but you'll get a front row seat to the realities of working on a large game (which shares a lot with the realities of any software project). Thinking mostly about programmers, this includes but is not limited to:

Picking up on all sorts of new techniques, patterns, and concepts that you didn't know before. This one is kind of a given, but might as well point it out anyway. I've yet to encounter a code base that didn't teach me something new.
Learning to work with other people's code/workflow/style/etc. Do you feel strongly about tabs vs. spaces or where your curly braces should go? That's nice. Nobody cares. Learning how to be productive in a large codebase that you're not familiar with/disagree with is a challenge that you will face over and over again throughout your career. This a a really great, low risk way to practice that.
Learning to coordinate with other devs, probably half way across the globe. I'm pretty sure that if you code in isolation for a month straight and then drop a 10,000 line pull request in Epic's lap you'll be laughed off of GitHub. Any large project requires small, atomic changes that play nicely with the hundreds of other small atomic changes that are going in that day. Some cooperation is necessary. Cowboys need not apply.
You'll get to see parts of the development process you didn't even know existed (I hope, anyway!) Continuous integration testing? What's that? There's a lot of infrastructure around any significantly large project that (in theory) prevents the whole thing from blowing up because Jimmy from the UK office forgot that JSON can't have trailing commas. Unfortunately while this sort of rigor is necessary for basically all commercial products it can be actively harmful to the productivity of small personal projects, so you rarely get a chance to see it in action before you land in a job that requires it. I'm really hoping that Epic pushes this part of things out in front of the community, because it would be an excellent primer for new developers.
You'll discover hard truths about software/game development. I mean, really, why didn't they leave the gun that fires monkeys in the game, huh? Isn't that just the coolest idea ever! Yes, it was, until the dev team realized that the requirements of integrating MonkeyProjectile with the rest of the code base was adding ridiculous amounts of bloat and fragility to all other projectile types. (Why, exactly, does my GrenadeProjectile code need to process OnEncounterBannana again?) Decisions get made about software all the time that has pretty much nothing to do with how awesome a feature is. It's easy to gripe about from the outside, but hard to argue with when you can watch all the considerations that go into it.
You might, just possibly, end up with something resume worthy. I wouldn't recommend calling out in future job applications that you changed the default friction constant from 3 to 3.1 (Yes, I know it totally changed the feeling of the controls. Good for you.) but if you actually contribute something significant then it's worth calling out.

I'm not saying that participating in the development of Unreal Tournament will net you a job at Epic or anywhere else. If you put some effort into it, though, you'll walk away more experienced for having done so. Don't be scared that you're "not good enough", and don't be discouraged if nothing you do actually makes it into the final product. Experiment! Watch! Learn! Ask questions! (And don't be afraid to ask stupid questions! That's a skill you need to learn fast.) Make some changes and submit them. You will be asked to make changes, large or small, to your code before your contribution will be accepted. That is not a sign of failure. It's part of the normal development process. Above all, take advantage of the fact that a team of professionals is willing to build something right in front of your eyes. If you have any interest in this field at all, you owe it to yourself to at least watch carefully.

That's what I'd tell 15-year-old-me if he was here, anyway.

Oculus, Facebook, and the dreaded "Ecosystem"

2014-03-29T23:10:00.002-06:00

It's taken me several days to get over my knee-jerk, emotional, Vader-esque "NOOOOO!!!" response regarding Facebook's purchase of Oculus. I wrote up a "Please cancel my DK2 pr-order" email immediately after hearing the news, but resisted hitting the send button because I knew that anything I did at that point would be out of spite and not because I truly no longer believed in the product. Now that I've had some time to think on the matter, though, I feel like I can actually enumerate my concerns pretty clearly.

First off, let's get the bias factor out of the way: I really don't like Facebook. This is a sentiment that I seem to share with much of the internet. While I can honestly say that I admire the technical chops it takes to keep a service of that magnitude running, from a consumer point of view I don't like their product, don't trust their policies, and don't agree with their vision for the future of the internet. The only reason I use it at all is the incredibly stereotypical "because my Mom is on it" and as such it happens to be the most effective means of sending her photos and videos of her Grandson.

Oculus, on the other hand, represented pretty much everything I loved to see in a tech company. Community oriented, incredible talent, ambitious vision, and the type of product that people have been dreaming about for years made real. It also doesn't hurt that John Carmack is one of my personal heroes and someone that I've looked up to ever since I realized that I liked this whole programming thing. In almost every sense they were the classic "little guys" that you just couldn't help but root for.

As such learning that they were being acquired by Facebook was sort of like hearing that Luke Skywalker had decided to join the Empire because he felt like their resources and influence would be massively helpful in spreading the word about the Force and kickstarting that whole "Jedi revival" thing. Not to mention that it's really very hard to reconcile VR with Facebook as we know it. It's all too easy to think that Zuckerberg finished reading "Ready Player One" a few weeks ago, walked into the next board meeting, threw the book on the table and said "I want this!"

But that's the emotional side of me talking. Let's look at things more objectively:

For the sake of argument let's assume that VR does actually "work" this time around. This is something that Facebook can really only affect in a positive way. There are plenty of really stupid things they could do that would kill off Oculus' efforts, but no matter what your feelings on their products you have to admit that Facebook's leadership isn't stupid. No, if VR is going to flop it will be because the public, already wary of tech like Google glass, soundly rejects the idea of wearing these weird goggles as anything but a curiosity for super-geeks. That's a danger no matter who's name the HMD has stamped on it.

Having Facebook's cash to back up the hardware production will undoubtedly be a positive thing. The Oculus crew can now get access to better parts an reduce manufacturing costs. This comes across to us as consumers/developers as cheaper headsets, better headsets, or both! Hard to argue with that.

There's also the question of exposure. A week ago the name "Oculus Rift" was one that was only well known within the tech world. Facebook, though, is pretty mainstream. It will be a while before the concept of VR worms it's way into the day-to-day discussions of your average Joe, but having Facebook's name attached will increase visibility in ways that very few other companies could achieve (Apple, Google, and maybe Comcast are probably the only others that could compete in terms of public perception.) This is only a bad thing if you're the type of hipster that complains when your favorite unknown band becomes popular.

I know that a lot of people immediately jumped on the idea of the Rift becoming overwhelmed with ads or turning into a tool to report your every head movement back to Facebook's servers for some sort nefarious purpose. I don't think either of these scenarios are particularly realistic. Facebook will probably build a Second Life-ish social app in the not-to-distant future, and within those virtual halls all bets are off. Otherwise I seriously doubt that there's going to be an integrated Ads circuit in the device, and constantly reporting head jitter to the mothership would add too much latency to be practical. :)

(Oh, and for the record I'm also not terribly sympathetic with the Kickstarter backers who complain they didn't get a piece of the $2B purchase price. What did you think you were paying for? I donated enough to get a T-shirt. Shortly thereafter I got a T-shirt. No complaints on that front.)

In my opinion the real cause for concern is something that CliffyB hinted at it a recent blog post:

"More importantly, they needed an ecosystem. IF their system is going to be (hopefully) a dedicated system instead of a (ugh) peripheral they need their version of whatever the app store would be. Your device is only as good as the store and community around it;"

On one point he's absolutely right: Developers need their apps to be discoverable and they need ways to let users pay for them. If that comes with a community or similar perks all the better! So yes, the Rift needs an app store. If you ask me I want that app store to be Steam. I've already got a huge investment in Valve's marketplace and they've built up a lot of trust from the gaming community. Give me a tab in their store where I can browse VR-enabled titles and I'm set. But just because I want to use the Steam store doesn't mean I want it to be the only option. I want to see VR-enabled titles on Origin and the Windows Marketplace and the Apple Store and GOG and all the rest of them. And sure, there will be Titanfall-like exclusives for some stores, but that choice would be in the developer's hands.

But CliffyB didn't say "app stores." He said "store." And "dedicated system." And *shudder* "ecosystem." And this is my big, huge, overriding worry. That the decision makers at Facebook, already obsessed with their walled garden and eager to show investors the benefit of their $2 billion dollar oddity, will decree from on high that the Rift is no longer a PC peripheral but a Platform. And henceforth anything going in or out of it passes by the Facebook tollbooth first for a mere industry-standard 30% cut.

I would love to say this sounds silly. And for anyone already familiar with the state of the Rift it should seem absurd. It's feels like Logitech announcing that they're going to build a DRM-laden app store around their new mouse. But we're living in a post-iPhone world, where nobody seems too particularly worried that Apple actively bans apps which have the gall to link to pages which may possibly link to other pages where people might give other people money without giving Apple a cut. In fact, investors seem to adore this type of totally developer/consumer antagonistic behavior, and have made Apple the most valuable company in the world because of it. It's easy to see the siren's call of the closed ecosystem from the point of view of the megacorp: "I get a huge cut of everyones profits and I can control what people run on the device, banning competitors products on a whim? Sign. Me. UP!"

For the users and developers, though, there's nothing but negatives to be had here. As a developer I now not only have no option but to give up potential profits to the store curator (and possibly pay for a development license), but I'm also completely dependent on a third party that could care less about my little indie studio for promotion within the store. Assuming, that is, that they let it in at all. This takes apps about potentially sensitive subjects largely off the table, because the store owner is a huge company that's trying to (legitimately) protect it's reputation. It amounts to a high-level form of censorship. And none of that is far fetched at all because it's happening on iOS right now.

The need to pass through a gatekeeper may also discourage developer experimentation, which is incredibly negative for a new tech like this one. In order for VR to really succeed it needs it's Mario, Doom, or Minecraft moment. That killer app that people just "get". The type of thing that you call your friends over because they've just "got to try this", and having done so they then run home and figure out how they can get it themselves. I've heard a few times now the point that Minecraft couldn't have started on the Xbox because the politics surrounding how indies get a game on the console is just too complex. A tightly controlled ecosystem could have the same effect on VR.

And should Facebook create a walled VR garden and if it succeeds, there are of course sure to be competitors that crop up around it. And since it's a proven model they'll all have their own closed ecosystems too. (See: Windows 8 and to a lesser extent Android) Now the consumer gets screwed, because they started off with a FaceRift but got creeped out when the terms of services started talking about pupil scanning and so they want to get an Apple iSight instead. But, of course, this involves not only leaving behind the old hardware but all the apps you bought for it and all the account details that were associated with it and oh so many in-app purchases...

(To be clear: I don't object at all to Facebook having an app store for the Rift. I think it's pretty much a given, and would actually view it as a positive thing. I am very concerned about them having the only app store for the Rift.)

What's best for users, of course, is the peripheral that CliffyB finds so distasteful. Given some reasonable standards (even if just defacto) that the various HMDs all work with and developers can build against it no longer becomes a choice of "Which ecosystem do I commit to" but "What combination of hardware/software works for me?" I want my purchase of a FaceRift to be based on the fact that it's got a higher resolution and lower latency than the Razer Vizor, and because I don't care about that product's automatic IPD calibration. Let the choice of whether I punch my credit card into Steam for Half Life 3 or Origin for Titanfall 2 be a separate issue.

So what it really comes down to, the reason I find the acquisition upsetting, is this: The Rift will be the first product of it's kind to market. That means they're currently making up the rules that all of their future competitors will be playing by. I was okay with Oculus writing the "how to monetize VR" handbook because of their heritage. John Carmack is, after all, one of the most vocal open source advocates in the game industry. I trust that team to do what's right for users. I don't trust Facebook to do the same, and I challenge you to provide me with evidence to the contrary. And as much as I believe that the Oculus team will still fight on behalf of the users should this subject come up the fact is that they don't get to make that decision anymore. That's what happens when somebody buys you for $2 billion dollars: You do what they tell you to, even if you disagree.

For my part, I ultimately decided that I trust the Oculus team to try and do the right thing for developers enough not to cancel my DK2 order. After all, Facebook can't possibly build a wall around that garden in time for July shipments. I'm going to be watching their next steps very closely, however, and if there ever comes a point where I need to start logging in with a Facebook developer ID in order to download the SDK I'm walking away and never looking back.

Xbox One controller in Chrome on OSX

2014-02-21T11:27:00.001-07:00

I've been working lately on better Gamepad API support in Chrome (call it my "20% project"). It's been a fun change of pace from WebGL, primarily because it's much more straightforward code. In the process of learning the ropes, I decided to try something a little crazy: I noticed that our code path for supporting the 360 controller on OSX didn't rely on any drivers, but instead used IOKit to parse the data packets manually. (Check out that code here.) So I figured, hey! Why not try the same thing with the Xbox One controller, ignoring the fact that it's currently supported on exactly zero non-console platforms.

Somewhat unexpectedly, it really wasn't too hard!

Allow me to preface the rest of the post by stressing that I have absolutely no idea what I'm doing. This is entirely a result of me futzing around with APIs that I only learned about 3 days ago, and dumping a lot of hex out to the screen to see what happened when I poked a button. I'm probably doing something laughably wrong, but I could care less. This is fun! :)

The only real "trick" involved here is that the controller is basically non-responsive until you send it some magic bytes. In my current implementation I'm sending (0x05, 0x20) which I gleaned from Kyle Lemon's github repo. This lights up the "Xbox" logo on the front and the controller starts reporting sensible values. It seems like other sets of bytes work for this as well, not sure what the common thread is.

From there the buttons and axes are all reported in one tightly packed 18 byte blob, which is pretty easy to parse. It seems each data packet has a 4 byte header, the first byte of which tells you the type of data it contains. For buttons the type is 0x20, and the structure looks like this (once you strip off the header):

struct XboxOneButtonData {

// Buttons

bool sync : 1;

bool dummy1 : 1; // Always 0.

bool start : 1; // Not entirely sure what these are

bool back : 1; // called on the new controller

bool a : 1;

bool b : 1;

bool x : 1;

bool y : 1;

bool dpad_up : 1;

bool dpad_down : 1;

bool dpad_left : 1;

bool dpad_right : 1;

bool bumper_left : 1;

bool bumper_right : 1;

bool stick_left_click : 1;

bool stick_right_click : 1;

// Axes

uint16 trigger_left;

uint16 trigger_right;

int16 stick_left_x;

int16 stick_left_y;

int16 stick_right_x;

int16 stick_right_y;

};

The triggers have a range of 0-1023, and the sticks have a range of -32767 to 32768 (the full int16 range). There's a separate packet type (0x07) that gets sent for when you push the guide button, so that's easily detectable but I'm not currently monitoring it in the Chrome code. The only other packet type I've observed is some sort of heartbeat (0x03), but I'm not sure what purpose it serves.

Beyond that I haven't explored how anything else works. I don't know how to trigger the vibration motors, for example, but in this context I don't care because the gamepad API doesn't have an interface for it anyway. The one big thing I'm missing, and the main blocker before I would consider releasing this, is that you have to somehow explicitly tell the controller to shut down (turn off the LED) when you are done and I don't know how to do it. This means that the controller stays alive until you unplug the USB cable once you've visited a gamepad-enabled page. If anyone happens to know more in this regard I wouldn't mind a hint, and until then I'll poke around myself and see what I can find.

[UPDATE: After talking with some people online it seems like the controller staying on until disconnected might just be how it's designed to work. Game consoles do have different usage patterns than PCs, after all. As a result, I've finished cleaning up the code a bit and submitted the Chrome patch.]

In terms of platform support, I'm only really looking at OSX right now. I'll leave it up to Microsoft to support their own controller. (It sounds like they'll eventually make it XInput compatible later this year.) I'm also not too concerned about the Linux side of things, since once a more complete picture of the protocol is available support will inevitably make it's way into the little OS that could.

[UPDATE #2: Microsoft now has official drivers for using the Xbox One controller on Windows]

Anyway, that's been my little side project for the last couple of days. Thought it would be interesting to share with everyone!

How Blink has affected WebGL, Part 2

2014-02-17T15:28:00.001-07:00

In a previous post I detailed some of the ways that the migration to Blink has affected the WebGL pipeline. The short version is that we were able to remove some layers of abstraction without changing the code which was executed, which cleaned up our dependency diagram a bit. At the time we ended up with a data flow that looked roughly like this:

That was 8 months ago, and I've been busy in the mean time! So let's take a quick peek at the current state of affairs.

After doing the initial cleanup of the code immediately after the switch to Blink, it quickly became apparent that the GraphicsContext3D class was largely unnecessary. As I explained in the first post, it's purpose was to provide an abstract interface for all of the various WebKit ports to communicate to the graphics hardware, with each port implementing it in whatever way worked best for them. We were able to consolidate the abstract interface and Chrome's implementation in the Blink code, but this left us with a lot of function calls that effectively looked like this:

GraphicsContext3D::clear(GC3Duint flags) { m_webGraphicsContext3D->clear(flags);}

(That's not actual Blink code, but it communicates what was happening pretty clearly.)

There were a couple of other minor functions that this class provided, but the majority of it was simple passthroughs like this, and as such wasn't strictly needed. In spite of this, most of the Blink codebase was using this class as it's primary means of making hardware accelerated drawing calls. As I'm sure it would most developers, having such an obviously unnecessary layer in the code rubbed me the wrong way and so I set out to do something about it.

This ended up being a bigger task than anticipated.

You can check out the Chrome ticket where I tracked all the progress on this task to get an idea of what was involved and how many code changes were necessary to finally kill off this vestigial abstraction layer. The short version is it took a little over two months and a good deal of back and forth with the code to bang everything into shape. The end result was worth it, though, and yielded some surprising side benefits.

(Warning! Nitty-Gritty browser implementation jargon ahead!)

One of the first things that I had to contend with was that the GraphicsContext3D class (henceforth referred to as GC3D, because I'm lazy) hosted the graphics enums that were used throughout the code. Things like GC3Dint, GC3Dfloat, etc., which were all re-declarations of the equivalent GL types because WebKit didn't want to force a specific version of the OpenGL headers on implementors. In Blink however, we were fine with including the GL(ES) headers directly, and so were able to convert all of the GC3D enum types over to their standard GL equivalents throughout the code, which in my opinion made things much more readable.

I did have to resolve some conflicts with other parts of the code that included their own GL type declarations (NaCl and Skia were the big ones) in order to start including the Khronos headers, however. Specifically, I needed to make sure that 64-bit types were declared consistently throughout the code base.

After I got the entire code base using the standard GL types, the next step was to go through the code class by class and convert each to start storing and calling the WebGraphicsContext3D (Chrome's underlying 3D context implementation, henceforth WGC3D) directly. This was pretty mechanical, and didn't present too many problems.

As that conversion was being done, however, I realized that another class, the Extensions3D object, was in a very similar position of simply passing along function calls to the actual implementation (which was also the WGC3D in this case). Since it wasn't used by too many other classes we were able to eliminate it completely without too much fuss.

Next, I had to update any code that was acquiring a GC3D and make it request a WGC3D instead. This proved tricky due to differences in the memory tracking models used for each class. The GC3D class used Blink's RefPtr to track how many classes held references to it at any given time, and would automatically delete the object of that number ever reached zero. This is convenient, but it requires that the class being tracked inherits from the RefCounted class. The WGC3D class, on the other hand, is simply an interface which Chrome implements, maintaining a separation between the Blink and Chrome code. Since we can't require that Chrome provides us with a RefCounted object, I had to manage lifetime of the WGC3D objects much more explicitly. This was doable, but required some careful consideration of who "owned" the object at any given time.

Another happy side effect at this point is that I was able to get rid of a helper class called SharedGraphicsContext3D at this point, because we realized that the object tracking it provided was no longer necessary in this new GC3D-less world. Bonus!

Finally, with everyone calling WGC3D methods directly now the GC3D was left with only a few, largely static functions left dealing with generic image conversion and some extension management helpers. These I was able to break out into separate classes (WebGLImageConversion and Extensions3DUtil), and drop the GC3D class entirely! Poof! One layer of abstraction gone!

As a final bonus, while factoring out the Extensions3DUtil class I realized we were repeatedly checking to see if some extensions were enabled that Chrome guarantees to be available at all times, which made those checks unnecessary in Blink! Yay for small optimizations!

So after all that, we now have a data flow that looks like this:

When put that way it's not actually a massive change, but the difference in code clarity is significant. As with the previous refactors, the goal here was not to improve performance (although it's probably a tiny bit better as a result) but to decrease unnecessary layers of code. By removing abstractions that didn't apply to the Blink codebase we make it easier for developers to work on new features, track problems, and find relevant code snippets. That, in turn, translates into faster turnaround time on the fixes and features that you as developers want! Everyone wins!

Notch, WebGL, Dart, and ramping up quickly

2013-12-02T15:57:00.000-07:00

Just a little something I thought was worth pointing out. @notch, the creator of Minecraft, has been posting updates on his latest toy project to his Twitter account recently. I've found this very interesting because he's doing a WebGL app in Dart, which just makes me all sorts of happy. Ignoring that, however his tweets illustrate something that I feel is very important.

This was a week ago:

After an entire day of taking the very long way around, I finally got a triangle to display on the screen. My work here is done. Ship it!
— Markus Persson (@notch) November 25, 2013

Proof. Isn't that pretty? pic.twitter.com/ibS0S0eDIF
— Markus Persson (@notch) November 25, 2013

We've all been there, right? "I got SOMETHING on screen! YAY!" But then...

Starting to tame WebGL in Dart. GLSL shaders are fun. pic.twitter.com/zq6g5HAHwv
— Markus Persson (@notch) November 26, 2013

256 colors, double wide pixels, colored lights, (animated) noise, and dithering. pic.twitter.com/SWf66t8ygj
— Markus Persson (@notch) November 27, 2013

And animated: http://t.co/X5jom8RZx8
— Markus Persson (@notch) November 27, 2013

Two days later and he's gone from "Triangle on the screen" to "Blocky environment, mocked up UI, and text rendering".

Uh. Its 6:30 am, and my physics still don't work. I should go sleep. I love programming.
— Markus Persson (@notch) November 30, 2013

I think I might have developed a case of the stupids. Convex polygon/Axis aligned cylinder intersection, 8 hours, still very broken.
— Markus Persson (@notch) November 30, 2013

Phew, finally. Had to actually work out all the math and not guess. It's working now! http://t.co/iLRnkKhE79
— Markus Persson (@notch) November 30, 2013

Some hangups on the math, and now there's basic collision detection!

Billboard sprite sheet shader. 25000 sprites. pic.twitter.com/T6Dd0Ex8Z5
— Markus Persson (@notch) December 1, 2013

And sprites.

Screw this, I'm doing deferred lighting
— Markus Persson (@notch) December 2, 2013

WebGL doesn't like render targets other than RGBA888. But for low fidelity stuff that's good enough. Win! Deferred lighting is a-go!
— Markus Persson (@notch) December 2, 2013

Changed over to deferred lighting, and now I can't find the cap on how many sprites I can render. Currently two million @ 60 fps.
— Markus Persson (@notch) December 2, 2013

Fake area lights. Woot. pic.twitter.com/Ker57JVb1M
— Markus Persson (@notch) December 2, 2013

And then, hey, let's swap out the entire lighting system on a whim! Why not!

So, to recap, in the space of a week Notch went from rendering one triangle to being able to walk around a deferred lit environment. In a language that he's only just started with, no less!

Now, I'd love to claim that using Dart and WebGL has something to do with being able to iterate so quickly, but in all honesty it's going to be far more accurate to chalk that up to Notch being an experienced and skilled developer. (I do believe the web, by it's very nature, allows for fast development iteration. But that's a different story.)

No, the real reason I point all this out is not to promote any particular tool/language/API but to discourage something that I see time and time again: Over-Architecting.

I can't tell you how many time I see hobby developers saying "I'm building a game!" and what they actually have to show for it is a really complicated system for loading meshes and shaders. It's all well and good to think about long term goals and engine structure and such, but if you're going to build a game then please build a freaking game! Don't build an engine that you will someday build a game on top of, because you will never get past step one.

(And, for the record, I'm just as guilty of this as anyone!)

Now, I haven't seen one line of Notch's code. I have no idea if it's well structured or spaghetti. I would make an educated guess and say that there's a lot of the basics in this project that have been ported over from the Minecraft or 0x10c code. The point is that regardless of the coding styles at play this is a project that went from nothing to basic world interaction in a week, which is a great jumping off point for experimenting with actual gameplay.

That's awesome, and I kind of feel like that's the sort of velocity that you should strive for when you set out to "build a game." Establish a goal of some point of basic, minimal usability and do whatever it takes to hit that point first. Then iterate the crap out of it!

UPDATE:

Well, this post seems to have become fairly popular. I'm worried that people are taking away a message other than I intended, so let me clarify:

By no means am I encouraging bad coding practices, nor am I implying that Notch is trading speedy development for clean code. It's perfectly possible to do both, it just takes experience. I am trying to discourage people from building out a bunch of infrastructure that they don't actually need. This is both a personal bad habit and one that I see echoed in many forums, blogs, and tweets across the web. Worrying about making sure your shader system efficiently handles every imaginable effect before you have any meaningful game interaction is probably putting the cart before the horse. Unless, of course, you are actually trying to build a "game engine". Then by all means, design away! But at that point you shouldn't delude yourself into thinking that you'll end up with a game on the other side.

(And of course none of this applies to professional game studios. We're talking about garage developers here. The rules change entirely when you have a building full of employees and a few million in funding at your disposal.)

If you are a beginner and just want to build a game, I strongly suggest looking into Unity or something similar, as they've done all of the hard architectural stuff for you and you can focus on the game mechanics instead.

failIfMajorPerformanceCaveat: With great blacklisting power comes the need for self restraint

2013-12-02T10:56:00.000-07:00

Chrome is gaining a new WebGL context creation attribute: failIfMajorPerformanceCaveat. In the words of the spec, when you've set this useful little boolean to true:

Context creation will fail if the implementation determines that the performance of the created WebGL context would be dramatically lower than that of a native application making equivalent OpenGL calls.

Essentially, if there's any reason why the browser feels that the requested WebGL context will have significant performance overhead setting this flag allows the browser to simply not give you a context. It's a way for you as the developer to say "This context needs to perform really well, and if you can't deliver that then I'd rather not get one at all." In Chrome's case specifically, there's only one reason we will currently fail to return a context and that's if the SwiftShader software renderer is being used. (We may add more cases in the future.)

I think the kneejerk reaction many developers will have upon learning about this is: "Hey! I always want my WebGL to run fast! I'm going to set this flag on every context I ever create forever!"

If that sounds like your thought process, I give a you a stern look of disapproval: ಠ_ಠ

Look, everyone wants their content to run fast, and nobody is thrilled about the idea of your fancy WebGL app running on an ancient Windows XP laptop with little to no GPU support. I get it. But if you blindly set this flag on everything you are telling the world "If you can't run this at 60 FPS then you don't deserve to see my precious Javascript. Get off my site!" And that, my friend, makes you kind of a jerk. Even if it's going to be a slideshow, often times slow performance is better than no performance.

So why did we add the flag at all if I'm immediately going to tell you not to use it? The trick is to use it with care, considering the whole user experience. Consider these guidelines:

Use failIfMajorPerformanceCaveat when you have a fallback.

Let's say you have a 2D, sprite based game with two renderers: A WebGL one and a Canvas2D renderer. The natural assumption is that WebGL should be faster than it's Canvas2D counterpart, so you've put all sorts of cool particle effects and shader tricks in your WebGL version, but still maintain the Canvas version for all those poor WebGL-less iPad owners.

This is a perfect scenario for failIfMajorPerformanceCaveat! It may be that the fancy WebGL-based special effects mean that the Canvas2D version actually performs better when SwiftShader is in use. In that case this flag will help you fall back to the code path that provides the best user experience without unnecessarily blocking people from your content.

Use failIfMajorPerformanceCaveat when speed is an absolute must.

If you've created an awesome twitch-driven action game where a single dropped frame is the difference between success and failure then this may be the right flag for you. Also, you're very brave to be building such a game in the browser, and I absolutely want you to send me a link! I would still suggest at least trying your game with the software renderer first, though! You may be surprised, as it can sometimes outperform integrated chipsets!

If you do block users for this reason, though, please make sure they land on a page that informs them as to why they didn't get the content they were hoping for. Nobody likes a black screen and a console error.

Use failIfMajorPerformanceCaveat to work around known bugs.

Maybe you've discovered that for some reason your app simply fails to work if the software renderer is active. It's unfortunate, but it happens. In this case, please once again make sure that you tell the user what happened with a friendly landing page, and please make sure to tell us about the bug!

Don't use failIfMajorPerformanceCaveat for your latest WebGL demo.

If your WebGL app isn't an actual game and doesn't provide some sort of useful service (ie: a demo) then please don't set this flag! There's really no reason why your spinning model or shader demonstration has to run at 60 FPS. Yes, you'll get visitors that pull up your content on a phone and get a whopping 5 frames per second, but I'm willing to bet that those people's reaction is going to be more along the lines of "Holy Crap! Look what my phone can do!" and not so much "Gosh, I really wish they had blocked me from seeing this."

In essence: don't use failIfMajorPerformanceCaveat to hide content. Use it to help deliver better user experiences!

Using WebGL on Chrome (AKA: "Why don't I have the WebGLs?!?")

2013-11-03T14:08:00.000-07:00

It occurred to me the other day that although WebGL is now available on more devices with Chrome than ever before, Google actually hasn't said a whole lot about who has it, how to get it, and what to look for if you don't have it. Today I'm going to try and fix that!

The TL;DR version is that with some predictable exceptions most devices that run Chrome can get to WebGL at this point, and an increasing number of them have it turned on without any action on the part of the user. If you don't have WebGL on your device of choice it's most likely because we've explicitly identified problems with your hardware or your device hasn't implemented the appropriate safeguards, but in most cases you can still turn it on manually if you really want to.

Do I have WebGL?

First off, if you're not sure if WebGL is working on your device of choice, just visit http://get.webgl.org/. See the spinning cube? You've got WebGL! Congratulations! Nothing more to see here. Go play with ShaderToy!

Should I have WebGL?

Okay, so you didn't get a spinning cube. The next step is to determine if you should have WebGL.

First off, is your device an iPhone, iPad, or iPod touch? If so, sorry but no WebGL for you. This is due to a well documented restriction on Apple's part that only allows one browser backend on the platform. Any browser other than Safari must use Apple's WebView component, which is a more limited version of the same code Safari uses. This means that until Apple either changes that policy or starts exposing WebGL to Safari on iOS there's no (easy or legitimate) way of getting WebGL on your iDevice. Sorry.

On just about anything else, first you need to make sure that you have the right version of Chrome. Go to chrome://version/ to see what the current version of your browser is. On the desktop you need to have at least version 9 or greater, but if you are still stuck on Chrome 8 at this point I think you have far bigger issues than WebGL support! You also need to be running a supported OS, which at this point includes:

Windows Vista/ Windows 7/ Windows 8 (Windows 7 or later recommended)
Mac OS 10.6 or later (10.8 or later recommended)
Linux
Chrome OS

(That list pulled from the helpful WebGL and 3D Chrome support page) Yes. XP is not on that list. Terribly sorry, but I'll personally vouch for the fact that Windows 7 is a worthwhile upgrade!

On Android it's a little more complicated. On Chrome 31 and up, WebGL is turned on by default if your device supports the GL_EXT_robustness extension. We require this extension on mobile devices because it means that the GPU is guaranteed to be well behaved in various aspects important for security and stability. Nobody wants their phone to crash because they clicked on a link, least of all the Chrome team! As such it's important that we can trust the devices graphics drivers before we let WebGL loose on your phone.

You can check to see if your device has it by looking at chrome://gpu/. Look for the GL_EXTENSIONS row in the table, which will list all of the OpenGL (not WebGL) extensions that your devices supports. If GL_EXT_robustness is in that list then you should have WebGL support, and there's probably something else blocking it if you don't. On Chrome 25 through 30 WebGL is available as well, but you have to turn it on manually, which we'll discuss in a moment. If you have a version of Chrome less than 25 on your Android device... go upgrade! It's free!

I still don't have the WebGLs!!! Why?!?

If you don't have WebGL but you do have a supported OS and Chrome version then you can figure out why you don't have it by visiting the aforementioned chrome://gpu/. Typically the reason why we haven't enabled WebGL is listed under the "Problems Detected" section. Probably two of the most common reasons are:

You device has been blacklisted. This means that we've determined that your device either can't run WebGL correctly or can't run it safely (or both!) and have disabled it to prevent it from causing you problems. In some cases upgrading your graphics drivers may fix the issue, but many times the hardware simply isn't up to the task and never will be.
Chrome's GPU process has crashed multiple times, which can indicate serious stability issues. In this case we prevent WebGL from running for the rest of the session. Simply restarting the browser might fix it, and upgrading your graphics drivers is worth a try if the problem persists.

As a general rule we try to turn on WebGL anywhere that we know it's safe to use. If we've blacklisted a device, it's because we know that there are serious issues with your GPU or driver, issues which could pose security or stability problems. Or, you know, maybe it just flat out doesn't work. Some GPUs just aren't up to the challenge. As such, we don't recommend overriding these policies. But, if you're aware of the risks and absolutely must see WebGL running on your device...

But I really really want it!

Okay, but be aware that we're now leaving the happy safe zone of "supported configurations".

On desktop devices (including Chrome OS) visit chrome://flags/ and look for "Override software rendering list" By enabling this, you instruct Chrome to ignore the blacklist and attempt to run no matter what. (You should also make sure that "Disable WebGL" isn't enabled, as that will block WebGL whether it's supported or not.)

On Android devices the same "Override software rendering list" flag also forces WebGL on when you are using Chrome 31 or above. If you are using Chrome 25-30, though, look for an "Enable WebGL" flag. Enabling that flag is the only way to get WebGL running in those version of Chrome for Android.

I don't trust da Google. How do I WebGL with my browser of choice?

Seeing as how I'm a software engineer on the Chrome team, I have a lot more insight into Chrome's inner workings than I do the other browsers. I recommend following the links on http://get.webgl.org/, as it will generally point you in the right direction for your browser. I will say, however, that Firefox has a pretty great WebGL implementation, and Opera is now using the same rendering engine as Chrome so they inherit a lot of Chrome's WebGL goodness.

At last! Chrome D3D11 day has come!

2013-09-19T10:14:00.000-06:00

As of revision 223716, Chrome Canary will have the ability to use Direct3D 11 (via ANGLE) as the rendering backend! Woot woot! The new backend can be enabled using the "Enable D3D11" flag in about:flags. To tell which backend you are using visit about:gpu and look for the GL_RENDERER string.

That's cool, but should have no impact whatsoever on your day to day Chrome use (and if it does, we want to hear about it!) So why do I bring it up? Because along with the shiny new backend comes a much awaited WebGL feature: Multiple render targets! This extension has been available to Mac and Linux devs for a little while now (it's still behind the Draft Extensions flag), but had some spec issues that prevented it from being implemented in the ANGLE D3D9 backend. Now with the new D3D11-based backend the extension should be available to a much wider range of developers.

(Want to know if you have it? Look for "WEBGL_draw_buffers" in the supported extensions list on http://webglreport.com/)

So go, brave WebGL developers, and defer all your renderings for great justice!

[UPDATE: I initially said that the D3D11 backend may be enabled by default on some systems. Unfortunately that's not the case just yet, and I apologize for the error.]

What's coming in WebGL 2.0

2013-09-16T11:57:00.001-06:00

The WebGL working group has just released a public draft of the WebGL 2.0 spec. Hooray! Of course, being a public draft things are still subject to change and there are still plenty of TODOs, so don't be too surprised if things get chopped, tweaked, added or completely reworked before WebGL 2.0 reaches a browser near you.

The nice thing is, however, that since the entire goal of the spec is to bring OpenGL ES 3.0 capabilities to the browser the chances of things deviating too much from that spec are pretty minimal, and it's probably safe to assume that most ES 3.0 features will make their way into your browser soon. (With a few notable exceptions that I'll discuss in a bit.)

So what is new in 2.0? I realize that reading specs is rarely fun, so allow me to summarize for you. While this won't be a complete rundown of the upcoming features it should provide a good sense of where things are headed.

(Note that links to the OpenGL Wiki may refer to desktop OpenGL features that are not available in ES 3.0.)

[UPDATE: Apparently the words "This won't be a complete rundown" weren't explicit enough, so allow me to point out that the following are not the only features in WebGL 2.0! Array Textures, Depth Textures, NPOT textures, Floating point textures, New compression formats, and all the rest of your favorite ES 3.0 features are coming unless we find serious compatibility issues with them. Don't freak out because something isn't explicitly listed below. (Unless it's listed in the "Not gonna do it" section, then feel free to complain to the mailing list.]

Promoted Extensions

Some of the new features are already available today as extensions, but will be part of the core spec in WebGL 2.0. The big benefit to being part of the core being, of course, that support is guaranteed. No longer will you have to code fallbacks for Multiple Render Targets! If your device supports WebGL 2.0 then you have these features, period!

Multiple Render Targets

Currently exposed through the WEBGL_draw_buffers extension. Allows a single draw call to write out to multiple targets (textures or renderbuffers) with a single draw call. This is "the big one" in a lot of developers eyes because it makes many of the modern deferred rendering techniques that have become such a core part of modern realtime 3D practical for WebGL.

Instancing

Currently exposed through the ANGLE_instanced_arrays extension. Instancing allows you to render multiple copies of a mesh with a single draw call. This is a great performance booster for certain types of geometry, especially when rendering highly repetitive elements like grass.

Vertex Array Objects

Currently exposed through the OES_vertex_array_object extension. VAOs allow you to store all of the vertex/index binding information for a set of buffers in a single, easy to manage object. This is very similar to the IDirect3DVertexDeclaration9/ID3D11InputLayout interfaces in DirectX land.

Fragment Depth

Currently exposed through the EXT_frag_depth extension. Lets you manipulate the depth of a fragment from the fragment shader. This can be expensive because it forces the GPU to bypass a lot of it's normal fragment discard behavior, but can also allow for some interesting effects that would be difficult to accomplish without having incredibly high poly geometry.

New Features

These features are brand new to WebGL, and bring us to near-feature-parity with OpenGL ES 3.0.

Multisampled Renderbuffers
Previously if you wanted your WebGL content to be antialiased you would either have to render it to the default backbuffer or perform your own post-process AA (like FXAA) on content you've rendered to a texture. That's not a terribly happy situation if you're concerned about the quality of your rendered output. With this new functionality you can render to a multisampled renderbuffer and then blit that multisampled content to a texture. It's not as simple or fast as rendering directly to a texture, but it's still a lot better than not having multisampling at all for non-default framebuffers.

3D Textures
This feature is pretty self-explanatory. A 3D texture is essentially just a stack of 2D textures that can be sampled with X, Y, and Z coords in the shader. This is useful for visualizing volumetric data (like medical scans), 3D effects like smoke, storing lookup tables, etc.

Sampler Objects
Currently when you create a texture it contains two distinct types of data. First is the image data: the array of bytes that makes up the actual pixel data at each mipmap level. Then there's the sampling information, which tells the GPU how to read that image data. This is things like filtering and wrap modes. It's often useful to separate those two concepts, however. For example: Today if you want to read from the same texture twice in one shader, one instance with linear filtering and the other with nearest filtering, your only option is to duplicate the texture! Less critical but still annoying: currently you have to set the sampling information on every single texture you create, even if they'll all use the same settings.

Sampler objects allow you to store all the information about how to sample data from a texture separately from the texture itself, which becomes nothing but a container for the image data. During your draw loop you can pair textures and samplers as needed. It's a simple thing, but surprisingly useful!

Uniform Buffer Objects
Setting shader program uniforms is a huge part of almost any WebGL/OpenGL draw loop. This can make your draw calls fairly chatty as they make hundreds or thousands of gl.uniform____ calls. Uniform Buffer Objects attempts to streamline this process by allowing you to store blocks of uniforms in buffers stored on the GPU (like vertex/index buffers). This can make switching between sets of uniforms faster, and can allow more uniform data to be stored. Additionally, uniform buffers can be bound to multiple programs at the same time, so it's possible to update global data (like projection or view matrices) once and all programs that use them will automatically see the changed values.

This functionality is reasonably close to Direct3D 11 Constant Buffers.

Sync Objects
With WebGL today the path from Javascript to GPU to Screen fairly opaque to developers. You dispatch draw commands and at some undefined point in the future the results (probably) show up on the screen. Sync objects allow the developer to gain a little more insight into when the GPU has completed it's work. Using gl.fenceSync, you can place a marker at some point in the GPU command stream and then later call gl.clientWaitSync to pause Javascript execution until the GPU has completed all commands up to the fence. Obviously blocking execution isn't desirable for applications that want to render fast, but this can be very beneficial for getting accurate benchmarks. It may also possibly be used in the future for synchronizing between workers.

Query Objects
Query objects give developers another, more explicit way to peek at the inner workings of the GPU. A query wraps a set of GL commands for the GPU to asynchronously report some sort of statistic about. For example, occlusion queries are done this way: Performing a gl.ANY_SAMPLES_PASSED query around a set of draw calls will let you detect if any of the geometry passed the depth test. If not you know the object wasn't visible and may choose not to draw that geometry in future frames until something happens (object moved, camera moved, etc) that indicates the geometry might have become visible again.

It should be noted that these querys are asynchronous, which means a queries results may not be ready for many frames after the query was originally issued! This makes them tricky to use, but it can be worth it in the right circumstances.

Transform Feedback
Transform feedback allows you to run geometry through the vertex shader and write the resulting vertices into a buffer. That buffer could then be used to re-submit the draw calls without going through the full vertex transforms again. This could be used to capture the positions of a GPU-driven particle system, or write out the results of mesh which was skinned on the GPU.

What didn't make the cut

The nature of browsers and Javascript means that WebGL can't simply be a dumb wrapper over the OpenGL C interface. Most of the time this doesn't pose a big problem, and can actually yield nicer interfaces in some cases (like gl.texImage2D accepting img tags), but in some cases the realities of the environment simply don't allow for some functionality to be exposed in a reasonable manner.

Please understand that nobody likes dropping features, but these decisions have been well thought out (and in some cases fought over.) That said, if you feel very strongly that the wrong decision was made now, while the spec is in draft, is the time to voice your opinion!

Mapped Buffers

Javascript VMs simply cannot, for many reasons, expose a raw unchecked chunk of memory. This means that any attempt to expose a mapped buffer interface would involve WebGL allocating a managed typed array, copying the desired data into it, and then going through some extreme contortions to detect changes and copy them back to the actual mapped memory. This would completely destroy any of the performance benefits of using this API, and since the whole point of the API is to achieve better performance it would be disingenuous at best to try and expose something that looked like mapped buffers but didn't actually act like it.

In place of the mapping functions, however, WebGL 2.0 will provide a getBufferSubData function to allow you to query back portions of a buffer. It's not a complete replacement, but it will help cover at least one of the use cases.

Program Binaries
This is a feature that sounds really useful when you first hear about it, but the shine wears off quickly when you learn about the realities of how it works. The fact is that even if we exposed this to Javascript is would be extremely difficult for developers to use in a meaningful fashion.

Instead WebGL implementations can attempt to cache program binaries after they are first built and will re-use that cache for subsequent runs. This is behavior that is already in place, and will continue to function with WebGL 2.0.

drawRangeElements
Similar to mapped buffers, there doesn't appear to be a good way to expose this function and deliver the performance benefits that are assumed to come with it's use. (The WebGL implementation would have to re-validate all it's indices to ensure that they all fall within the range specified.) As such drawRangeElements it probably won't make it into the final spec, but we'd appreciate external feedback on how widely used/necessary this function is in ES 3.0 or desktop GL content.

What does this mean for you?

For the time being, not much. This is a draft spec, not an announcement of implementation. Full implementations of the spec won't be appearing in browsers for a little while, and even when they do there will be a period where it will likely be hidden behind a flag. As such it's going to be a little while before the average user will be able to see your web content.

As I mentioned earlier, however, some of the more exciting pieces of functionality are available right now as WebGL 1.0 extensions. If you'd like to start prepping your content for WebGL 2.0 feel free to start playing with WEBGL_draw_buffers and ANGLE_instanced_arrays now and you can feel confident that those features will "just work" when WebGL 2.0 lands. As for the rest of WebGL, nothing is being removed or deprecated and WebGL 2.0 is fully backwards compatible. This means that all it should take to "upgrade" your existing content is to add a "version: 2" the context attributes. That won't magically yield any performance or quality gains, of course, but it's nice to know that the code you are writing today won't be obsolete a year from now.

Holistic WebGL

2013-08-16T15:51:00.001-06:00

ho·lis·tic

Adjective

Characterized by comprehension of the parts of something as intimately interconnected and explicable only by reference to the whole.

Characterized by the treatment of the whole person, taking into account mental and social factors, rather than just the physical...

Definition from Google Search

As I've established way back on the very first post on this site, WebGL is awesome. So awesome that one might be tempted to start seeing opportunities everywhere around the web to throw some 3D goodness at it. After all: when your hammer is a 3D API, every problem looks like a vertex array... er, or something like that. Now never let it be said that I discouraged the use of WebGL, but as developers we should also be aware that every technology has a time and place.

What is holistic WebGL? It's application of the technology in a way that takes in to account the entire web experience, not just the content of your canvas tag. Holistic WebGL content is considerate about it's context and respectful of the user's time, attention, and battery life. In essence, it's the exact opposite of this:

Now I'm guessing that very few visitors to this site were actively working on "Punch the Polygonal Monkey" banner ads, but it's not hard to envision such a thing. After all, we all know that "eye catching" is an advertiser's friend. What's a poor ad designer to do now that their beloved Flash is being slowly smothered? Why, hellllooooo there HTML5! What lovely WebGL you have! And now we have pages where a deluge of banner ads all rendering beautifully shaded cars, watches, and pharmaceuticals makes your battery commit suicide, your fans squeal, and your page chug.

But wait! It doesn't have to be like that! WebGL can be used in a way that's both exciting and engaging while simultaneously being non-obtrusive and (crucially) non-rage-inducing. Much of it involves simply being aware of what place your WebGL content in the page as a whole.

"Full Page" WebGL games and apps.

The easiest scenario to envision is a WebGL game or WebGL-centric app (like the new Google Maps). This will most likely be the primary content of the page, and will probably be very large or even full-screen. Most other page content will be secondary to it. This is going to be the most permissive scenario, and allows developers to focus on high performance, detailed graphics with the assumption that the user is continuously interacting with it. Given this environment it's easy to think that you have the whole GPU to yourself. But wait! Even in this best-case scenario you need to take some basic steps to ensure that your application recognizes the realities of the web.

The user can switch tabs at any time!

Have you actually seen how people use web browsers? Honestly it's kind of horrifying. It's not unusual for people to have 100+ tabs open across several different browser windows in a session that's been running continuously since sometime in the late 90s. Of course, it's the browsers job to deal with that kind of load, not yours, but it does demonstrate a point: Unlike your standard desktop games where the app went fullscreen and that's pretty much all your PC did till the user quit when you're talking about content in the browser you must expect that your page may be shoved into the background at any moment, no matter how engaging you make it. Planning for this and handling it gracefully is an absolute necessity for any WebGL application that wants to be seen as more than a curiosity.

Fortunately the browser provides several mechanisms for coping with this. First and foremost if you're using requestAnimationFrame (and if you're not then you are doing it wrong!) then the browser can automatically scale back how frequently your render routine is called when your tab is not in the foreground. This behavior is "free" to you and simply a benefit of using the right API for the job.

However, we can do better! If the user isn't looking at your page why animate it at all? For this we have the Page Visibility API. This API will notify your page when the user can't see it anymore. This may be for any number of reasons: The user switched tabs, minimized the window, changed desktops, maybe even covered up the window with another window. The exact scenarios are up to the browser, but in general all you should care about is "The user can't see me anymore."

The most user/battery friendly thing to do when you get a visibilityChanged notification is to completely pause your app. Stop animating, halt your game logic, kill any sounds, and generally shut down any other aspects of your app that continuously update. In some games that may not be an option (multiplayer or other timing sensitive games), but in these scenarios you should still stop any visuals and sounds until the user comes back.

If you do pause a game, though, resist the temptation to resume the moment you are notified that the page is visible again. Show your "pause" screen instead, let the user take a moment to reorient themselves, and allow them to choose when to resume. It's far less startling and frustrating that way. Keep in mind that your user may be returning to you many days after they left! Don't expect them to jump right back in without warning.

Nice context you have there, shame if something were to happen to it...

The other big part of respecting the user's browsing environment is to properly Handle Context Loss. As I mentioned a moment ago, the user may navigate away from your page and not come back for a good long time. What are they doing during that time? Using other WebGL apps? Installing new graphics drivers? Playing desktop games? Changing monitor configurations? Maliciously attempting to crash your GPU? Who knows! The point is that there are any number of reasons why the browser may decide that it needs to kill your WebGL context, sometimes even while your page is in the foreground! When this happens it's up to you as the application developer to:

Indicate to the browser "Yes, I can recover from this" by calling event.preventDefault() in the webglcontextlost handler.
When the context is finally restored, reload all of the necessary GPU resources.

For the record, step 2 sucks. It's something that you have to design your entire app around: Keeping track of all the resources being used at any time and being able to rebuild/reload them at a moment's notice. But if you don't, the alternative isn't great. In Chrome, we'll throw the following delightful message up in front of your user, and your WebGL canvas will turn into a black box.

Don't want rats in your HTML? Handle context loss!

WebGL content as an appetizer, not the entrée.

What about the banner ads that we were talking about earlier, though? Or, more specifically how should we handle WebGL content that is somehow secondary to the rest of the page? Beyond ads there are a lot of really cool uses for WebGL content as part of a larger page whole. Here's some great examples:

In these cases WebGL is used not as the centerpiece of the page but as a way to visualize content in a way that static images and 2D rendering simply fail to capture. These sites are unquestionably better off for having WebGL content embedded in them, but they do so in a way that's not antagonistic to the user. So what are the guidelines that should be followed for this kind of content? For one, everything listed above still applies. You should still be sensitive to page visibility and context loss, and for the love of battery life you should be using requestAnimationFrame! But now there's some additional things to take into consideration. (Conversely, many of the following tips can apply to full page apps as well, but they are especially important in mixed-content pages.)

Light on the shaders, please! (And meshes, and textures, and...)

Let's be blunt: If your WebGL content is not going to be the focus of the page it's in your best interest and the users to scale it back a bit. Yes, we know you can render normal mapped, subsurface scattered skin with realistic sweat and veins, but did you really need to load 5 megs of textures for a mesh that takes up 120x240 on the screen? Make sure your content compliments the page, and accomplishes it's goal as minimally as is reasonable. Faster loads make for happier users. (Unless, of course, the entire point of your content was to render realistic skin. Then have at it! WebGL would be awesome in science textbooks!)

Prefer user interaction to starting automatically.

As a general rule, you should let the user tell you when they want to see your content, rather than revving it up the moment you get your page load notification. There's a couple of reasons for this:

Cuts down the annoyance factor considerably. You may have the coolest animation in the world, but if the user has to endure it every time they visit your page it's going to get old fast. This is only exaggerated if there are multiple blinking, swirling, bouncing elements on the page.
Reduces page load time. You can cache your resources on load to make sure your element is ready to go when the user asks for it, but that should only happen when the rest of the critical content is ready. Users will be very forgiving if your 3D widget loads 2-3 seconds after the rest of the page. They will hunt you down with pitchforks if the article text is delayed because you just HAD to show them that spinning cube first.
Battery life! I keep mentioning this, but it's super critical! Especially considering that WebGL is starting to be enabled by default on some tablets and phones. If your page starts cranking up the GPU the moment users land there they will want to spend as little time on your page as possible, and possibly avoid it all together.

Keep in mind that there are ways that you can do this without requiring an explicit "play" button, which can make your 3D elements feel more natural and interactive. Let's say that you create a cloth simulation that interacts with the user's mouse. (Okay, that demo's not WebGL, but work with me...) It would be perfectly reasonable to start the simulation at rest, and render a single static frame instead of continuously animating. Then, when the user's mouse passes over the canvas you can kick the animation loop on and let the cloth react to their movements. (Of course it would be preferable to switch back to a static frame again if when the simulation comes to a rest.) This is great because it provides a fun, discoverable surprise for the user by letting parts of your page spring to life while still remaining respectful of their overall experience.

This isn't a hard and fast rule, of course. Two of the pages I linked earlier had animations that started immediately on page load. It's important to realize the context of these effects though. In both cases they are the only animated element on the page (in view, anyway), and the animation is subtle and not too distracting. Less is definitely more in this case. It's also worth mentioning that in the case of the Acko.net page the rest of the interactive elements wait for the user to interact with them before they start showing off their goodies.

Give the user a way to say "I'm done."

Imagine if YouTube didn't have a pause/stop button? Videos just looped forever as long as the page was up, playing Gangam Style in the background while you browsed other clips, made comments, etc. The collective Internet Rage pointed in their direction would be enough to telepathically melt their servers.

Having some way to stop an animated element for animating is a great way to help your users feel in control. If you don't provide that, you have implicitly told the user that if they want the distracting, battery sucking element to stop dancing around they can always leave your page. If driving users away is your goal then great! Otherwise, give them some mechanism to stop your content.

Case in point: Florian Boesch's page, Codeflow.org, has a subtle little "wandering balls" effect in the header. It's simple, not terribly distracting, and probably not going to strain anyone's CPU. It also has an "On/Off" button next to it. If that simple little effect is worthy of a pause button, so is yours.

Don't draw what hasn't changed.

This is pretty simple: If your content isn't moving, stop drawing! Take Google Maps, for example. Interaction is smooth and beautiful, but when the map isn't moving the image doesn't change. It would be crazy for the maps to sit there and render at 60Hz just to draw the same static image over and over again, and would guarantee that your laptop would overheat any time you left the page open. Obviously unacceptable from a major app like Google Maps, but equally offensive if it's some random simple 3D widget sitting in a corner that only moves on mouse down.

Don't flood the page with Canvases (Canvasi?)

ShaderToy pushes the limits of what's acceptable here, but it works in their case because the shader gallary is the focus of the site. (Note that they also take pains to follow some of the tips listed here, like not animating the thumbnails unless your mouse is hovering over them.)

Not only is moderation in this area important for a good user experience, but it's also something that's currently strictly enforced by the browsers. Both Firefox and Chrome have static limits on the number of WebGL contexts you can have on a page at once, Chrome's limit being 16. That sounds harsh, and eventually we would like to move to a more flexible, resource-centric limit, but we can't simply allow web pages to spin up an unlimited number of GL contexts. At some point you'll run out of the resources to do so, and probably crash your tab in the process. In any case, the limit helps prevent the worst abuses, but as a developer you can be more intelligent about how to allocate your pixels.

A favorite trick of mine is to have a single canvas that is detached and reattached to the page as needed, replacing static images with live content when requested. This can simplify resource management too, since everything is shared by a one context.

Be kind to users that don't have/don't want WebGL.

WebGL is becoming more ubiquitous, but there's still plenty of browsers (especially on mobile) that don't have access to it. Try to be helpful to these users!

By no means am I suggesting that you should built 2D Canvas fallbacks for everything, but anything beyond showing a blank black rect on the screen is appreciated. In many cases a static filler image hinting at what content the user could be seeing with WebGL support is sufficient. It's also a great idea to provide a link to resources like get.webgl.org that can help users figure out how to get WebGL on their platform. If you do have WebGL controls on your page that are critical to it's function, however, you better be ready to provide fallbacks! (I have yet to see WebGL-based submit buttons, but mark my words: it will happen. Weep for your page accessibility.)

Most WebGL content I've seen does a pretty good job of this already, but it's still worth having on this list.

It's your content! Make it shine!

Everything that's been covered here is great in general terms, but the truth of the matter is that there's no single "Right Way" to make web content of any form, 3D or otherwise, user friendly. A lot of these tips are aimed at games, 3D apps, small embedded visualizations, or ad-like content. But like any good technology WebGL has unlimited uses, and your application of it may completely invalidate some or all of the above. It's up to you to ensure that your content respects your users, but there's a lot of motivation for you to do so! It's safe to say that any content that's put online is put there in hopes of finding an audience of some sort. Content that users find annoying, distracting, or that negatively impacts their performance is not content that people want to share.

Holistic WebGL == Happy Users == More eyes on your awesome content. (Isn't it great when we can all win?)

WebGL instancing with ANGLE_instanced_arrays

2013-07-03T16:27:00.000-06:00

And still no Shakespeare...

Ever find yourself in a position where you absolutely, positively have to get a ton of monkeys on screen with a single draw call?

Yeah, okay, me neither. (At least, not until I put together the demo for this post.)

But there's certainly occasions when it makes a lot of sense to redraw the same mesh many times with some minor tweak like position, orientation, color, etc. Simple examples of this are trees and other vegetation, streetlights, boxes in a warehouse, soldiers in an army, and so on.

Traditionally with WebGL the optimal way to draw those repeated meshes, commonly referred to as instances, would be something like the following pseudocode:

bindMeshArrays(gl);
for (i = 0; i < meshInstances.length; i++) {
  var instance = meshInstances[i];
  gl.bindUniform3fv(meshPosition, instance.position);
  gl.bindUniform4fv(meshColor, instance.color);
  gl.drawElements(gl.TRIANGLES, indexCount, gl.UNSIGNED_SHORT, 0);
}

This is good, because we only set up the vertex attributes (implied to be in bindMeshArrays) once, then make many draw calls in quick succession, only changing the properties that are different from instance to instance. In this case just position and color. This makes for a near minimal amount of overhead for each draw call, and in an environment like Javascript where each call is expensive that's important!

(This example is pretty simple, and in fact you can actually optimize it further by getting creative with how you provide the GPU with the data. Check out Gregg Tavares' wonderful Google I/O talk for more ideas about how to render lots of geometry very quickly.)

Even in this scenario, however, you have to make at least three WebGL calls for every instance of the mesh. That doesn't sound terrible but in the real world that would probably be quite a bit more, and the fact is that in Javascript every call into native code (Like the WebGL API) carries a certain expense. Thus while drawing repeated meshes in this way works, and generally works well, it could still be better.

And that's exactly why we have ANGLE_instanced_arrays.

[Update: Peter Jacobs on Google+ brought up a potential point of confusion. Despite the name, this extension works on any device with the appropriate hardware support (ARB_instanced_arrays and ARB_draw_instanced), not just on Windows when using ANGLE. The ANGLE in the name simply indicates that the extension spec was written by the ANGLE authors.]

This extension, which is currently available in the Chrome Dev channel behind the draft extensions flag, allows you to pack information about each mesh instance into an attribute array, just like you do vertex position, normal, etc. But instead of listing these values once per vertex, you only have to specify them once per instance. Lets take a look at how it works:

// A nice little line of monkeys down the X axis
var offsets = new Float32Array([
  0.0, 0.0, 0.0,
  1.0, 0.0, 0.0,
  2.0, 0.0, 0.0,
  3.0, 0.0, 0.0
]);

var colors = new Float32Array([
  1.0, 0.0, 0.0, 1.0, // Red monkey
  0.0, 1.0, 0.0, 1.0, // Green monkey
  0.0, 0.0, 1.0, 1.0, // Blue monkey
  1.0, 1.0, 1.0, 1.0, // White monkey
]);

var offsetBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, offsetBuffer);
gl.bufferData(gl.ARRAY_BUFFER, offsets, gl.STATIC_DRAW);

var colorBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, colorBuffer);
gl.bufferData(gl.ARRAY_BUFFER, colors, gl.STATIC_DRAW);

This should look pretty familiar! In fact, there's no difference between this code and your standard Vertex buffer setup except that the information is intended to be used once per instance. The real difference come in when you go to draw with that data.

var instanceCount = 4;
var ext = gl.getExtension("ANGLE_instanced_arrays"); // Vendor prefixes may apply!

// Bind the rest of the vertex attributes normally
bindMeshArrays(gl);

// Bind the instance position data
gl.bindBuffer(gl.ARRAY_BUFFER, offsetBuffer);
gl.enableVertexAttribArray(offsetLocation);
gl.vertexAttribPointer(offsetLocation, 3, gl.FLOAT, false, 12, 0);
ext.vertexAttribDivisorANGLE(offsetLocation, 1); // This makes it instanced!

// Bind the instance color data
gl.bindBuffer(gl.ARRAY_BUFFER, colorBuffer);
gl.enableVertexAttribArray(colorLocation);
gl.vertexAttribPointer(colorLocation, 4, gl.FLOAT, false, 16, 0);
ext.vertexAttribDivisorANGLE(colorLocation, 1); // This makes it instanced!

// Draw the instanced meshes
ext.drawElementsInstancedANGLE(gl.TRIANGLES, indexCount, gl.UNSIGNED_SHORT, 0, instanceCount);

This is actually doing the same thing as our first code snippet with the for loop, but as you can see there's only one draw call here! The shader would have to be updated, but essentially you would only change the position and color inputs from a uniform to an attribute. The important bits in this code are the calls to vertexAttribDivisorANGLE and drawElementsInstancedANGLE.

vertexAttribDivisorANGLE specifies for the given attribute location how often a value should be repeated. A divisor of 0 means the attribute isn't instanced. A divisor of one means that each value in the attribute stream is used for a single instance. A divisor of two would mean that each value is used for two consecutive instances. For example:

ext.vertexAttribDivisorANGLE(offsetLocation, 1);
ext.vertexAttribDivisorANGLE(colorLocation, 2);

Using the data above, this would render two red monkeys at X=0 and X=1, and two green monkeys at X=2 and X=3.

ext.vertexAttribDivisorANGLE(offsetLocation, 2);
ext.vertexAttribDivisorANGLE(colorLocation, 1);

This, on the other hand, would render a green and red monkey at X=0 and a blue and white monkey at X=1.

(To be honest, I don't see much real-world use for divisors other than 0 and 1, but it's nice to know what they do.)

drawElementsInstancedANGLE (and drawArraysInstancedANGLE) are straightforward replacements for their non-instanced counterparts drawElements and drawArrays, the only difference being that each takes the number of instances to render as an additional argument.

Pretty simple, right? When used correctly this can lead to better performance and less code, both noble goals.

And now for a quick confession: The demo I created is actually a pretty terrible example of this API. On most systems I've tested it on using hardware instancing (that is, using the extension) either yields the same performance as the software instancing (using a for loop) or only provides 2-3 extra FPS. That's because this scene isn't really draw-call bound, and so instancing doesn't have much opportunity to speed it up.

As with anything in graphics, you shouldn't just blindly throw this API at everything you can but instead measure what the bottlenecks actually are. If you are fill-rate bound or vertex-bound then instancing won't solve your problems. But if your seeing your app get hung up on making lots of draw calls while your GPU spins it's wheels waiting for Javascript to catch up, this is the API for you!

No more Google+ comments

2013-06-14T22:16:00.001-06:00

I tried out the new Blogger Google+ comments for a while, but after experiencing some weird behavior (like comments not displaying on mobile) and hearing from a couple of people that avoided leaving comments because they didn't want a Google+ account I've decided to turn them off. This may cause a couple of comments to disappear, and I apologize for the inconvenience, but I'd rather have a reliable commenting system.

A Tale of two Web Technologies

2013-06-14T14:46:00.000-06:00

Let's play a quick game: I'm going to describe a relatively recently announced web technology, and I want you to guess what it is.

It was developed by a generally well respected company that focuses on moving the web forward.
It was started to enable the web to move beyond the inherent limitations of javascript.
It allows developers to run applications programmed in a language other than javascript.
The company behind it encouraged others to join in its development, and has actively been improving it themselves.
It was designed so that when run in a specialized VM it could run much faster than an equivalent Javascript app.
To preserve backwards compatibility, however, the code is cross-compiled into standard javascript, allowing it to run on most modern browsers.

So if I stop there, what's your guess? If you've been keeping up with these things I bet you're probably thinking asm.js, which is a good guess because it meets all the criteria up above. Let me add one more hint, though:

When announced this technology was regarded with skepticism and vocal opposition.

Ah, that makes a difference. What's your guess now?

"What is Dart, Alex?"

Bingo! We have a winner!

[Update: I originally had an item in the list implying that asm.js had been developed behind closed doors. I've been corrected on this point, as the asm.js project was open source on Github since its inception. It would be more correct to say that it was not publicized until they had enough complete to accurately benchmark. Thanks to the folks on Twitter that pointed this out to me, as it paints a more complete picture of the reasons behind public attitudes!]

Look, I know that this is not an Apples to Apples comparison. But if we're being honest both of these technologies are motivated by similar goals: Javascript is too slow and awkward for the demands of really high performance applications. (Like, say, Unreal Engine 3?) I know that that statement alone will annoy the Javascript faithful, who come out swinging Atwood's law around like a club. I actually like Javascript, and would be willing to argue the same in most cases. But there's a certain class of applications that it simply was never designed to handle. Technologies like NaCl, Dart, and (most recently) asm.js are all attempts to fix that.

So, getting to the point of this post: I'm a confused as to why something like Dart, which I have found to be a great language to code in, inspires either annoyance or apathy from many web developers while the same crowd seems largely enthusiastic about asm.js.

For my part I think they're both awesome technologies with different use cases. If I were building a new web app from scratch I'd want to code it in Javascript or Dart but would prefer Dart due to its cleaner syntax, built-in libraries, and potential speed. (Not to mention it supports things like SIMD which is awesome for realtime 3D!) If I had an existing iPhone or Android game that I wanted to bring to the web asm.js (or NaCl, which the caveat of browser support) provides a great tool for expanding my audience with the minimal amount of effort. Likewise for non-trivial, pre-existing C/C++ libraries. Different tools for different purposes, both useful in their own right and both trying to let us do more than Javascript allows for today.

So I'd like to drum up some discussion on this one: What's your opinion on these (or any other similar) technologies? Why do you like them or dislike them? From some of the feedback I've already heard it seems like there's may be some confusion about what these various technologies actually do, and if so I'd like to do my part to clear that up. I'm also trying to get a bit of the community pulse to help inform the direction of some future posts.

(Oh, and I promise that my next post will be far more interesting to WebGL developers! Got something really cool on the way!)

WebGL Draft Extensions in Chrome

2013-06-05T09:44:00.000-06:00

I mentioned this was coming in my last post, but it's a significant enough change that it's worth addressing on it's own. New Chrome builds, starting with today's Canary build, will begin including a new option in about:flags called "Enable WebGL draft extensions"

(This can also be activated via the command line with the --enable-webgl-draft-extensions flag)

When you enable this it will allow WebGL to access extensions that Chrome has implemented but are still in draft status on the WebGL Extension Registry. Previously these extensions would have been exposed without the flag, but with a "WEBKIT_" prefix. That goes against Blink's policy of avoiding vendor prefixes wherever possible, however, hence the new flag.

The only extension this flag exposes currently is the newly implemented EXT_frag_depth, and any extensions that you previously queried using a prefix will continue to work just fine.

So what does this mean for you, daring WebGL developers?

The biggest change is that you probably won't want to release WebGL apps publicly that rely on a draft extension. The flag is there to let you develop against upcoming features, so that they'll be ready to go the moment they become publicly available, but 99.9% of your sites visitors won't see the extension till it's been approved by the community. You can always pop up a dialog telling users how to go flip the flag, but that's not a terribly good first impression for your app to have.

That sounds like a bad thing, but despite the inconvenience it's actually beneficial long term. For starters, features have historically been hidden behind a prefix/flag when they're not finalized. This means that in some cases things may change drastically between the initial vendor-prefixed version and the final product. (See: CSS gradients). The problem is that prefixed features are still visible to users without any effort on their part, so sites begin to rely on the experimental syntax/behavior. This means that even when everyone collectively decides that the initial implementation was a bad idea the browser must continue to support it forevermore or risk breaking sites and angering both developers and users. So the browser bloats over time and it becomes harder to add new features because you're constantly tiptoeing around crusty old features that nobody likes anymore.

It also means we're less likely to break your live app. If you're using an extension that's in flux on a live site and we change the behavior (it happens, especially when it comes to WebGL security issues) suddenly your site has broken and you get to go digging through browser commit logs to figure out what changed. Joy. WebGL extensions are far less likely to undergo sweeping changes once they've been moved out of draft status, though. By encouraging you not to rely on them till they've solidified we make life easier on everyone.

Of course, this implies some responsibility on the part of the WebGL implementors as well. We need to approve extensions in a timely manner to prevent them from being stuck behind a flag in perpetuity. That's actually something that you as a developer can help with! If you are using a flagged extension please provide us with feedback on the public WebGL mailing list! We want to hear about any implementation bugs you find or problems with the extension specs, but we're also happy to hear about your uses of the extensions as well. If we receive a lot of feedback that a draft extension is being widely used or eagerly anticipated it's far more likely for that extension to be moved forward quickly.

How Blink has affected WebGL?

2013-05-29T17:40:00.000-06:00

One of the topics that was suggested when I recently took a poll on Twitter/G+ of potential blogging topics was what kind of impact the switch to Blink has had on Chrome's WebGL implementation. I thought this would be a great area to talk about, because it allows me to dig into the guts of how WebGL works a bit more than most of you are used to.

If you're not familiar with the situation already, Chrome recently switched rendering engines from WebKit to Blink, which is based off the WebKit source. The fact that we're so early in the life of Blink means that the two rendering engines haven't diverged too much yet, aside from dead-code cleanup on both ends, but even this early there are a few changes in how Chrome handles WebGL.

Before we get into that, though, lets talk about what exactly happens when you make a gl.whatever call in your javascript. If you were to ask most people, even many WebGL developers, how the browser handles WebGL commands they would probably give you a fuzzy description of a model that looks like this:

This is based on a couple of commonly understood facts: For security purposes WebGL commands must be validated/secured to ensure that malicious web pages can't crash the users graphics driver or OS. Secondly, on Windows ANGLE is often used to translate GL commands into DirectX commands to take advantage of what have traditionally been more stable and better optimized DirectX drivers.

That's all well and good, but there's an awful lot that could go wrong here. Graphics drivers, especially on older systems, can be pretty unreliable. The fact that they are hooked so deeply into the OS means that there's a lot of opportunity for bad behavior. Since the browser wants to protect the user from both unexpected driver behavior and malicious code that is attempting to provoke bad behavior from the driver Chrome sandboxes all driver interaction in a locked down GPU process. This allows us to carefully control what commands we send to the GPU, and should something go wrong a crash in the GPU process won't take down the entire browser.

So updating our model to take this into account, we get a diagram that looks a bit more like this:

So... that's pretty close to the real thing. Right?

The reality is, as usual, quite a bit more complex. Lets take a look at a really quick-and-dirty diagram of the actual classes/files involved in processing a WebGL call in Chrome's original WebKit backend:

Just a wee bit more complex than the first chart, yes? While not completely technically accurate, this shows the basic flow of a command through the first half of the WebGL pipeline. Yes, that's right. This is only half of it. I'm omitting the entire GPU process, which is just as complex as this diagram. I'm also skipping over the implementation of resource objects like textures and buffers, all of which add their own classes to the party.

So why so many layers of code? Isn't this a bit excessive?

Well, since WebKit is used by many different browsers, not just Chrome, a certain level of abstraction and separation is required for any interface like this. There's a lot of code that's handled by the WebKit core, but anything platform or browser specific is deferred to port-specific implementation code (denoted here by the Chromium logo). It's necessary to achieve the "runs on anything" flexibility that WebKit enjoys, but creates complexity.

Let's run through a quick overview of what each of these steps do:

V8WebGLRenderingContext is automatically generated from the WebGL interface IDL. It marshals data to and from the Javascript VM (V8) into more user-friendly formats, which are then passed to...

WebGLRenderingContext. This is where most of the WebGL-specific logic happens. Anywhere that The WebGL spec has many additional restrictions above and beyond the OpenGL ES 2.0 spec, and those validations are all done here. Once the data is squeaky clean and WebGL-approved, we hand it off to...

GraphicsContext3D. The browser uses OpenGL for more than just WebGL. A lot of work goes into making the browser utilize your GPU as effectively as possible on every page to make scrolling smoother, rendering faster, and interactions snappier. We don't want all of the browsers internal code to be subject to the same restrictions as WebGL, however. This class exposes an API very close to OpenGL ES 2.0 to any portions of the WebKit codebase that need to perform GPU work (with occasional detours through Extensions3D to handle, well, extensions.) But since WebKit must work on multiple platforms, this class is fairly abstract. It has different backends on different platforms, but in Chrome's case it is implemented by...

GraphicsContext3DChromium/GraphicsContext3DPrivate. These two classes form Chrome's implementation of WebKit's GPU interface. On some platforms this might be where the actual GPU calls are made, but in Chrome's case we want to take advantage of the sandboxed GPU process. As such, this class primarily feeds GL commands directly to...

WebGraphicsContext3D. This class facilitates communication with the GPU process via a "Command Buffer" that packs the GL commands into safe little memory packets which are pushed across process boundries via shared memory. Actually, that's a lie. This class doesn't do that directly, but do you really want the chart to be more complicated? It's accurate enough for our needs.

Most of the non-WebKit Chrome code will use this class directly instead of the GraphicsContext3D since there's no need for the abstraction once you're in Chrome specific code.

Finally, everything ends up in the GPU Process, which is quite complex by itself but which I won't be covering in-depth here. After unpacking the GL commands from shared memory it does another round of validation to ensure that it's been fed valid instructions (it's the GPU process' policy to not trust external processes that communicate with it.) Once it's satisfied that everything is safe, it finally makes the actual driver call, which in some cases actually becomes an ANGLE call to redirect through DirectX instead.

Whew! Bet you never realized how much your WebGL calls got tossed around before hitting your graphics card! But all of that is how Chrome used to work when it was based on WebKit. Coming around to the original question: How has Blink changed this process?

Let's look at the same data flow in Blink as it stands today:

Still complex, but noticeably more straightforward. Specifically what has happened here is that we've taken the places that WebKit required to be split up for the sake of abstraction and merged them into a single class. The code from GraphicsContext3DChromium and GraphicsContext3DPrivate are now part of GraphicsContext3D. Likewise for the Extensions3DChromium, and as a result a whole layer of abstraction melts away.

Something to take note of: the actual code that gets executed is actually almost identical between the Blink and WebKit versions. The only thing that has really changed is that we've merged what used to be port-specific code into the base classes and taken the time to remove dead code paths that Chrome never would have hit anyway. This means less files to juggle and less abstractions to work around for the Chrome developers, but will probably yield very little in terms of performance (We've eliminated a couple of virtual function calls, but that's about it.)

In the end changes like this are about speeding up development, and will have almost no impact on end users, but that's very much in line with Blink's goals of code simplification and quicker iteration. Eventually we may find optimization opportunities that result from this refactoring, but for now it's all about cleanup.

Other than implementation simplification, though, I would expect the switch to Blink to have very little effect on the evolution of WebGL. Apple wasn't blocking WebGL features or anything like that, and pretty much all of the WebGL implementors are on the same page when it comes to what should and shouldn't go into the API. We're already pretty happy with the direction WebGL is going, so don't expect any drastic changes to be pushed because of Blink.

There is one user-facing, Blink-specific change that's on it's way, though: Keeping with Blink's policy regarding CSS and Javascript prefixes, as we expose new extensions we won't be prefixing them anymore (so no more "WEBKIT_" in front of extension names). Instead, we will hide extensions that have not yet been approved by the community (known as "draft" extensions) behind a flag. This will allow developers to continue experimenting with upcoming extensions without the worry of production sites becoming dependent on them, which creates problems if the spec changes before the extension is approved.

So there you have it: Blink has allowed us to simplify WebGL's implementation a bit, and it's prefix policy will change how developers try out new extensions, but otherwise it will continue to be the WebGL you know and love.

Hope you enjoyed the anatomy lesson!

[EDIT: Want more? Check out Part 2]

How Microsoft could possibly, just maybe, become WebGL's biggest supporter

2013-05-22T12:10:00.002-06:00

Ready to jump head-first into wild speculation land? Come on, it'll be fun!

It occurred to me today that there's a slim possibility that in the near future Microsoft may actually become the biggest driver of WebGL adoption in the industry. Yes, that's right, "WebGL considered harmful" Microsoft. Sounds crazy (probably is crazy) but I feel the possibility is definitely non-zero. Why? Let's string together a series of massive assumptions, shall we?

[XBOX LAUNCH UPDATE: So now the XBox One is actually out, and we can say with certainty that WebGL is not in it. You can look at Microsoft's official guide to IE on the XBox One to see what is included, but it's pretty explicitly stated that it launched with IE10, not IE11, and WebGL isn't mentioned anywhere. That's not to say, however, that it may not be included in a future update.

Interestingly enough, however it turns out that the PS4 actually supports WebGL to some degree! It's not accessible to the browser, unfortunately, but several parts of the UI, including the entire PS4 Store, are driven by WebGL! Who knows if Sony will ever make WebGL available outside their UI, but it's awesome that it's such a core part of the experience!]

First off, we know that someone at Microsoft has been toying with WebGL in IE11. It's actually working in the leaked builds of Windows 8.1 (or "Blue") with a little bit of registry hacking. That by no means guarantees that it will make it into the final release, as there are any number of very legitimate reasons why it may not be fit for release when the time comes. Personally I'm hoping they do release with WebGL support but I won't be surprised if it doesn't make it into the final product. However, for the sake of this post lets assume that when IE11 lands it brings a bundle of WebGL goodness along with it.

[UPDATE: So apparently right as I was posting this Microsoft started teasing WebGL support in IE on their Vine account. Looks like this part, at least, is looking a whole lot more plausible!]

Windows 8.1, and along with it IE11, will be released "later this year, in time for the holiday season" according to Microsoft reps. That's interesting because there's also another big Microsoft property that's getting an update "Later this year": The Xbox.

The Xbox One, as revealed in a TV and Sports-heavy event yesterday, will be running "Three Operating Systems", one of which is built on the Windows kernel. It hasn't been explicitly stated, but one would imagine that they would be basing it on at least the Windows 8 kernel, and probably something that resembles 8.1. I would expect it to be modified to meet the needs of the hardware, but I also don't think it will stray too far from the kernel we see on the desktop. After all, the Xbox One (and PS4, for that matter) are architecturally just beefy x64 PCs.

We also know that Xbox One will include a version of Internet Explorer. Again, Microsoft hasn't confirmed what version of IE will be include, but once again it's not too much of a stretch to think that it should at least be IE10 and might actually be IE11 since the release windows seem to roughly correspond. If I were to place my bets I'd say it's something that falls in between. IE10 customized for the console with pieces of IE11 pulled in during development. It probably won't be a massive change from the desktop version in any case because, once again, the console is essentially a PC. Even if the Xbox One doesn't launch with IE11 though, I think it's a reasonable assumption that the embedded browser will see updates over the life of the console.

So... to recap our speculation thus far: The Xbox One might launch with the Windows 8.1 kernel which may included IE11 which could possibly support WebGL. Thus, there is a teeny tiny chance that The Xbox One could support WebGL, either at launch or via some future update.

Still with me?

Now, even if the Xbox One did launch with IE11, and even if IE11 on the desktop did support WebGL there's a lot of reasons why it simply might not be available on the console. I would expect that any WebGL implementation that Microsoft would release would be a wrapper on top of DirectX, similar to how Chrome and Firefox use ANGLE on Windows. But even though the Xbox 360 (and presumably Xbox One) use something very DirectX-like, it's not directly compatible. Thus a WebGL implementation for the console would probably require a custom implementation, which may be more technical debt than Microsoft wants to take on. There may be security concerns unique to that hardware, there may be OS-level restrictions that prevent the necessary GPU access, there's probably political forces that would push back against it... There's a whole slew of scenarios that I can imagine that would keep WebGL off of that box.

But on the off chance that it actually did make it in, WebGL would suddenly gain a massive new audience, and developers would have a lot of motivation to create WebGL content. After all, now there's a gateway onto Microsoft's device and into peoples living room that sidesteps the process and politics of traditional console development. Assuming gamepad and fullscreen support (which I think is pretty much a given if WebGL is there) players could jump into a game with a simple bookmark and immediately be in a console-like experience with very little friction. Targeting just Chrome and Firefox may be a hard sell, but targeting the Xbox, while simultaneously capturing desktop players too? That's hard to resist.

WebGL on the Xbox also has the opportunity to be the fastest implementation of WebGL around. In Chrome there's large chunks of code that do nothing but work around platform and driver issues. There are loads of code paths that are in place to ensure that you can't get "bad" data back, that everything is initialized to a known good state, and to make sure that we don't have to trust the driver to provide security for us. In a well-known environment like a console, however, the implementation could be far more streamlined.

That all sounds awesome, but why would Microsoft allow it? After all, providing a way to put games on the console that haven't gone through Microsoft's licensing and/or store is depriving Microsoft of potential profits, right? Certainly Microsoft might see it that way, and that alone could make this whole idea a non-starter, but I feel like the fringe benefits to Microsoft just might outweigh the downsides in this case.

First and foremost, there's the inescapable fact that a WebGL game, even on a super-optimized browser, will never match up to the performance of platform-specific native code. Nobody is going to be building Battlefield 4 in the browser anytime soon. Not to mention WebGL games will have to fetch their resources from the web instead of a speedy Blu-Ray drive. They won't be able to earn achievements or access your friends list or use the other fancy social tools the native games will. They won't be promoted on the home screen, they won't have TV tie-ins from Stephen Spielberg... put bluntly you'll have to go through Microsoft if you want to give players that "premium" experience. WebGL on the Xbox would always be a second class citizen.

But that's okay. Look at the massive number of Flash and HTML5 games online today. Many of these are simple, silly, proof-of-concept type games that would never have been appropriate for full-scale console development in the first place. Microsoft would never have made money off of these kind of games in the first place, but it does benefit them to provide access to these games through their device as it further cements the console as the "Entertainment hub" of the house. Furthermore, rather than IE being a perpetual afterthought for online game developers (as it is today) suddenly IE11+ becomes the primary platform many web games target, because it represents a massive audience on well-known hardware that are already on a gaming-centric device. It also means that games will be more likely to target Microsoft controller, even on the desktop, and it provides one more potential feature to set it apart from Sony's console.

Is that enough to convince Microsoft it's worth it? No idea! It makes for an intriguing thought experiment, though. We'll find out by the end of the year, and I really wouldn't recommend getting your hopes up in the meantime. For my part, though, I'm going to stay naively hopeful until proven wrong. :)

Welcome to the Next Gen™ (Same as the Last Gen, but better lit)

2013-03-27T11:34:00.001-06:00

So footage for the next installment in the "Battlefield" series was posted today, and it's got a lot of people excited. For good reason too, it's gorgeous! Totally worth your time to watch the full 17min preview:

Much respect goes out to the engine developers and artists, those are some pretty pixels!

Too bad the gameplay doesn't look like it got a similar overhaul. This is basically the same game we've been playing since Call of Duty: Modern Warfare in a new coat of paint.

For starters the enemy AI looks pretty braindead. They can't seem to aim, don't notice people running around 3 feet behind them, tend to stand in open "scope-approved" places, and don't really move around much. Yawn.

But that's okay, because the real point of playing a game like this is to be shuffled from one tightly scripted, minimally interactive set piece to the next. Right? Of course at some point that stops becoming a game, and that's what's got me a little down.

It's very interesting that this preview starts out showing us the ending, then bounces back in time to show how we got there. Flashbacks can be a very effective literary technique, but in the context of a game it carries some unwelcome subtext: "Nothing you do over the next hour matters, because you will always end up right here." And why not? After all, the animators spent a lot of time on that scene, the artists custom built an "upside down car underwater set" just for the occasion, the script writers agonized about the precise placement of all those swear words to convey the appropriate panic with a "mature" feel. There was a lot of blood, sweat, and tears (and money) that went into that 5 minutes of barely interactive cutscene, it would be a tragedy if the players missed it!

And as a result no matter what you do you're going to end up upside down in a car underwater. You'll never be able to outrun that helicopter, you'll never be able to save Soldier 3's leg, never be able to successfully defend the pickup point, and never spare your helicopter support her untimely end. What you can do (the "interactive" part) is fail to live long enough to witness yourself failing to do all these things. Did someone order a side of fatalism with their escapist hobby?

Maybe it's just me, but I was hoping that the next generation of game consoles would let us get away from these kinds of on-rails experiences. And yes, maybe that would mean that some game designers would have to give up on their secret Hollywood aspirations, but I'd have a lot more fun with the end result. I'm not saying games can't have a well-written narrative, but I'd really like to play a more active role in shaping it. By all means, have the helicopter pilot radio in and say "Extraction point is that building, go!" And then let me figure out how to get there. Sure, throw the enemy chopper at me right before I escape. But rather than forcing me into some pre-scripted "hanging off the building" scene, let me ACTUALLY defend the building! Wouldn't be awesome to be ducking around that shell of a structure, desperately trying to get pot-shots off, while the helicopter systematically rips the building to shreds around you? And if I or my teammates die in the process, so be it. It will be far more meaningful to me if I knew I could have saved them, rather than knowing that they're doomed from the start.

And hey, go ahead and keep that whole underwater car thing in there! There's a decent chance I'll be forced off the road anyway while swerving to avoid the hellfire raining down above me. It would be extra cool to know that the developers planned for that and didn't just throw up a "game over" screen when the car hit the water. But let me actually escape sometimes too!

Emergent gameplay is a tricky thing, because by necessity it's not going to be nearly as consistently intense as a carefully curated experience (though our increasing technological capabilities allow it to be more interesting). The times where it all comes together just right, though, are the moments that we as gamers remember forever. Those are the moments that we post to YouTube and rave about in forums. Those are the moments we share with friends the next day with giddy excitement. Rather than "Hey, have you reached the helicopter scene yet?" "Yeah! That was awesome!", which is a great reaction in it's own right, it becomes something personal, but with just enough shared context that everyone can relate:

"Hey, so you know that helicopter scene? So get this, I'm trying to fight it back but right before I fire my RPG the building supports get shot out and the whole floor starts to topple over! My shot goes wide, and I'm trying not to fall while desks and boxes and stuff are sliding past me. Then the chopper swings back around and I think I'm toast, but the RPG that I fired earlier hit that smoke stack off to the side, and it fell down right on top of him right as he began to fire! I had 8 health left! It was incredible! And so then I..."