Meeting minutes
Acknowledgements (20 minutes)
Alex+Brandon: Discussion on how to collect acknowledgements and what counts
brandon: Every contribution counts, scribing is ++ count
bajones_: google form worked before
… but any form that allows for automated collection/contributor driven collection
bajones_: a github bot would be perfect
ada: I can build it
ada: what details do we need?
bajones_: we've taken emails before but maybe any contact will work
bajones_: we at least want company
ada: @bajones could you add a html comment ican hook into and i can make a bot that will turn acknowledgement issues into PRs
bajones_: how should we distribute the call for acknowlegemenrts/
… last time was on the mainling list but maybe not wide enough, how about a comment in the bs?
… it maybe too noisy if we do that.
ada: if we get too much spam we can just take it out.
… we can probs move onto the next topic
Break (10 minutes)
taking a break since we had to skip one issue and now are running fast
<cabanier> ada: are we starting again at 9?
<bajones_> That's what I thought
<bajones_> Slides for lighting estimation topic: https://
<bajones_> (Also linked in the agenda)
Lighting Estimation (@toji) (30 minutes)
RafaelCintron: what are the numbers that are returned?
… and what colorspaces are they in?
bajones_: for arcode, every color is 16 bits
… I still need to investigate more
… the floats are bounded from 0 to n, not 1
… it seems that there are assumptios
… it seems that there is a mapping to opengl and vulcan
… it seems it doesn't map cleanly to srgb space
… the raw blob of numbers shouldn't be surfaced up to the web
… it will come back as a texture
RafaelCintron: if it's just an array of number, what is the height/width
bajones_: it's a 16x16x16 cubemap
… so it's not a problem internally
… it's to contrast with arkit which hands you a texture
… so either metal has to do the conversion or you have to extract them yourself
… we'd rather have the texture so systems that use that aren't disadvantaged
RafaelCintron: what if you call it, what is the lifetime of the texture
bajones_: I think it needs to be a new one every time
… especially if mipmapping needs to happen
… so if the developers changes it, we don't want to leak thing
… which means we have to return a new one
… since it only happens once a second and in response to an event, it's not a big deal
… authors shouldn't pull it in every frame
… they should listen to the event and then do the right think
RafaelCintron: wrt the light probe, and a long time later you passed it in, are you getting information from the past?
bajones_: no, the light probe should update over time
… arcore only has 1 light source
… and it's attached to the viewer and basically gives the light conditions around your phone
… in arkit, you can place a lot of spaces and they are tracked over time
… you can say which probes should be updated so they are tracked when the user moves
… which is something you can do on iOS
RafaelCintron: so when you call it repeatadly, you get the same information?
bajones_: yes. Until recently, you would even get the same object back
… if you called it. Now that I added options it's no longer the case
… there is a repo for this and it has an open PR
… Manish has given the thumbs up but I'd like to get the opinion of more people
ada: it would be great if we can get someone from webkit
… has dino been involved?
bajones_: at the Seattle F2F, I asked Apple about that.
… and after that, the documentation suddenly got better
ada: if there are no other questions, should we move onto the next topic?
Layers + Dom Overlay Hotspots in Head Mounted & Dom Overlay
<klausw> Slides: https://
<atsushi> (will drop shortly around 3:30am, go to sleep)
<DaveHill> I think full interactivity is a use-case we care about in the VR flavor of this.
alexturn_: screenspace and headmounted dom overlays have different user requirements.
cabanier: how are these annotations done, through a script or the UA?
klausw: a script
… how this will work in stereo presents many issues
bajones_: we could provide an option to facelock the UI
alexturn_: what would stop developers opting in by default
bajones_: we need to make it a little unpleasant so people know it should not be the defualt path
alexturn_: whats the security implication behind 2 dom layers?
klausw: issues regarding user surprise as they may not tell what is dom vs webgl
alexturn_: proposing an extensio nto support world positioned dom overlays
klausw: Folks focusing on phones won't use an API if it's more complicated
klausw: For non-interactive quads, why not just handdraw it yourself?
klausw: If you had a head-attached quad, it would not be easy to interact with
klausw: 1. Do we have what we need to make non-world-attached DOM overlay work well
klausw: 2. For world-attached content, what do we need for apps that make screen-space assumptions?
<Zakim> ada, you wanted to ask about facelock plane distance and to respond to klausw
ada: For detached elements, if you had an interface in your 2D browser where the buttons are slightly popped out in the Z direction, and that interface is carried with the user into the environment, the user would expect the buttons to retain their 3D aspect
bajones_: If you do DOM overlays across multiple overlays, we could perhaps deal with the privacy impact, though there is still a concern
bajones_: We need a more dynamic API than the current "set it and forget it" fullscreen API for one overlay
bajones_: You'd need to move them around
alexturn: people have to do maths to place stuff where they want it
alexturn: for the dots people have to work it out themselves
… if we can provide a hook for them to calculate it we would get people onto the right path
alexturn: for the headlocked version, we might need to think about the privacy issues
… but I am not sure I can see what they are the risks seem the same to me
cabanier: If you render the DOM overlay on top, how do you render the controllers?
cabanier: For detached elements, there's something you could do there, but it would be hard to specify how to do that
ada: Rolled a 9 to pick a scribe
<Zakim> DaveHill, you wanted to why not just draw it yourself on a quad
<DaveHill> does the group have a way of thinking about phone-only vs. all-device features
<DaveHill> do we have principles around this?
<DaveHill> I feel like the gist of Brandon's argument is "people are going to do this anyways"
alexturn: As a headset vendor, principle is all device features. Failure if by virtue of using a feature you've built a phone only app
… Except for transient input, which you wouldn't get on a headset there isn't too many things (inline excepted)
DaveHill: This is the first feature that feels hard phone only oriented
… Feels like we're going down a path of adding phone only features, would we apply that to headset only features?
Ada: In that situation, developers want to use the feature in VR because you get a significantly better result
… HTML is valuable for historical reasons
DaveHill: The DOM overlays feels very phone oriented
… Not pushing on the use of 2D HTML, the issue is that it doesn't feel as though it translates well to headsets without 3D translations we're unsure of
AlexTurn: I'd push back on anything about "having UI" being phone oriented
… Rolling your own UI is a pit of failure for devs, so need a path to have it work universally
… Need to ensure we use our power for good and dissuade developers for writing head locked content that is not universal
Klausw: The DOM overlay API is a compromise for how to do DOM in XR. Not perfect for all usecases. Solves some important usecases
… Clear intention to allow simple features like a start button / model selector to work on the headset without the app dev needing to do anything specific to support headsets
… "How can we make this not completely break on headsets and what should developers do instead"
… Questionable whether a more complex API would be attractive to developers to use
… Fundamental differences in UX between phone and HMD modalities
… Hard to have an API which for solves for both
… There are expectations, e.g. no input events for cross origin content
… Will need UI hints to indicate that
Rik: If you hit the DOM overlay your controller disappears?
Klausw: Spec language about the host content not having access to XR input data, so will need UI hints e.g. reticle
AlexTurn: DOM overlays are doing a good job as a honeypot to prevent devs from falling into bad patterns like screenspace
… is there incentive to use further API vs just doing screenspace ?
… Overylays you showed were independent in the world. We could do a lot of work for devs to simplify the API
… For input routing, similar to system UI (controller / hand is captured by system and not routed to apps)
… For DOM overlay we have the same problem
… UA would have to route input away from Apps based on where it is pointing and what content it is aiming at
… have to solve this problem anyway to be ready for multiple overlays
<bajones_> Boo! I want the DOM to stretch around me as an equirect!
<bajones_> ;)
Ada: Suggests layer would load about:blank and then content would be added
AlexTurner: Would want to load this way with transparent content initially
… Start from some kind of new tab page until switchover can be seamless
… Sibling property to set the background color might be meaningful
Rik: There was a metatag proposal to set whether the window is transparent
AlexTurner: If transparent and took a few frames to load things it'd be fine. If it's white then takes a few frames it would be noticable
Rik: Transparent windows are allowed so can spec it without breaking any assumptions
Klausw: Like it, more flexibility but restrictions on the DOM content to make it work. Not allowing 3rd party iframes seems like a good tradeoff. How do nonflat HTML elements work? (e.g. select)
… Might there need to be restrictions?
Rik: Don't they all get rendered to the same surface?
Klausw: No. e.g. on Android its a different UI.
Rik: Delegate render to a different surface?
Klausw: On Chrome it is platform specific
AlexTurn: Edge may run in different modes. Separate mode which constrained itself to the window border for e.g. windows phone. This mode was used for HoloLens
… May have solved the relevant issues there
… Whatever solution for e.g. comboboxes may be transferable to this domain
Klausw: If you want to impl DOM overlay, you need a full feature browser engine because there is no restriction on DOM
… With restrictions, would not need the full engine
… Another issue is system UI, e.g. does the DOM cause virtual keyboards to appear?
Rik: If you pop up a window, your controller poses go away and you use the virtual keyboard with system controllers.
… Users would see the window popping up and see different controllers appear
bajones_: What determines pixel size (css?)
… What is the semantics of width and height. What resolution does the page render at?
Rik: App needs to determine meters to CSS pixel mapping
bajones_: Not sure there is a significant downside to mapping to meteres directly
… matching the aspect ratio of the in-world element seems reasonable
… Don't want to leave it to UA's as the incentive as resolutions get higher, they'll want to scale everything up
… Proposal as is prevents you from using DOM from the 2D page itself.
… How do you determine whether I'm interacting with DOM vs World?
Rik: Hit-test from pointer to DOM content
Klausw: Could capture onselect with DOM elements if they contain listeners for it
… Could allow event cancelling
AlexTurn: If you hover away we end up taking the pose away, to prevent double rendering from multiple layers
… Layers not always on top
… which means app content may obscure it.
… Interesting negotiation between app and system around it
bajones_: We could ensure that any time you're interacting with the DOM the system MUST render a cursor
… Expect app to render its own controller in its own way, but don't allow the app pretending to point their ray somewhere its not actually pointing to trick the user into clicking DOM
… Should have the option to NOT capture the input (e.g. tooltip)
… waste of resources and unwanted
Raphael: WebGL wanted an extension to draw DOM to WebGL textures
… Similar problems. CORS limitations
… Youtube would not work in that case (or gmail or office)
… +1 to manditory cursor suggestion
AlexTurn: Paused based on initial scenarios being CORS related (e.g. loading youtube) which halted discussions previously
… Good place to start with just constructing a UI using DOM
… If this API existed and DOM overlays didn't, developers on phones would just make 1 quad and fill the screenspace rect
… Now they've unwound headset compat
… Here you could do both explicity, but we've lost the incentive for phone devs to fall into the right pattern
… orders of magnitude more phones to expect content to come from there which jeopardizes future compat with hmds
… Is there a role for a medium / high policy version closer to DOM overlays in addition to this?
… Game engines want means to render 2D UI
Manishearth: Have concerns on architecture
… If creating new window objects not from iframes, browser developers won't like it
… iframes are already weird in terms of implementation and spec
… window objects that have rendering behavior implies a lot of things that need to be explicitly spec'd
… Wondering if there is another way to go about this
… not allowing cross origin simplifies things but still have to worry about rendering capabilities of the device
… moving windows around requires complex rendering behavior, which is cause for alarm. Need to discuss this.
… Should consult low-level iframe / window infrastructure people and also the TAG
… Is potentially tangled with other spec's, not an XR specific thing
<Zakim> ada, you wanted to mention CSS centimeters and meters and to ask about equirect domlayer
Ada: CSS already has a concept of units
… e.g. meters
<bajones_> CSS also says 1in == 96px. Which... 😒
Ada: Other topic is equirect layers. Not sure what would happen there
bajones_: cylindrical layer, wrap around 360 deg. Context in a tube around you, no need to deal with poles
… Don't want to encourage combing coconuts
<ada> q/
Rik: a css cm is not always a cm, its in pixels and depends on px / cm
bajones_: css px are not always device px
Rik: cross-origin usecase could be made to never receive input, and just be a dead texture
… in that case you'd not get access to window contents, only the layer metadata
<Zakim> klausw, you wanted to say about scalability - hundreds of annotations?
<bajones_> I was concerned about this too, Kalus
klausw: People are going to use scalability in odd ways, which may make browsers unhappy. Need a gl style min / max limit for implementations
… e.g. min = 4
Rik: Nothing preventing layers being huge. No min / max. Only limit is supporting 1 projection layer
… One of the issues with windows, new process / memory
<DaveHill> +1 to having some kind of minimum limit
bajones_: There exist reasonable limits for the base layers API
… Shouldn't allow terrapixel large quad layers
… pixel content being determined by pixel size in meters may not be a good idea
… Better to push our content and make it huge then put it close to your faces
<klausw> We just need an angular resolution CSS measurement
bajones_: Implies IMAX res screens. Nobody wants to render their web content that large
<DaveHill> pixel density feels like the right way to handle this -- UA can determine CSS to physical pixel scaling factor
bajones_: thus caps on size + control from the user is warranted
Rik: Assumed transform would cover that but scale requires more
… Would want to scale the normal way (maintain aspect ratio in all cases)
bajones_: Maintain aspect ratio and reflow content, even if more expensive