Event system shouldn't assume a single, builtin Raycaster

[AndrewPrifer]

Edit: for future discussion see #2133, where these ideas are being refined and implemented.

The title is a little vague so let me explain.

How the current event system works is:

• the renderer adds all objects with interactions to store.interaction
• on an event, objects in store.interaction are raycast using the builtin Raycaster and default camera
• the system iterates over the hits according to the rules of event propagation and calls event callbacks

This system works correctly for most of the use cases, however it fails if certain objects need to use a different Raycaster, for example to render an overlay scene for a gizmo. For example, in this Drei example, if we were to add pointer event handlers to the main mesh calling e.stopPropagation(), the gizmo would stop being interactive when it's covering the geometry, since the event system assumes that it is using the same Raycaster and according to the hit, it is behind the mesh. Even if we don't call e.stopPropagation() on the mesh, we would have the issue that we wouldn't be able to stop clicks from propagating to the main scene when the gizmo is clicked.

The above use case is common enough that Drei provides a useCamera helper that helps use a Raycaster set from the perspective of a camera different from the default one. However, useCamera is a massive hack, because it works by overriding the raycast method of meshes with one that ignores the passed Raycaster and instead uses the Raycaster created by the useCamera hook from the passed camera. Meaning r3f's event system has no knowledge of this and will fully assume that these meshes are also raycast with the internal Raycaster, which results in the above described incorrect behavior.

Proposal

• Extend the Instance declaration with the optional eventLayer property that takes a { raycaster: Raycaster, priority: number } object.
• Group objects added to store.interaction by raycaster.
• When an event occurs, we iterate over the raycaster groups using their own raycaster and doing the regular event handling for each group successively. Calling e.stopPropagation also stops the event from propagating to lower priority raycaster groups.

Objects without an eventLayer prop would be grouped to the builtin default raycaster using the default camera.

This solution also has the advantage that current projects using useCamera could be adapted with minimal effort: useCamera would still create its own raycaster, however instead of returning a hacked raycast method, it'd return the eventLayer object that can be passed to meshes.

---

What do you think? I'm curious to hear any corrections, extensions or counter-proposals! I'm sure I left some things out and some parts might be unclear, please ask any questions you might have.

[drcmda]

also quite unhappy with the current implementation tbh. while useCamera hacks things like HUD's it won't fix gl.skizzor or (sub-)events on planes that portal content.

would you want to implement a draft? pls target v8 branch though.

[AndrewPrifer]

Sure thing, just wanted to get opinions on this first :)

[gsimone]

In light of Andrew's PR, I think it might make sense to talk about adding a Scene primitive already, I think we have been gravitating towards this for a while.

const MyApp = () => {

   return <>
     <Scene camera={{}} renderTarget={...}>
          <mesh>...</mesh>
          <OrbitControls />
         <color args={['black']} attach="background" />
     </Scene>
  </>
}

the Scene object would:

• creating the scene and portal
• override the useThree state with the new scene/camera object
• take care of rendering and clearing ( either rendering to a target or to the main buffer )
• take care of event layers ( by overriding something? )

some examples:

GUI:

<MyModel />

<Scene autoClear={false} orthographic camera={{ zoom: 100 }}>
   <Text>Hello GUI</Text>
</Scene>

FBO stuff:

const fbo = useFBO()

<Scene renderTarget={fbo} > ... </Scene>

<mesh> 
 ...
 <meshBasicMaterial map={fbo.texture /> 
</mesh>

This could also be made to work with the invalidate API (invalidate per scene or global, for example).

Now the default Canvas could be expressed as Canvas (webgl context) + Scene (scene graph concerns) ( probably transparent to the user? )

Pros:

• optional, you can still do everything manually
• simple GUI story
• simple render-to-texture story
• simpler multiple-scenes story

Cons:

• possibly complicates the lib code since we have to add the any-state-override thing for useThree
• potentially the additional component exported from r3f, although we could build this as a drei component - as in, just a suggestion of a possible use of the API

Notes

• Scene might not be the best name, rather something like "Layer" or "View" or somesuch, because "Scene" means something very specific in gamedev (and also quite possible outside of it) (And also because it doesn't have a 1:1 relation to THREE.Scene) ( verbatim @hmans )

[AndrewPrifer]

Re: @gsimone's Scene component, yesterday I brought up exactly this to Paul which resulted in this beautiful but hacky user land prototype, thought I'd paste it here so it doesn't get lost. This is a generalized solution that lets you hijack the whole r3f context.

I do agree that ideally we'd also have an abstraction for the render target, which this solution doesn't provide.

import { ReactNode, useMemo } from "react";
import { RootState, useStore, context } from "@react-three/fiber";

function resolveState(
  state:
    | Partial<RootState>
    | ((state: Partial<RootState>) => Partial<RootState>),
  current: RootState
) {
  return typeof state === "function" ? state(current) : state;
}

interface InjectProps {
  children: ReactNode;
  state: Partial<RootState>;
}

export const Inject = ({ children, state }: InjectProps) => {
  const useOriginalStore = useStore();
  const injected = useMemo(() => {
    const useInjected = Object.assign(
      (sel: any) => {
        // Fetch fresh state
        const current = useOriginalStore.getState();
        // Execute the useStore hook with the selector once, to maintain reactivity, result doesn't matter
        // @ts-ignore
        useOriginalStore(sel);
        // Inject data
        const altered = { ...current, ...resolveState(state, current) };
        // And return the result, either selected or raw
        return sel ? sel(altered) : altered;
      },
      {
        getState: () => {
          const current = useOriginalStore.getState();
          return { ...current, ...resolveState(state, current) };
        },
        subscribe: (callback: any) => {
          return useOriginalStore.subscribe((current) =>
            callback({ ...current, ...resolveState(state, current) })
          );
        },
      }
    ) as typeof useOriginalStore;
    return useInjected;
  }, [useOriginalStore, state]);
  // Wrap the hijacked store into a new provider using r3f's default context
  return <context.Provider value={injected}>{children}</context.Provider>;
};

[AndrewPrifer]

Another thing we talked about is maybe probably tying the event layer creation to createPortal, since we couldn't think of an event layer use case that didn't involve portals.

[AndrewPrifer]

Btw I'd like to direct everyone to the event layer PR (#2133) for any discussion strictly about event layers, since this issue wasn't updated with newer ideas incorporated in that PR.

[looeee]

Extend the Instance declaration with the optional eventLayer property that takes a { raycaster: Raycaster, priority: number } object.

This would be great - it would allow me to remove some hacky stuff in my own code (didn't know about useCamera though, will look into that).

Does this require two THREE.Raycaster instances? Or can it be done with one? If you need to raycast from more than one camera in a single frame then presumably you'll need to raycast twice. But if you just need event layers then raycasting once should be enough I think?

BTW here's an example that does weird stuff with cameras. It does a transition from ortho -> perspective by lerping the projection matrices while also lerping camera position and rotation. It was previously breaking lots of raycaster stuff. I think we got that all resolved now, but it's still probably a useful testing ground for raycaster/camera changes since there's a weird intermediate pers/ortho matrix during the transition (there's also still a few bugs in the code, especially if you keeping hitting space 😅).

[looeee]

I like the idea of a <Scene /> component. I find the <Canvas /> name a bit confusing as it lies somewhere between a THREE.Scene, a THREE.WebGLRenderer, and a HTML <canvas> in terms of functionality.

Scene might not be the best name, rather something like "Layer" or "View" or somesuch, because "Scene" means something very specific in gamedev (and also quite possible outside of it) (And also because it doesn't have a 1:1 relation to THREE.Scene) ( verbatim @hmans )

Yeah, this is a good point. I think it's important to consider the terms used in the greater CG/graphics/three.js community when choosing names here as it can lead to confusion for people coming from a gamedev background instead of a webdev background, or people referring to the three.js docs and then looking for matching functionality here - see my confusion over the term "viewport", for example: #1892. It can also be a limitation if we later decide functionality of the "actual" term is needed, then we need to create another new name leading to even more mismatch in terms.

[drcmda]

@gsimone this isn't straight forward, as well as the inject hacks. the problem is this:

• you create a new onion layer where everything below will receive that camera and not root camera.

but events do not come from within the component tree, they don't have access to it or even have context, it's just addEventListener on the canvas dom node. when they raycast they need to access the camera from root-state. they don't know that some mesh had an onion layer way up the component tree and iterating upwards doesn't seem feasible.

a solution could be https://github.com/facebook/react/tree/main/packages/react-reconciler#getchildhostcontextparenthostcontext-type-rootcontainer

theoretically a node can inject a context (any object) that is then known throughout the sub graph. react-dom uses that for svg's.

<inject camera={foo} />
  <group>
    <group>
      <ComponentThatNeedsTheSystemCamera />

but the api is a bit strange: getChildHostContext(parentHostContext, type, rootContainer) only gives me a string type, no access to props. but maybe i can create an empty {} in getChildHostContext, mutate it in createInstance, and update in prepareUpdate. i should now be able to access the context anywhere down the nested sub graph, even in the event handlers.

getChildHostContext(type) {
  if (type === "inject") return {}

createInstance(type, props, context) {
  const instance = new THREE[type]
  if (type === "inject") {
    context.type = type
    context.props = props
    instance.r3f.context = context
  }

prepareUpdate(type, props, context) {
  if (type === "inject") {
    context.type = type
    context.props = props
    instance.r3f.context = context
  }

Sorry for all this reconciler gibberish 😅

[AndrewPrifer]

@looeee I already have an implementation here: #2133, which improves on these ideas quite a bit, if you'd like to discuss implementation specifics :)

[drcmda]

i looked into the reconciler stuff and sadly, it won't work: facebook/react#24138

[CodyJasonBennett]

Continuing scene/inject discussion in #2142.

[CodyJasonBennett]

This looks to be implemented with #2153 of v8, but happy to re-open if anything comes up.