Refactor scene pointer events + support rectangular selection #157
Conversation
|
As you asked if lasso- or box-style selections would be better, I think that depends much on the case where to use it. My experience is mostly grounded in photoshop that features all kind of selections. I personally never use the lasso-selection. Instead I use polygon selections as they only require precise clicks, but not a precise mouse-path. However, in 3D-Space I assume that requirements differ in several ways: viewport-camera-projection should not affect the selection too much. Thus I think box-style selections can only help with selecting entire objects. For areas of objects the box doesn't make sense as long as the viewport doesn't feature an axis-aligned orthographic projection (as for example fusion 360 does). With nerfstudio I usually have gsplats in mind. This makes selections tricky, as they preferably should not select hazey clouds in front of the objects. If I should come up with the best and sophisticated single selection tool it would work like this: The selection would be created with a brush. By holding down a key, the brush becomes an eraser. The brush shape in space is a perfect sphere. To make a selection I have to enable selection mode and with that enabled, the sphere is always drawn as transparent sphere at the position of my mouse-cursor. Next to the button to enable the selection I have a slider that allows to control the size of the selection sphere. For better usability there might be a way to adjust the size with the mouse-wheel too. |
|
I think it might be good to first support box-selects for now, to get us started. This is probably enough to get us started on selecting gaussians (hopefully)! This is the current version of the box-select; all the mesh vertices are projected onto the viewport camera's image plane, and the vertex colors are changed to red if they lie inside the 2D selection box. Screen.Recording.2024-02-07.at.8.59.05.PM.mov
This is probably something to deal with in the python / nerfstudio end, but we could potentially make a heuristic for this using the discrepancy in the gaussian position and the rendered depth. There is definitely a strong use case for polygon select / multi-click selects, and brush-selects, like you mentioned. The code should make this easy to do in the future, since now the code returns a list of 2D screen positions (for the box). |
|
|
|
There was a problem hiding this comment.
Made some minor changes but looks good to me overall!!
@chungmin99 two questions:
- Are we sure about naming the rectangular select event
box? Is this descriptive enough? - Is the behavior when we attach two scene pointer events well-defined? When I clicked "Paint mesh" and then "Add sphere" consecutively it seemed like there was some unexpected behavior, like the cursor changing back to the standard pointer after I did a rectangular select even though it was still waiting for a click.
|
Ok!
|
|
Example of the single-scenepointerevent listening: Screencast.from.03-22-2024.11.49.42.AM.webm |
brentyi
changed the title
…udio-project#157) * Initial commit for 3D scribble, WIP * Add Viewer2DCanvas, use that to draw cursor feedback! * Remove scribble support, smoothen out box select, add example * Correctly update canvas size on resize * Refactor ScenePointerMessage, send 2D coordinates for box * Update example, walk through OpenCV/NDC coordinates * Rename SceneClickEnableMessage->ScenePointerEnableMessage * ruff, mypy * Put camera in a reasonable initial location * Add event type to on_scene_pointer * Rename example file from scene_click->scene_pointer * mypy, ruff * Add backwards compatibility * ruff, mypy * ruff, mypy * Rename click->pointer * Use resizeobserver for element size tracking * Use OpenCV image coords (normalized) * mypy * Remove irrelevant file * mypy * ruff, pyright * mypy * Nits * Try to fix mypy * Rename box to rect-select * Listen to only one scenepointerevent at a given time. * mypy, ruff * ruff * Track events for clients! * Clear pre-existing callbacks for both server/clients, include warnings. * ruff * Code nit * Nits * oops * Move pointer cleanup logic --------- Co-authored-by: Brent Yi <[email protected]>
Currently,
ScenePointerMessagesupports only "click" events (single-click).Here, we add support for mouse drag, by sending a list of ray origins/directions.
Similar to how a person may wave around their pointer finger!
Not sure what is more useful; a scribble, a lasso (similar to https://discourse.threejs.org/t/how-to-make-3d-lasso-tool/23207/2), or a box-style drag selection. Open to suggestions!
It would be trivial to add a similar logic for
SceneNodeClickMessage(perhaps to move/push objects around).Screen.Recording.2024-01-04.at.12.01.49.PM.mov