Add bounding boxes plot frontend components #5227

julieg18 · 2024-01-22T23:19:10Z

`main` <- this <- #5241 <- #5250 <- #5305 <- #5313

Demo

~~https://github.com/iterative/vscode-dvc/assets/43496356/68bf681b-d6e3-4069-818f-d3d5d2052633~~

Screen.Recording.2024-02-15.at.11.49.48.AM.mov

To Do

Add score filtering
Handle bounding box zoomed in plots
Hide other box text labels when user is hovering over a particular box (Add bounding boxes plot frontend components #5227 (comment))
Add way to deselect (or deselect/select) all boxes at once

A start to #4917

codeclimate · 2024-01-23T20:51:52Z

webview/src/test/boundingBoxesFixture.ts

+}
+
+const boundingBoxImgCoords: { [rev: string]: ComparisonPlotBoundingBoxes } = {
+  'exp-83425': [


Similar blocks of code found in 3 locations. Consider refactoring.

codeclimate · 2024-01-23T20:51:52Z

webview/src/test/boundingBoxesFixture.ts

+    { boxes: [{ h: 75, w: 100, x: 100, y: 100 }], label: 'traffic light' },
+    { boxes: [{ h: 30, w: 30, x: 190, y: 310 }], label: 'car' }
+  ],
+  'exp-e7a67': [


Similar blocks of code found in 3 locations. Consider refactoring.

codeclimate · 2024-01-23T20:51:52Z

webview/src/test/boundingBoxesFixture.ts

+    { boxes: [{ h: 100, w: 100, x: 100, y: 100 }], label: 'traffic light' },
+    { boxes: [{ h: 30, w: 30, x: 190, y: 310 }], label: 'car' }
+  ],
+  'test-branch': [


Similar blocks of code found in 3 locations. Consider refactoring.

julieg18 · 2024-01-23T21:05:15Z

extension/src/plots/webview/contract.ts

 export type ComparisonPlots = {
  path: string
+  boundingBoxLabels: ComparisonBoundingBoxLabels
  revisions: ComparisonRevisionData


I've added two values to the PlotsComparisonData. ComparisonBoundingBoxClasses inside of ComparisonPlots (holds the data on all classes in that plot/image row) and ComparisonPlotBoundingBox (holds the class/coordinates for all boxes in each single image).

Currently, I was thinking that our backend would manage the "selected" state of the classes, but we could also manage that in redux instead.

webview/src/plots/components/comparisonTable/ComparisonTableRow.tsx

julieg18 · 2024-01-23T21:22:31Z

webview/src/test/boundingBoxesFixture.ts

+  ]
+}
+
+export const addBoundingBoxes = (


I've created a util in the frontend that adds the bounding boxes to a comparison fixture for testing and storybooks. Once we've got the backend part of this working (iterative/dvc#10198), this will all be removed and bounding box data will be part of our original comparison fixture.

julieg18 · 2024-01-23T21:28:34Z

webview/src/test/boundingBoxesFixture.ts

+  PlotsComparisonData
+} from 'dvc/src/plots/webview/contract'
+
+const boxColors = [


Looking at other yolo and other bounding box examples, they seem to rely on a variation of a Tableau color palette. I've grabbed the set of colors that yolo uses for the boxes for now.

codeclimate · 2024-01-23T21:42:42Z

webview/src/plots/components/comparisonTable/cell/ComparisonTableBoundingBoxImg.tsx

+import { ComparisonTableBoundingBoxColorFilter } from './ComparisonTableBoundingBoxColorFilter'
+import styles from '../styles.module.scss'
+
+export const ComparisonTableBoundingBoxImg: React.FC<{


Similar blocks of code found in 2 locations. Consider refactoring.

webview/src/plots/components/comparisonTable/ComparisonTableRow.tsx

sroy3 · 2024-01-24T15:09:31Z

webview/src/plots/components/comparisonTable/ComparisonTableRow.tsx

+                      />
+                      <label
+                        className={styles.boundingBoxClassesButton}
+                        style={{ background: color }}


Can we ever get a background color on which to foreground color makes the text not accessible? If so, we can maybe calculate whether it should use a dark or light foreground color.

If I understood the libraries that I got the colors from, they always use white as the text color. While testing, while I think a couple colors could benefit from a darker color, white
does seem to work well enough:

Yes, apart from the neon green they all look fine. And I don't think there is anything we can do about that neon green anyway.

sroy3 · 2024-01-24T15:14:28Z

webview/src/plots/components/comparisonTable/cell/ComparisonTableCell.tsx

+      {plotImg.url && plotImg.boundingBoxes ? (
+        <ComparisonTableBoundingBoxImg
+          src={plotImg.url}
+          boxCoords={plotImg.boundingBoxes}


Instead of passing down complex objects (here and further up), can we just pass down an id and go grab the coordinates where they are used instead?

Good point on the fact that we're passing down complex objects! I'll look into moving the bounding box data into an object that ComparisonTableBoundingBoxImg could grab from.

If I understood the code correctly, plotImg.boundingBoxes is not too big of an object and only one level down, you could also pass each property from it separately using the spread operator.

webview/src/plots/components/comparisonTable/styles.module.scss

webview/src/plots/components/comparisonTable/ComparisonTableRow.tsx

extension/src/plots/webview/contract.ts

mattseddon · 2024-01-28T23:46:39Z

webview/src/stories/Plots.stories.tsx

+WithBoundingBoxes.args = {
+  data: {
+    ...defaultPlotsData,
+    comparison: addBoundingBoxes(comparisonPlotsFixture)


~~[Q] Why not add these for a single image and have them in the base fixture?~~

I can see this is the plan

mattseddon · 2024-01-28T23:50:29Z

webview/src/plots/components/comparisonTable/cell/ComparisonTableBoundingBoxImg.tsx

+      )}
+      <image href={src} width={naturalWidth} height={naturalHeight} />
+      {boxCoords.map(({ label, boxes }) => {
+        const labelColor = classes[label].color


[Q] What happens if there is no color found?

[Q] Can you clean up the names? i.e. boxCoords and classes. When props are getting drilled down they should have a consistent name the whole way through.

[Q] What happens if there is no color found?

Good catch! I've updated the code to not create any boxes if it can't find a color so nothing breaks. We could also still render the boxes with a default grayish color 🤔

[Q] Can you clean up the names? i.e. boxCoords and classes. When props are getting drilled down they should have a consistent name the whole way through.

Done!

mattseddon · 2024-01-29T00:02:27Z

webview/src/plots/components/comparisonTable/cell/ComparisonTableBoundingBoxImg.tsx

+        const labelColor = classes[label].color
+        return boxes.map(({ h, w, x, y }) => (
+          <React.Fragment key={label + h + w + x + y}>
+            <text


With the current design, I don't think we need this text as we are repeating a lot of information:

I.e. there is a big box that shows traffic light objects are yellow. I would drop this text unless we include the threshold along with the text.

Once there are a lot of boxes this will become even more crowded:

WDYT @dberenbaum?

Let's also consider how we can filter the boxes by a provided threshold (i.e. "Objects detected with prediction threshold >= 0.5")

Doesn't have to be implemented as part of this PR but the data structure/UI will probably need to support it.

I.e. there is a big box that shows traffic light objects are yellow. I would drop this text unless we include the threshold along with the text.
Once there are a lot of boxes this will become even more crowded

Good point! I chose to add labels since all classes toggle buttons may not be showing (if there are too many, we're going to add a "Show More" button that shows the hidden classes).

No strong preference though, I'm happy to remove them!

@AlexandreKempf, what do you think about keeping text labels on each individual box or would it be better to remove them?

Working on object detection, we get used to overlapping text ^^", so we can keep them it is not a problem.
In my opinion, the best behavior would be to have the image with all the labels when not hovered. When hovered, only the bounding box below the mouse gets its text displayed. It is pretty tricky to get which bounding boxes to select based on mouse location, especially with lots of overlapping cases. So generally, what we end up doing is filtering the bounding boxes based on label or score ;)

In my opinion, the best behavior would be to have the image with all the labels when not hovered. When hovered, only the bounding box below the mouse gets its text displayed. It is pretty tricky to get which bounding boxes to select based on mouse location, especially with lots of overlapping cases.

Sounds good! I'll add this feature to our todo list.

mattseddon · 2024-01-29T00:25:43Z

Something else that we will need to revisit/implement:

How are bounding boxes displayed on/with a zoomed image? In VS Code we currently open the image in an editor and that is it.

codeclimate · 2024-02-01T13:55:03Z

webview/src/test/boundingBoxesFixture.ts

+  rev: string
+  classes: ComparisonPlotClass[]
+}[] = [
+  {


Similar blocks of code found in 4 locations. Consider refactoring.

codeclimate · 2024-02-01T13:55:03Z

webview/src/test/boundingBoxesFixture.ts

+    ],
+    rev: 'exp-83425'
+  },
+  {


Similar blocks of code found in 4 locations. Consider refactoring.

codeclimate · 2024-02-01T13:55:03Z

webview/src/test/boundingBoxesFixture.ts

+    ],
+    rev: 'exp-e7a67'
+  },
+  {


Similar blocks of code found in 4 locations. Consider refactoring.

codeclimate · 2024-02-01T13:55:03Z

webview/src/test/boundingBoxesFixture.ts

+    ],
+    rev: 'main'
+  },
+  {


Similar blocks of code found in 4 locations. Consider refactoring.

AlexandreKempf · 2024-02-05T08:36:38Z

Images shown by Matt on your work looks super nice ! Congrats 🎉

codeclimate · 2024-02-20T17:32:35Z

extension/src/plots/model/index.ts

@@ -310,10 +332,26 @@ export class PlotsModel extends ModelWithPersistence {
    )
  }

+  public toggleComparisonClass(path: string, label: string, selected: boolean) {


Similar blocks of code found in 2 locations. Consider refactoring.

codeclimate · 2024-02-20T22:06:27Z

Code Climate has analyzed commit 511dfea and detected 3 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Duplication	3

The test coverage on the diff in this pull request is 95.0% (85% is the threshold).

This pull request will bring the total coverage in the repository to 95.3% (0.0% change).

View more on Code Climate.

julieg18 · 2024-02-20T22:31:58Z

extension/src/cli/dvc/contract.ts

 export type ImagePlotOutput = {
  revisions: string[]
  type: PlotsType
  url: string
+  annotations?: { [label: string]: BoundingBox[] }


DVC spec has been changed from boxes to annotations. I've updated the code accordingly.

mattseddon · 2024-03-12T05:21:34Z

Will get back to this later

julieg18 added 2 commits January 22, 2024 17:04

first iteration

3590a0a

Merge branch 'main' into bounding-boxes

a1f080a

julieg18 added the product PR that affects product label Jan 22, 2024

julieg18 self-assigned this Jan 22, 2024

julieg18 added 3 commits January 23, 2024 09:23

fix vscode tests

f16d1bb

clean up

5d5886b

Add frontend tests

84c48f7

codeclimate bot reviewed Jan 23, 2024

View reviewed changes

julieg18 changed the title ~~Add bounding boxes plots (wip, just frontend components)~~ Add bounding boxes plot frontend components Jan 23, 2024

personal review

c8400fd

julieg18 commented Jan 23, 2024

View reviewed changes

fix windows test

69530c9

julieg18 added the do not merge label Jan 23, 2024

codeclimate bot reviewed Jan 23, 2024

View reviewed changes

julieg18 commented Jan 23, 2024

View reviewed changes

webview/src/plots/components/comparisonTable/ComparisonTableRow.tsx Outdated Show resolved Hide resolved

julieg18 added 2 commits January 23, 2024 17:35

fix bug in storybooks

c2b3552

Rename labels to classes

12695c7

julieg18 marked this pull request as ready for review January 24, 2024 14:23

julieg18 requested review from mattseddon and sroy3 as code owners January 24, 2024 14:23

sroy3 reviewed Jan 24, 2024

View reviewed changes

julieg18 added 2 commits January 24, 2024 11:41

minor css cleanup

d4b5f54

type change

28cff74

julieg18 commented Jan 24, 2024

View reviewed changes

extension/src/plots/webview/contract.ts Outdated Show resolved Hide resolved

julieg18 mentioned this pull request Jan 26, 2024

Move bounding box class legend into its own collapsible row #5240

Merged

Move bounding box class legend into its own collapsible row (#5240)

1ecec8f

julieg18 mentioned this pull request Jan 26, 2024

Store bounding box plot coordinates in single obj #5241

Merged

mattseddon reviewed Jan 29, 2024

View reviewed changes

julieg18 mentioned this pull request Jan 30, 2024

Add backend logic for bounding box plots #5250

Merged

julieg18 and others added 2 commits January 31, 2024 09:00

Clean up naming and handle missing bb colors

3150a57

Store bounding box plot coordinates in single obj (#5241)

5999e1d

codeclimate bot reviewed Feb 1, 2024

View reviewed changes

sroy3 approved these changes Feb 1, 2024

View reviewed changes

julieg18 mentioned this pull request Feb 2, 2024

log_image: log bounding boxes iterative/dvclive#766

Open

4 tasks

Merge branch 'main' into bounding-boxes

940a736

julieg18 mentioned this pull request Feb 16, 2024

Improve bounding box frontend components #5305

Closed

This was referenced Feb 20, 2024

Plots: Add the bounding boxes informations in the dvc plots diff command so that VScode can display them. iterative/dvc#10312

Closed

Add Bounding Boxes annotations iterative/dvclive#776

Closed

Add backend logic for bounding box plots (#5250)

45c88b3

julieg18 mentioned this pull request Feb 20, 2024

Fix image by step plots with bounding boxes #5313

Closed

codeclimate bot reviewed Feb 20, 2024

View reviewed changes

rename dvc img output from boxes to annotations

511dfea

julieg18 commented Feb 20, 2024

View reviewed changes

mattseddon closed this Mar 12, 2024

mattseddon deleted the bounding-boxes branch March 12, 2024 05:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bounding boxes plot frontend components #5227

Add bounding boxes plot frontend components #5227

julieg18 commented Jan 22, 2024 •

edited

Loading

codeclimate bot Jan 23, 2024

codeclimate bot Jan 23, 2024

codeclimate bot Jan 23, 2024

julieg18 Jan 23, 2024 •

edited

Loading

julieg18 Jan 24, 2024

julieg18 Jan 23, 2024

julieg18 Jan 23, 2024

codeclimate bot Jan 23, 2024

sroy3 Jan 24, 2024

julieg18 Jan 25, 2024

sroy3 Jan 25, 2024

sroy3 Jan 24, 2024

julieg18 Jan 24, 2024

sroy3 Jan 25, 2024 •

edited

Loading

mattseddon Jan 28, 2024

mattseddon Jan 28, 2024

mattseddon Jan 28, 2024

julieg18 Jan 31, 2024 •

edited

Loading

mattseddon Jan 29, 2024

mattseddon Jan 29, 2024 •

edited

Loading

julieg18 Jan 30, 2024

julieg18 Feb 2, 2024

AlexandreKempf Feb 5, 2024

julieg18 Feb 5, 2024

mattseddon commented Jan 29, 2024

codeclimate bot Feb 1, 2024

codeclimate bot Feb 1, 2024

codeclimate bot Feb 1, 2024

codeclimate bot Feb 1, 2024

AlexandreKempf commented Feb 5, 2024

codeclimate bot Feb 20, 2024

codeclimate bot commented Feb 20, 2024

julieg18 Feb 20, 2024

mattseddon commented Mar 12, 2024

Add bounding boxes plot frontend components #5227

Add bounding boxes plot frontend components #5227

Conversation

julieg18 commented Jan 22, 2024 • edited Loading

main <- this <- #5241 <- #5250 <- #5305 <- #5313

Demo

To Do

codeclimate bot Jan 23, 2024

Choose a reason for hiding this comment

codeclimate bot Jan 23, 2024

Choose a reason for hiding this comment

codeclimate bot Jan 23, 2024

Choose a reason for hiding this comment

julieg18 Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeclimate bot Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sroy3 Jan 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julieg18 Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattseddon Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattseddon commented Jan 29, 2024

codeclimate bot Feb 1, 2024

Choose a reason for hiding this comment

codeclimate bot Feb 1, 2024

Choose a reason for hiding this comment

codeclimate bot Feb 1, 2024

Choose a reason for hiding this comment

codeclimate bot Feb 1, 2024

Choose a reason for hiding this comment

AlexandreKempf commented Feb 5, 2024

codeclimate bot Feb 20, 2024

Choose a reason for hiding this comment

codeclimate bot commented Feb 20, 2024

Choose a reason for hiding this comment

mattseddon commented Mar 12, 2024

julieg18 commented Jan 22, 2024 •

edited

Loading

`main` <- this <- #5241 <- #5250 <- #5305 <- #5313

julieg18 Jan 23, 2024 •

edited

Loading

sroy3 Jan 25, 2024 •

edited

Loading

julieg18 Jan 31, 2024 •

edited

Loading

mattseddon Jan 29, 2024 •

edited

Loading