Slow performance on showing image in CameraFrustumHandle

[abcamiletto]

I am currently using the .image setter of the CameraFrustumHandle. In my use case I have a lot of images to show in parallel (120+) and it becomes quite slow since the input is a numpy array that is then encoded into bytes.

The current implementation does this

viser/src/viser/_scene_handles.py

Lines 354 to 369 in faa9c57

	def image(self, image: np.ndarray | None) -> None:
	from ._scene_api import _encode_image_binary
	if image is None:
	self._image_data = None
	return
	if self.image_media_type is None:
	self.image_media_type = "image/png"
	self._image = image
	media_type, data = _encode_image_binary(
	image, self.image_media_type, jpeg_quality=self._jpeg_quality
	)
	self._image_data = data
	del media_type

Which make sense to me but does not scale. I am decoding the jpg images from disk, converting them to numpy array only to convert them back to jpg and visualize them.
Even with threading this cannot be done in real time with large number of cameras.

I am very open to work on a PR to support either accepting paths or streams of bytes in the image setter method.
Please let me know if that would be welcomed and how you would prefer this to behave!

[brentyi]

Hi @abcamiletto!

One thing you could try is the _image_data property that the .image setter uses under the hood, eg:

from pathlib import Path

frustum_handle._image_data = Path("some_image.jpeg").read_bytes()

If you can confirm that this is much faster, I'm also open to supporting already-encoded images by (1) changing the type of the .image property from np.ndarray | None to np.ndarray | bytes | None and (2) bypassing the encoding logic if bytes are passed in. I'd like to be consistent here though, so if we make this change we should also do it everywhere else an image can be passed as an argument in viser.

[abcamiletto]

Hi @brentyi, thanks a lot for the answer!
Your suggestion actually works flawlessly and as fast as it can be.

For anyone having this problem in the future, it's also necessary to set the image media type as follow to trigger the rendering

from pathlib import Path

frustum_handle._image_data = Path("some_image.jpeg").read_bytes()
frustum_handle.image_media_type = "image/jpeg"

I do agree that having the option to provide images as bytes could be nicer, but I am afraid implementing this across the project could go over my availability for the PR.
Thank you so much for the suggestion!

[abcamiletto]

This also leads to a weird behaviour:

from pathlib import Path

frustum_handle._image_data = Path("some_image.jpeg").read_bytes()
frustum_handle.image_media_type = "image/jpeg"

print(frustum_handle.image) # None

I assume the underlying getter should check if _image_data is present but _image is None and decode it if that is the case.
If you'd be ok with this solution I can open a PR for it

[abcamiletto]

@brentyi
I opened this PR to address this side effect.

I could simply fix it from my side of the application, but it seems to me this can be beneficial to viser itself