Remove panics from planning logic #771

Job	Run time
build-vegafusion-jupyter-packages	7m 53s
build-vegafusion-packages	50s
build-vegafusion-python-embed-linux-64	4m 52s
build-vegafusion-python-embed-linux-arm64	5m 47s
build-vegafusion-python-embed-osx-64	4m 55s
build-vegafusion-python-embed-osx-arm64	5m 37s
build-vegafusion-python-embed-win-64	5m 58s
build-vegafusion-server-linux-64	4m 6s
build-vegafusion-server-linux-arm64	4m 46s
build-vegafusion-server-osx-64	4m 12s
build-vegafusion-server-osx-arm64	9m 30s
build-vegafusion-server-win-64	6m 44s
build-vegafusion-wasm	4m 19s
check-rust	8m 50s
test-rust (ubuntu-20.04)	34m 32s
test-rust (windows-2022)	37m 13s
test-rust (macos-11)	37m 43s
test-vegafusion-python-linux-64	7m 17s
test-vegafusion-python-osx-64	10m 5s
test-vegafusion-python-win-64	15m 47s
test-vegafusion-jupyter-linux-64	30m 32s
	4h 11m 28s

Provide feedback