Bevy UI and Scene Evolution Proposal

[cart]

Introduction

My highest priority right now is to prepare Bevy's UI system for the Bevy Editor (and a secondary priority is improving the Bevy Scene system). For the past week I have been deep diving on the prior art in this space, experimenting, and consolidating my thoughts on the direction we should be going in.

I have looked at 3rd party Bevy UI frameworks like kayak_ui (author: @StarArawn), belly (author: @jkb0o), cuicui (author: @nicopap), and bevy_lunex (author: @IDEDARY). I'm combining that with my thoughts and opinions I've developed via professional experience with webdev (largely React and Angular).

I've also looked into bevy_proto (author: @MrGVSV): a surprisingly feature complete bevy scene system.

The TLDR is that I think we should take the best ideas from the frameworks above (especially kayak_ui, bevy_proto, and belly), add in a few new ideas and unify them into a single cohesive whole. I've started building a prototype to prove this out (which I have linked to below). My goals for this discussion are:

1. Share where my head is at after this first week of investigation
2. Outline a rough plan for the future to kick off the discussion (with prototypes).
3. Iterate with the community until we have something we all believe in. I want to get as many experts on board as possible (especially the authors of the libraries above) so we can build toward the future together.

Here are some general thoughts and goals:

• Unified and Bevy-Native: We need a nicer way to define UIs and scenes, both in code and in external assets. The existing Command spawn APIs (and ..default syntax) are a pain point for many people / don't scale/nest well. There is way too much overlap to treat UI and Scenes as different things. Defining UI should be done using the Bevy Scene system. It should build on top of (and feel at home alongside) existing Bevy pillars like Bevy ECS, Bevy Assets, Bevy Scenes, Bevy Reflect, etc.
• A New Bevy Scene Format (BSN): Rust developers should feel "at home" composing it, but it should be as terse as possible. Scenes must be human composable. It should be possible to cleanly express hierarchies of entities, with each entity in the hierarchy having zero to many "things" (I'm saying "things" instead of Components on purpose). Existing formats like XML aren't ideal because we want to cleanly support hierarchical entity nesting and adding multiple "things" to an entity. New syntax can make this much clearer. XML also doesn't map very well to Rust or to Bevy ECS. The current Bevy Scene format (while much better than it used to be) is not human-composable to the level it needs to be, nor would it be a good fit for defining scenes in rust code.
• Code and Asset Driven Workflows: Developers should be able to compose Bevy Scenes in code or in asset files. Asset scenes and code scenes should use the same syntax. Code scenes should be able to use asset scenes (and vice versa).
• The bsn! proc_macro: To accomplish this unification, we need a proc macro that can produce "bevy scene rust code" from the bevy scene format. This macro should also enable passing data from systems into the scene.
• Efficient Code-Driven Scenes: When defining scenes in code, it should be efficient. It should be as close to the cost of spawning Entities/Components/Bundles using direct World apis as possible. Basic scenarios (spawning an entity with a schematic / spawning children) should not involve unnecessary costs like allocations, hashing, or reflection.
• IDE Integration: When defining scenes in code, users should have IDE features like autocomplete, go to definition, and syntax highlighting. Rust Analyzer must play nicely with the bsn! macro.
• Scene Inheritance / Nesting: It must be possible for a given entity in a scene to inherit from a separate scene (either defined in code or in an asset file). This should include children. It should be possible to inherit from more than one scene (more on this below).
• Property Overrides: It should be possible for an entity inheriting another scene to override specific properties in that scene. For example, you should be able to spawn multiple Rock scenes in a parent scene, and override the default position of each rock. Other engines like Godot embrace this "overriding" pattern (although not to the extent that I am proposing).
• Cascading Properties: Properties should "cascade". If a scene defines a Style::color property on an entity, it should override that specific property in any scenes inherited by that entity, but keep the other Style properties defined in the inherited scenes. This isn't something unique to Styles. All scene properties should be override-able / cascade-able. If inheriting from multiple scenes, properties should "cascade" across scenes in the order the scenes are referenced
• Styles are Scenes: I don't think we need a separate style sheet system. This "cascading" scene overriding / inheritance is already a style system. Just inherit from zero to many "style scenes" and override any fields that you want. Multiple scenes will "cascade" in the order they are referenced (much like CSS). However unlike CSS, this is "fully local" (no global style overrides from arbitrary sources). Modern web frameworks have started to head in this direction and I also think it is a good idea. We can still support "theming" scenarios using the component-theming patterns adopted in frameworks like React. I don't think we need a new "style sheet" system. Scenes also have the benefit of being fully extensible. We don't need to limit ourselves to a fixed set of Style properties. Developers can define their own Schematics and use them as styles using the exact same system as the built in Bevy UI system styles. Good luck doing that in CSS! I personally think this unification is an extremely powerful (and maybe novel?) idea. Hyper flexible while also reducing conceptual load.
• Schematics: We need a new scene data model that maps to the runtime data model (very much like Schematics in bevy_proto). Schematics define arbitrary input data that is then fed into the ECS World to produce arbitrary outputs. Schematics, when being spawned, have access to the "current" entity and other scene metadata. This allows them to do things like "convert an asset path to a strong asset handle" or "convert an entity path to an actual Entity value". Schematics will largely fill the role that Bundles currently fill in Bevy. In fact, we will generally just add a Schematic derive to things like PbrBundle and SpriteBundle (although we can and probably should discuss adopting shorter names for schematics).
• Props / Properties: To enable the cascading and overriding features, Schematics fields should be Props, which are values that can be set (or unset) much like an Option type. But they have an apply function that performs the "cascading" behavior.
• Hot Reloading: We need to be able to hot reload asset scene changes (and apply those changes to everything that inherits from them). This should preserve existing entities / components whenever possible. We should also investigate "code hot reloading" in two forms:
  1. Because our code-driven format and asset-driven format are identical, I believe when in "dev mode" the bsn macro can register the source Rust file with the asset system to watch for changes. We can then hot-reload any changes that don't add new rust expressions to the bsn macro. This is inspired by the makepad approach.
  2. We can also look into dynamic reloading of Rust libs, which would allow "fully" reloading of all logic, but this is much more of a research project (but we'll be researching this anyway for the editor).
• Reactivity: We should have optional "reactivity" for scenes, fed using ECS data. This is largely motivated by UI, but I see no reason to limit it to that. This can likely build on some of the features from the hot reloading system (ex: entity stability across reloads, determining what to spawn and what to remove, etc). This should be roughly like the kayak_ui implementation, but there are improvements to be made (such as improving how we reload scenes when inputs change and making prop change detection more automatic and user-friendly).
• Entity Events: We need a Bevy ECS entity event system, with the ability to bubble events in the hierarchy (when necessary). Entities should be able to register their own handlers for events. These handlers should have access to ECS data / be ECS systems themselves. I believe bevy_eventlistener is already essentially what we need, so upstreaming that should probably be our goal. Although I think we will want to adapt the event dispatcher code to enable events triggering other events to be handled within the same frame. We will also want to disable event bubbling by default (as most user-defined events won't need bubbling I think).
• Context-free Scene Declaration: Declaring scenes in code should not require (direct) access to AssetServer, Commands, contexts, etc. They should just be data that can later be applied (using those contexts).

Note that I am currently focused on core Scene API design and dataflow. Once we resolve these things, we can focus on higher level concerns like "how will we support rounded corners", "should we use flexbox or morphorm", "what widgets will we build", "what will our Input widget look like", etc.

The Prototype

I have built a (very incomplete, super experimental, barely functional) implementation of many of these ideas, just to prove them out. Note that everything is open for discussion at this point (broad goals, implementations, syntax, etc). This is the result of a week of work and experimentation. It is not the platform we need to build on.

Note that this work is heavily inspired by bevy_proto and kayak_ui, with a little bit of belly and to a lesser extent the other libraries mentioned above. My goal is to take the best ideas from the ecosystem and mash them together with my own for a "best of all worlds" result. Note that I didn't take any code from any of these libraries, just ideas.

Also note that my prototype builds on Bevy Asset V2. This makes use of "direct asset loading" to enable producing final "combined" scenes from nested scenes. And the improved dependency tracking makes the code-driven BSN implementation much simpler.

A New Bevy Scene / UI Format

Here is what my proposed Bevy Scene (BSN) format looks like in its current form. Note that I have a working prototype implementation for everything I am covering here.

Here is a single entity with a single schematic:

Div { width: 10 height: 20 }

Note that the Div schematic I'm using in these examples is not a final api, just a way to illustrate with familiar terminology. Style would likely have its own field on the Div schematic rather than having fields like Style::width splatted onto Div.

Fields are entirely optional:

Div { width: 10 }

You can even skip everything!

Div

You can nest child entities like this:

Div [
    Div { width: 10 }
]

You can define fields and nest like this:

Div { width: 10 } [
    Div
]

Most entities will only need a single schematic. But you can add more than one using a tuple (just like we do in Bevy Rust code for Components):

(Div { width: 10 } Player { health: 100 })

You can assign names to entities using this syntax (very TBD). It will produce a Name component for the entity:

#Player1(Div { width: 10 } Player { health: 100 })

You can add the contents of a scene to an entity like this:

@"player1.bsn"

This is how you would add a scene as a child of another entity:

Div [
    @"player1.bsn"
]

You can override schematic values in scenes like this:

(Div { height: 200 } @"player1.bsn")

Assuming the player1.bsn scene contains the #Player1 entity defined a few code blocks up, this would result in a final scene that is:

(Div { width: 10 height: 200 } Player { health: 100 })

I was pretty sure that the BSN format does not fit well into Serde data model, so I built a custom parser that produces an AST that gets converted into the final in-memory Scene asset (using TypeRegistry). This means that we currently need FromBsn implementations for every type in the scene. I'm not sure this is the right move yet (I just took the path I knew would work). Serde might still be an option, either in part (ex: use FromBsn when it exists for a type and fall back to serde when it doesn't) or in full (make a serde-compatible BSN deserializer). We could also rely on Reflect for everything but "value deserialization" (at the cost of some overhead).

Using BSN in Rust Code

Scene asset files can be spawned like this:

fn spawn_scene(mut commands: Commands, assets: Res<AssetServer>) {
    commands.spawn(assets.load::<Scene>("player.bsn"));
}

You can also use the bsn! macro to define your scenes inside Rust code:

fn setup() -> Bsn {
    bsn! {
        Div { width: 10 height: 100 } [
            Img { handle: "branding/icon.png" }
            Div {
                width: 100
                height: 200
            }
            @("player.bsn")
        ]
    }
}

Rust Analyzer's autocomplete, import missing symbol, syntax highlighting, and go-to-definition all work because bsn! plays nicely with RA:

You can also pass data in from systems and define expressions inline:

fn setup(score: Res<Score>) -> Bsn {
    let score = score.0;
    let ten = 10;
    bsn! {
        Div  [
            Div { width: ten }
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

I've created a system adapter to spawn the returned BSN (not sure this will be the final api, but its convenient for testing!)

app.add_system(Startup, setup.pipe(spawn_bsn));

Note that the bsn! macro does not generate an Asset file (which would introduce more dynamic-ness and overhead than necessary). It directly creates efficient Prop instances inline, which are then turned into insertable Bundles and Components using the Schematic impl for the type and inserted into a given World. This makes it roughly equivalent to spawning Entities/Bundles/Components directly using World. The goal here is to make it an (optional) replacement for the current command.spawn() pattern people are using today in Bevy.

Defining Schematics (and Props)

The Div schematic used in the examples above is defined like this:

#[derive(Component, Reflect, Schematic, Default)]
#[reflect(Component, Schematic)]
pub struct Div {
    pub width: u32,
    pub height: u32,
}

This generates DivProps:

#[derive(Clone, Default, Reflect)]
pub struct DivProps {
    pub width: Prop<u32>,
    pub height: Prop<u32>,
}

impl Props for DivProps {
    fn apply_props(&mut self, other: &Self) {
        self.width.apply(&other.width);
        self.height.apply(&other.height);
    }
}

Prop is a simple Option-like type that looks like this:

pub enum Prop<T: Props> {
    Unset,
    Value(T),
}

In fact, we might be able to just use Option in the final impl!

Deriving Schematic for Div also produces impl FromBsn for DivProps code, which enables converting from the deserialized BSN AST to DivProps.

If you add the #[schematic] attribute to a field, it will treat the Prop as a schematic instead of "just" a prop value:

#[derive(Component, Reflect, Schematic, Default, Debug)]
#[reflect(Component, Schematic)]
pub struct Img {
    #[schematic]
    handle: Handle<Image>,
}

Handle implements Schematic and uses AssetPath as the Props impl:

impl Props for AssetPath<'static> {
    fn apply_props(&mut self, other: &Self) {
        *self = other.clone()
    }
}

impl<T: Asset> Schematic for Handle<T> {
    type Props = AssetPath<'static>;

    fn from_props(
        asset_path: Self::Props,
        context: &mut SchematicContext,
    ) -> Result<Self, SchematicError> {
        Ok(context.assets.load(asset_path))
    }
}

Note that schematics don't need to be components or bundles. They can "just" be fields.

Reactive Systems (Not Actually Reactive Yet ...)

I have not implemented reactivity yet (and this will be a challenge!). However I have wired up a pattern that I believe could be made reactive:

#[derive(Component, Clone, Schematic, Reflect, Default, Debug)]
#[schematic(example)]
#[reflect(Schematic)]
pub struct Example {
    score_multiplier: Prop<u32>,
}

fn example(In(props): In<Example>, score: Res<Score>) -> Bsn {
    let score = score.0 * props.score_multiplier.get();
    bsn! {
        Div [
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

fn setup() -> Bsn {
    bsn! {
        Div [
            Example { score_multiplier: 2 }
        ]
    }
}

The idea here is that we can use the ECS to change-detect whenever the input props change (because they are stored on a Component, which has automatic change detection). We could also probably add a new IsChanged trait for SystemParam that we implement for change detectable parameters like Res, which would enable us to re-compute the scene when score changes.

Not sure this is the best path forward yet. But I think it is a reasonable start.

Missing Features / Next Steps

Please note that this is just a prototype. In its current form it is not usable for anything. Too many missing features!

Nested Entity Overrides

It should be possible to override values in child entities

// foo.bsn
Div [
   #A(Div { width: 10 height: 10 }) 
]

// bar.bsn
Div [
    @"foo.bsn" [
        @A(Div { width: 20 } )
    ]
]

Proper Hot Reloading of Children

Hot reloading currently works, but it will create duplicate children on each reload. We need to add logic to write on top of entities whenever possible, preserve moved entities (if they have the same name), and ensure we never create duplicates. Non-trivial! I suspect we can use this logic for reactivity as well.

Resolve "relative entity paths" in Schematics

It should be possible for an EntityPath Schematic to exist that allows entities to reference other entities in the hierarchy:

#A(Div Foo { c_entity: "B/C"} ) [
    #B(Div Bar { c_entity: "C"}) [
        #C(Div Bar { b_entity: "../B" })
    ]
]

We also need to decide what restrictions we will place on this. Allowing arbitrary access to the hierarchy means we need to spawn all entities in a scene before we insert components. If we only allow accessing children, we can directly spawn "bottom up". Will we allow referencing entities "outside" of a scene?

For Loops in BSN Macro

We need to be able to generate children from iterators over inputs

bsn! {
    Div { width: 10 height: 10 } [
        Img { handle: "pretty.png" }
        for item in items {
            Div { width: item.width }
        }
    ]
}

Reactivity

As mentioned in the "reactive but not actually" example, this is a big chunk of work that needs to be done. We can learn from kayak_ui here, as well as more complete / battle tested reactive systems such as React, Solid.js, etc.

Versioned Imports and Migrations

Currently "asset bsn files" use the "short name" as defined in the TypeRegistry. This, on its own, is not a safe approach to referencing Schematics. Every schematic must have a version number associated with it so we can properly migrate it across Bevy versions. I think we'll want something like "versioned imports" to encode this. It should not be possible to access a type in a scene file without specifying the version (or at least, we should warn about this on load). For simplicity, we likely want a "bevy scene prelude" that pulls in all built in bevy schematics:

use bevy(0.11)::*

PbrBundle { mesh: "some.gltf#Mesh0" } [
    PbrBundle { mesh: "other.gltf#Mesh0" }     
]

We'll also want some way of resolving conflicts (ex: import aliases and/or fully qualified schematic names).

Deriving Schematic for Enums and Tuple Structs

#[derive(Schematic)]
struct Foo(String);

#[derive(Schematic)]
enum Foo {
    Bar,
    Baz,
}

Improved Errors

The BSN parser does not currently provide line information / "error location context" in the source asset. This is a must have.

Nested Code-Driven Bsn Scene Dependency Tracking

Currently when nesting a code-driven Bsn inside of another code-driven Bsn, I don't properly wait for scene dependencies in the child to load before trying to spawn the parent.

ToBsn

We'll need to be able to serialize Schematics to BSN (my prototype can only read the format).

Other Missing Features

• Enums and tuple structs are largely unimplemented or incomplete, both in the BSN asset parser and in the bsn! macro.
• The Name Syntax is not supported in the bsn! macro yet
• Separate Props derive. Currently the only way to derive Props is via Schematic. But for nested types we'll probably want non-schematic prop derives.

Open Questions

• Reflect + Schematic + Props + ToBsn + FromBsn derives for every Component (and Bundle) is a lot of codegen. We should make sure we are generating the absolute minimum amount of code required for all of the functionality we need. I believe Props remove the need for FromReflect (and therefore also Default) derives on components and bundles.
• Some might look at the Props generated in the Schematic derive (and the Prop-cascading system generally) and think "@cart why didn't you use Reflect for this?". I'm reasonably certain we could do this if we really wanted to (ex: something like Reflect::apply + DynamicStruct with Options wrapping values). I chose not to follow this path because that would incur some pretty serious overhead (hashing, string comparisons, dynamic dispatch, etc), making the system an unsuitable replacement for code-driven scene spawning (and making our asset-driven scene spawning less efficient too). I'm open to increasing our usage of Reflect (ex: maybe for removing the need for FromBsn derives everywhere), but not if it comes at the cost of good performance. I'm reasonably certain that Props are the right tool for this job. Reflect will still be useful for things like entity inspectors, auto-generating widgets for scene/schematic editors, migrations, diffing rust types for dynamic lib reloading etc.
• Layout: should we move to morphorm for layout and drop flexbox? It has a lot going for it.
• Do we like the BSN syntax? What should change?
• Can we use the proc_macro parser for both the bsn! macro and the scene format? Should we? It would ensure both behave exactly the same.
• Belly has the concept of "data bindings", which are built on a generic "ECS data binding system". This is a powerful concept that seems like it could be useful, but I'm also a bit worried about the complexity effects such a system would have on Bevy code (individual entities' components having arbitrary dependency arrows pointing anywhere in the ECS, with arbitrary graph-like dependency chains). We should decide if this is something we want in an MVP UI framework. I think the answer is no (in the context of UI) if we have "reactivity". Reactivity is a bit more principled + hierarchical, but it is also more limited! In a world where we have prop-driven reactivity, I see data binding as being more relevant to non-ui scenarios. My inclination is to table arbitrary data binding for later and focus on prop-driven reactivity for UI.
• We can (and probably should) support inline assets and resources in scene files. This could probably accomplished via schematics (ex: a Res<T: Schematic> schematic and an Asset<T: Schematic> schematic), but we might also want special syntax for it.

Call to Action

I've made this post as early as possible because I think our first priority should be synchronizing vision and consolidating efforts. Please read through this (especially if you are one of the "3rd party bevy plugin developers" listed in this post) develop your opinions on where we should be headed, and respond here.

[TheRawMeatball]

Quite exciting! Overall, I like the ideas that are present here, but would like to highlight some I think are important.

First, I want to emphasize the question around whether we want data bindings: The existing examples show promise as a way of deriving an inactive scene composition from existing game state, and keeping it up to date with said game state, but there's nothing about UI feeding back into the game state - not even a button. I'd like to see an example of how a button works under this framework, where it seems scenes are our "reusable widget abstraction", and ideally some more complex examples featuring where two-way data binding shines, such as a textbox or a slider, if anything so we know what tradeoffs we're making.

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

Third, another concern is how the current examples lack anything which could be considered "UI state". It seems the entire UI is derived directly from game state. For various things like animated widgets, we will most likely require a way for UI to have and manage some state of its own. This state could (and in my opinion should) be stored in the ECS at the end of the day, but its still a conversation that needs to be had, and I'm hoping i can start it.

To wrap things up, most of the points I raised here are stuff I ran into back when I was implementing my last UI prototype https://github.com/TheRawMeatball/ui4, after multiple failed prototypes, and which took a strongly data-binding based approach to things. It is old code, based on an ancient version of bevy, and ugly as sin, but it was feature-complete enough to implement some simple animations and a TodoMVC example, so perhaps it's still worth a look.

[hymm]

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

What I liked for this in web was css variables. For the bsn format I'd imagine something like:

// some "widget.bsn"
Div { --primary_color: red secondary_color: blue } [
    Div { color: var(--primary_color) }
    Div { color: var(--secondary_color) }
]

(primary_color: orange @"widget.bsn")

syntax is a placeholder. not sure what limitations we'd have here

[cart]

but there's nothing about UI feeding back into the game state - not even a button. I'd like to see an example of how a button works under this framework, where it seems scenes are our "reusable widget abstraction", and ideally some more complex examples featuring where two-way data binding shines, such as a textbox or a slider, if anything so we know what tradeoffs we're making.

Yup I agree this is a gap. Its on the "todo list", but I neglected to include it in the "missing features" section.

The general idea is to utilize something like bevy_eventlistener for the "event handling" side of things, which would in turn set state that results in a re-computation of the scene:

#[derive(Component, Clone, Schematic, Reflect, Default, Debug)]
#[schematic(my_button)]
#[reflect(Schematic)]
pub struct MyButton {
    width: Prop<u32>,
}

pub struct Clicks;

fn my_button(In(props): In<MyButton>, clicks: Res<Clicks>) -> Bsn {
    let clicks = clicks.0
    bsn! {
        (
            Div { width = props.width } 
            OnClick(|event: Listener<ClickEvent>, mut clicks: ResMut<Clicks>| clicks.0 += 1)
        ) [
            Label { val: {format!("Clicks: {clicks}")} }
        ]
    }
}

Note that I used a Resource here for simplicity (and how it easily ties into the IsChanged SystemParam trait I discussed), but I think it should be possible to store this state in components on other "state entities" and/or have the ability to store it on the UI entity itself. Worth exploring!

I think the desire for "for-each" style systems increases with reactive UI. It would be nice to be able to express:

fn my_button(In(props): In<MyButton>, clicks: QueryItem<&Clicks>) -> Bsn {
    let clicks = clicks.0
    bsn! {
        (
            Div { width = props.width } 
            OnClick(|event: Listener<ClickEvent>, mut clicks: QueryItem<&mut Clicks>| clicks.0 += 1)
        ) [
            Label { val: {format!("Clicks: {clicks}")} }
        ]
    }
}

Where QueryItem is populated by some my_button.run_on_entity(world, current_entity) (and it could also implement IsChanged).

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

This is a reasonable take. "reactive" scenes are currently "input prop encapsulated", meaning that it is not possible to override anything in the internal schematics (it must go through props).

We could consider adding something similar to asset files, but I suspect in the context of "editor / asset driven scenes" arbitrary nested override behavior will generally be more desirable.

Third, another concern is how the current examples lack anything which could be considered "UI state". It seems the entire UI is derived directly from game state. For various things like animated widgets, we will most likely require a way for UI to have and manage some state of its own. This state could (and in my opinion should) be stored in the ECS at the end of the day, but its still a conversation that needs to be had, and I'm hoping i can start it.

Agreed (see my reply to the first part of your message).

To wrap things up, most of the points I raised here are stuff I ran into back when I was implementing my last UI prototype https://github.com/TheRawMeatball/ui4, after multiple failed prototypes, and which took a strongly data-binding based approach to things. It is old code, based on an ancient version of bevy, and ugly as sin, but it was feature-complete enough to implement some simple animations and a TodoMVC example, so perhaps it's still worth a look.

Oops yeah it totally slipped my mind that you did all of that experimentation. Sorry about that. Reviewing that is now at the top of my stack!

[nicoburns]

+1 for no nested overrides, and making "everything props" such that if you want to override something deeply nested then you need to pass it through explicitly. We could also have something like React's "context" which is available for an entire subtree, but still requires the nested component to explicitly expose the thing you want to change as a parameter.

If that doesn't work for "scenes" then I'd take that as evidence that we shouldn't have a unified API for scenes and UI.

[JMS55]

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

Generally I think that should be solved by widgets-as-functions / allowing nodes as props themselves. Example (ignore the syntax not using bsn!, it's pseudocode):

fn main_menu() -> Bsn {
    column(
        text("hi"),
        submenu("foo", button("Exit", Style.red)),
        submenu(SubmenuProps { // Different syntax, struct as single arg instead of many arguments
            label: "foo",
            slot_foo: button("Exit", Style.red),
        }),
    )
}

[ickk]

In the scene format, I think we should consider giving props to scenes themselves as top-level values that can be referenced at arbitrary places within the scene. This makes scenes more "function-like" giving them inputs, and potentially simplifies the data flow (you could avoid implicit cascading. Cascading seems tricky because for instance we want to support selectively cascading a pair of different Colors to alternating children).

Scenes with top-level pluggable props also start to look like something that can be modelled and composed using a visual node-graph editor, which may be a usecase we want to support in the future.

Additionally, consider syntax for doing arithmetic in scene definitions, similar to css calc & var. A child may want to inherit some (or multiple) values from its parent but with some adjustment. I think this needs to be expressible in the scene format directly if we expect users to write & style UIs.

[CooCooCaCha]

Regarding calc I wonder if we could make it even more general-purpose by allowing scenes to "request" a value from whatever system is loading the scene. So for calc we could have a MathProvider that you register with bevy before loading a scene. Then scenes can use functions provided by MathProvider inside the scene definition. In this example, any time calc() is called MathProvider parses it and returns a value which is then injected into that spot.

[alice-i-cecile]

This is a feature where we'll want to think carefully about the exact powers here. Allowing full eval powers is likely to lead to both spaghetti and security concerns.

[CooCooCaCha]

I agree, although I think it’s worth discussion. Off the top of my head I can think of quite a few useful helpers like math helpers (min, max, calc), color helpers (darken, lighten, color mixing), localization helpers (string lookup, date/time formatting), custom game specific helpers, etc

Now that I think about it, this is similar to helpers in templating languages like handlebars. But I’m definitely not suggesting we go full templating language.

[alice-i-cecile]

Makes sense. It feels like the right way to do this is to define magic strings in your app that cause a call to a fixed Rust function that can then be inserted into your assets. Infinitely flexible, but still completely in the App's control.

[TesseractCat]

One way to add interactivity could be to use a functional hypermedia approach. Having systems/functions that return scenes, and then having components that connect to them and describe some sort of swapping strategy. This is inspired by the HTMX library.

I imagine this working something like this:

// Could query to get the element to be swapped, the element that triggered the swap, etc.
fn example(trigger: Query<(&SwapTrigger, &Example)>, score: Res<Score>) -> Bsn {
    let score = score.0 * trigger.single().1.score_multiplier.get();
    bsn! {
        Div [
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

fn time(t: Res<Time>) -> Bsn {
    let time = t.elapsed_seconds;
    bsn! { Label { val: {format!("Time: {time}")} }
}

fn setup() -> Bsn {
    bsn! {
        (Div Example { score_multiplier: 2. } SwapOnClick { func: "example" target: SwapTarget::Children }) // Swaps children when clicked
        (Label SwapOnUpdate { func: "time" target: SwapTarget::Self }) // Swap every frame
        (Div { width: 20 } SwapOnHover { func: "foo" target: SwapTarget::Name("/A/B") }) // Swaps neighboring element when hovered
        #A(Div) [ #B(Div) ]
    }
}

This example would require the more advanced entity paths described to be able to reference neighboring/non-children entities dynamically.

[CooCooCaCha]

Honestly this seems way more complex than simply having an if-else-statement that returns different things on hover.

[TesseractCat]

I think the main question is where that if-else-statement should go. There are a few different paradigms being discussed here: Immediate (like bevy-egui), reactive, or hypertext-y (my proposal).

I think this proposal is nice because it integrates pretty well with the ECS, where all UI updates are just systems that return markup. Also no need for a virtual dom or change detection. I'm not sure how well it would generalize though. I imagine more complex UIs would need a reactive approach. But for a lot of standard game UIs (HUD, options, main menu), this might be enough.

[lee-orr]

Just a few quick comments (I'll likely look into things a bit more later, but wanted to jump in with initial thoughts now):

• I like the scene syntax overall, seems fairly readable
• For styles - I think thought needs to be put on inherited styles as well. Cascading is great for importing (the equivalent of having multiple classes on an HTML), but for some things you want to bring things to children (font size & color, for example). Having a "css variable" like behaviour would be useful as well in some cases.
• For widgets/widget-like situations - it might be worth at looking at Web Components for inspiration and the shadow dom - where they expose parts for styling and the like, but generally function as a single node in the main UI hierarchy? This could allow for both UI-only state (like an element being hovered) & explicit styling of sub-parts
• Will there be a UI-world? Essentially making the UI in the main world act as a virtual dom?

[alice-i-cecile]

I'm increasingly sold by the notion of multiple worlds by default, as it makes common operations (reset all UI, serialize all game state for networking or save games) very intuitive. The increased robustness to avoid accidentally grabbing UI / render / window elements in gameplay queries is also nice.

I have no strong feelings about whether they should be truly distinct worlds or just partitions though.

[kaosat-dev]

I agree, currently , the stop-gap solution of placing everything inside specific scenes at the root level can only get you so far , and can get cumbersome ie , adding specific marker components to the root of the UI hierarchy, the root of "level/world" hierarchy, to bulk manipulate them.

[Wcubed]

One concern I have with splitting the ui and "game" world, is that there a re quite a lot of games in which the ui is an integral part of said game world. It's often attached to specific game objects (health bars, command radial menus) or can even be seen by the in-world characters themselves, like holographic screens.
I'm wondering if splitting the ui from the game won't make it harder/more boilerplate-y to do theze kind of things.

And then there are the kind of indu 2d games where the ui is the level. And the character can move across the scrollbars and such. Now for that kind of integration, you could make your own custom ui parts. But to me one of the big selling points of an ecs is that enities aren't inherently "ui" or "2d collidable". Ideally it should be possible to add a set of components to a "button" and suddenly it has a tower defense cannon, while still being a button as well.

[SanderMertens]

Prior art for an ECS based scene format/DSL that supports hierarchies/reactivity/property overriding/variables:
https://www.flecs.dev/flecs/md_docs_FlecsScriptTutorial.html

[cart]

Ooh yeah I definitely should have given this a proper review as part of this effort. Thanks for the link, definitely something we can learn from!

[cart]

For the flecs explorer hot-reloading, are you re-creating entities each time (with new ids) or do you have a method to retain existing entities? If so, what algorithm are you using? I'd like to hot-reload scenes in a running app without removing existing entities (and ideally manually created components) whenever possible.

[SanderMertens]

The gist of it is:

• The deserializer keeps track of all entity ids it creates
• Named entities are first looked up by name, and created if they don't exist. This is the default and is also used for things like hot plugin/component/system reloading
• For anonymous entities, check if the id was already issued by the deserializer. If not, (re)use it. If the id was already issued, create a new id and keep track of a mapping from serialized id => actual id
• Any relationships/components with entity fields that store anonymous entities use the mapping for id translation

The nice thing about this approach is that it works for both clean & populated worlds. There are no guarantees that the ids of anonymous entities remain stable, but it does guarantee that the deserialized result is otherwise equivalent.

[cart]

Much appreciated! What would happen if there are two anonymous entities that are "completely different" and the user just "reorders" them in the hierarchy? (where hierarchy order matters, such as a UI hierarchy)

[SanderMertens]

I'd store an incrementing counter in a separate component (that's also serialized), and use a sorted query to access the UI widgets. Using entity ids for ordering is hard to get right (I think), lots of edge cases even outside of hot reloading, like dynamic UI widgets that receive recycled ids.

[vultix]

I feel like it would be useful to have a list of high-level goals for Bevy UI, as it can help frame many of these discussions. An example of a goal I'm unsure of is how general purpose we want Bevy UI to be - should it be focused on video game UIs, or should it be a contender for the de facto Rust GUI framework?

I'd also love to know the relative importance of these various goals. For example, to what degree do we prioritize ergonomics over performance, or vice versa?

To help facilitate this, I've gathered a list of goals and non-goals from various other UI frameworks. These aren't in any particular order, and I'm sure I'm missing many, but I feel it's a good start.

• Familiar - using concepts from other UI frameworks, potentially building on existing tools (HTML, css)
• Simple - easy to use, small learning curve, etc.
• Concise - how many lines of code to create a todo app
• Flexible - if you dream a UI, can you create it in Bevy UI?
• Responsive - fluidly respond to changes in aspect ratio, resolution, scaling, e
• Internationalization - Easy to use, robust I18n support
• Accessibility - Should be the default, not extra work to add
• Gaming UI framework - well suited for building game UIs
• Generic UI framework - well suited for non-game UIs. Could we build an IDE with Bevy UI?
• Hot reloading - how quickly can you make changes and see them reflected in the UI
• Prototyping speed - how quick is it to throw together a quick UI prototype
• Portable - web, mobile, desktop, console support, potentially embedded as well
• Styles / Theming - ease of making pretty UIs, optionally supporting multiple themes (light / dark)
• Performant - 30fps, 60fps, 120fps, more?
• Extensible - if something isn't supported out of the box, how easy is it to add
• Modular - the ability to use some parts of Bevy UI and combine them in new ways
• Native UI - potentially mimic native look & feel?
• Power efficient - low battery usage, low CPU usage, etc
• Error resilient - how easy is it to panic? How do you recover from errors?
• Small binary size
• Low level vs High level - will Bevy UI only be the building blocks or also have high level components?
• Easy painting API - support a html canvas-like api for easy 2d painting
• Beautiful - it'd be great if the "out of the box" look and feel was beautiful (definitely very subjective)
• Animations - does it easily support performant animations?

It may also be worthwhile doing a survey of which frameworks make which tradeoffs. egui nails the simple and concise goals, at the tradeoff of efficiency and flexibility (due to using immediate mode). Is there any possible way to retain that simplicity? I've personally found it incredible how much progress I can make using egui.

[alice-i-cecile]

Another goal:

• plays nice with inspector-style tooling

Some non-goals (IMO):

• immediate mode
• web-like APIs

[griffenliu]

Build a bevy ide using the bevy engine like godot😀

[delDelta]

I think this is an accurate list of goals. These also happen to be good goals for the larger goal of a bevy editor and IDE, so I'm glad it's on your list 😄

[aristaeus]

Can we use the proc_macro parser for both the bsn! macro and the scene format? Should we? It would ensure both behave exactly the same.

You can solve this by putting the parser in a separate crate (e.g. bevy_bsn_parser). Then bevy_bsn_macro and bevy_bsn can both depend on it. If you avoid circular dependencies you could put the parser in bevy_bsn and import that, but it might include other stuff and increase your proc macro compile time ;)

[B-Reif]

Syntax

Do we like the BSN syntax? What should change?

I appreciate the terse expressiveness of these examples. It might be a little too terse; for example, I'd like a separate syntax for 'override' and 'child'. Maybe something like the familiar 'spread' operator:

fn setup() -> Bsn {
    // easier to spot 'override' at a glance
    bsn! {
        Div [
            Div {
                width: 100
                ..@("player.bsn")
            }
            @("player.bsn")
        ]
    }
}

In the "Nested Entity Overrides" example, this code looks like it spawns children under foo.bsn.

// bar.bsn
Div [
    // What is this?
    // A div, whose children I am overriding?
    // Or another node, which I am providing children to?
    @"foo.bsn" [ 
        @A(Div { width: 20 } )
    ]
]

In React, some 'override-like' configuration is often achieved by using props which accept React elements, just like children but under a different key. This allows a React component's API to define what exactly is or isn't configurable. In our case we might try something like this:

// `title` and `content` are props which take `Bsn`, the same type as `children`
bsn! {
  Header { 
    title: Span { width: 100 } [ "My Title" ]
    content: Div { width: 200 }
  }
}

This also gives a natural way of 'referring' to other entities/elements, by just exposing them as part of the API.

Generally I think what is or is not "overrideable" should be a first-class part of how our data is defined, and the cleanest way to do this would be through the type system.

Terminology

At risk of premature bikeshedding, I think some early terminology might make this stuff easier to talk about. Specifically I think we should have terms for:

• The syntax for declaring UI.
  • In this post, "BSN", which is well-defined already.
  • In React, this is analogous to "JSX", which is syntax sugar over React.createElement.
• The data type returned by this syntax.
  • In this post, also called Bsn. I'd like to disambiguate it.
  • In React this is an "element".
  • This post seems to describe this data as "scenes". We can bikeshed the exact name but I think it's worth disambiguating the return type from the syntax, e.g. fn my_system() -> Scene instead of returning a type called Bsn.
• Functions which return this type.
  • In React this is a "component" (obviously a name collision for us in an ECS engine).
  • Elm calls this a "view".
  • For us, this is a special kind of system function (like "Conditions").
• The input type of such functions.
  • In React, "props". In Elm, the "model".
  • Traditionally React has distinguished between "props", which are basically immutable from the POV of a given React component, and "state", which is subject to change.
  • In our case, we have both the usual SystemParam types of system arguments, as well as the configurable 'props' specified in bsn!. Might be worth exploring this syntax a bit more.

Philosophy

To me, what makes React successful is that it basically models UI as a pure-ish function of app state : UI. If our approach is basically reactive around a return type, that should keep us on the right track. Ideally the 'scene functions' which return Bsn would be akin to ReadOnlySystems, like Conditions are.

This also means we should have a robust model around side effects and impure operations. Something like React hooks would be worth investigating if we're committed to a 'pure-ish' approach.

[Nilirad]

One caveat about the BSN syntax I found, is that a Name can by any String, including for example whitespace and the ( character, which is used to open a tuple of schematics. Surrounding the name in double quotes will make it easier to parse.

#"My (favorite) entity"(Schematic 1 { prop: 10.0} Schematic2)

[iiYese]

Something that worries me that I don't see addressed here is that this looks like you're going to have to pass some context around like egui and the mentioned existing solutions. This has never been plesant to work with especially in ECS because you have this singular construct where everything is coupled fighting with the parallel & adhoc nature of ECS.

If I have multiple separate panels that deal with completely independent things (like for instance in an editor) shoving everything through the same context gets difficult to work with quickly.

[Nilirad]

I wonder if all Bundles can also implement Schematic, or even merge the concepts of bundles and schematics altogether by integrating all the functionality in Bundle.

I recognize that in the current concept, derive(Schematic) may come with a significant amount of generated code. However, I also think that it would be quite convenient, especially for prototyping, to have direct access to bundles and components in scenes without deriving an even one more trait. It also makes the whole system easier to understand, since the user won't have to learn a new concept.

If the amount of codegen is really a concern, I thought about two things that could be done:

• Put the schematic functionality behind a feature flag, on by default.
• Adding a derive macro attribute (I don't remember the exact name), to surgically remove schematic functionality from specific bundles. I don't know though if it can be done with components, since they impl Bundle indirectly.

[cart]

I think we might want the reverse actually (derive Bundle on Schematics by default).

Bundle is a very specific and very scoped ECS concept. Tacking on a bunch of other "implicit context" onto it would probably be too much.

I think Schematic (or whatever we end up calling it) is much better equipped to be the "everything derive". For schematics that don't need to implement Bundle, we could have an opt-out.

[Seldom-SE]

This is interesting! I'd like to share some of my own experience with these subjects, since there's a lot of overlap. At work, I've co-created an asset format with similarities to BSN. It's called bauble, and it's open source, but it's mostly undocumented. I've also co-created a Rust frontend for Unreal's Slate UI, called mgui, which I've gotten permission to discuss. It uses bauble for markup, and it has its own system for reactivity. For context, here's an example/explanation:

// Bauble files are checked for correctness, parsed, and serialized to binary by our asset system at
// build time. These are type imports from `mgui`. You can import any type that implements
// `FromBauble`, or any asset.
use mgui_backend::{
    script::ExplicitScript,
    ui::Ui,
    widget::{Border, Button, Column, Row, Text}
};

ui = Ui {
    // The $ references an asset from the scope. In this case, it's a Rust script (which may instead
    // be written inline).
    init: $init,
    // Attributes are usually used for optional parameters
    content: #[padding = 5, width = 2] Border(Row([
        // In bauble specifically, you can replace certain attributes and field with scripts. The
        // `ExplicitScript($color)` is a workaround. Normally, it'd just be `$color`.
        #[color = ExplicitScript($color)] Button {
            content: #[color = #FFFFFF] Text("Toggle Me"),
            press: $select,
        },
    ])),
}

// `copy` means the value is copied wherever referenced. Without `copy`, the full path of the
// referenced asset is given to `FromBauble` instead.
//
// The `#{}` syntax is for raw values, which are basically strings. The multiple `{}`s are to escape
// the inner `{}`s, similar to Rust's `r##""##` syntax.
//
// These Rust scripts are compiled to wasm at build time
copy init = #{{
    struct State {
        selected: u8,
    }

    ctx.insert(State {
        selected: 0,
    });
}}

// This is a generator, so it returns a value. Generators may generate bauble markup for UI. We use
// a `ui!` macro that's basically the `format!` macro, but it may have inline expressions.
copy color = #{{
    match ctx.local::<State>().selected {
        true => "888888",
        false => "CCCCCC",
    }
    .to_string()
}}

// This is an event, so it accepts a value, which is `()` in this case. It knows this because it's
// used in a button.
copy toggle = #{
    let selected = &mut ctx.local_mut::<State>().selected;
    *selected = !*selected;
    ctx.temp_hardcoded_function_to_react_to_the_selection_reactivity_is_hard_lol(*selected);
}

Here's Button's definition. It's split up weirdly because of how scripts are processed, but types don't have to use weird generics to implement FromBauble.

#[derive(Clone, Debug, Deserialize, FromBauble, Serialize)]
#[bauble(
    bounds = (
        C: for<'a> FromBauble<'a>,
        Co: for<'a> FromBauble<'a> + ColorTrait,
        P: for<'a> FromBauble<'a>
    ),
    path = mgui_backend::widget,
)]
pub struct Button<C, Co, P> {
    /// Child widget
    pub content: C,
    // Alignment of the child widget
    #[bauble(attribute, default)]
    pub h_align: Alignment,
    #[bauble(attribute, default)]
    pub v_align: Alignment,
    #[bauble(attribute, default = Co::WHITE)]
    pub color: Co,
    /// Event to fire when the button is pressed
    pub press: P,
}

// And in some other module somewhere

// `Generator`'s first generic is the type specified in bauble. The second is the type that a script
// should return.
pub type Button =
    mgui::widget::generic::Button<Box<MarkupWidget>, Generator<Color, String>, Event<()>>;

Our requirements are very different, but I hope it's useful to see another similar sort of solution. Anyway, about BSN and bevy_ui:

BSN

I like that you avoided XML syntax. We used attributes in bauble to avoid having to mix properties with child widgets, and it looks like you're going with something similar. I think I like that you have them appended to the name instead of prepended.

It's great to use the same syntax in the asset files as in code. Unfortunately, though, asset files tend to be more limited. In BSN's current design, it doesn't seem like you could, for example, make buttons with Rust closures, so I think people will prefer to use code when possible. We have the opposite problem, though, where buttons must use a wasm script defined in the asset system. If you could use wasm for the asset system and ECS stuff for runtime, you might be able to get best of both worlds. But wasi (wasm but not on the web) is such a mess to work with, so it might not work for you. UI without some sort of scripting tends to be very constraining, so I hope you find a good solution.

We have type checking on our bauble assets, and it's very nice. I'm not sure how you could do that in Bevy/BSN, though, since assets are loaded at runtime after startup.

I enjoy FromBauble, so I think FromBsn can be nice. We have a bauble(flatten) Rust attribute for structs with a single non-attribute field and enums whose variants each have a single non-attribute field. It lets you write the bauble for such fields directly, without a wrapper of a struct or variant, but you can still parse the attributes for the wrapper. It gets used a lot. It's how anything that accepts a color can instead accept a script that outputs a hex code for a color.

#[derive(Clone, Debug, FromBauble)]
#[bauble(flatten)]
pub enum Generator<L, S>
where
    L: Debug + for<'a> FromBauble<'a>,
{
    Literal(L),
    Script {
        src: ScriptSrc,
        // This is like `default`, but you may not specify it in bauble
        #[bauble(as_default)]
        _s: PhantomData<S>,
        // We don't actually have this field on this type, but I've added it to show how attributes
        // work with flatten.
        #[bauble(attribute)]
        my_attribute: String,
    },
    /// If `L` is a `Raw`, it will take precedence over `Script`, so use this if you need a script
    ExplicitScript(ExplicitScript),
}

Finally, I like the name "BSN", because I'm reading it as "bosun" in my head, which is kind of cute.

bevy_ui

Reactivity is really hard! In our first attempt at that, we gradually added more syntax to the markup, such as inputs, commands, if conditions, loops, etc, until it eventually became a programming language of its own. There's just always more you want markup to be able to do. A proper scripting language (wasm in our case) helps with that.

When I make UI in Slate, I tend to make generic widgets that accept other widgets and properties as
arguments. In bevy_ui/BSN, a way of specifically exporting certain properties to be filled in externally would be nice.

Bottom text

I like the ideas here. I have a feeling there will be footguns with BSN, just because of how expressive it's planned to be, but I don't have any particular footguns in mind.

[IDEDARY]

Statement

The reason why I created Bevy-Lunex is because I was overwhelmed by all the web-tech-inspired frameworks everywhere.
I'm a Rust beginner. I started learning Rust about a year ago. I finally comprehended how Bevy works just two months ago.
Bevy-Lunex was my personal learning project in which I slowly learned different kinds of concepts of how to handle UI from scratch.
I am saying this because I want to let you know I am not some kind of senior dev veteran, so you can see from which background my feedback is coming and value it as much as you see fit. I won't pretend I am some kind of expert :)

Beginner friendly

One of my main concerns is that Bevy UI should not become overwhelming to the beginner user. Let's paint a scenario in which you are a beginner indie dev who picked up Bevy recently and just finished his game jam prototype. He wants to quickly bash together a main menu and call it a day. The current proposal uses semi-advanced concepts that are not immediately understandable to a beginner. Just by looking at the code, I don't understand how data/logic relation is modelled. I will have to read a lot about it, which is a little discouraging (as the person). Personally, I think that if Bevy wants to become mainstream, it should be attractive to game jammers.

Office suite

I also like the idea that we are preparing the UI to be used for more classic stuff than directly tailored for games. Bevy already has a low power consumption mode. That can make it a very good choice for regular desktop applications, not just games. I am very excited about this.

Data relation

I am also curious about how we can manage data across the divs. Rusts borrow checker is a little stubborn when it comes to UI and we need to dance around it. It doesn't let us store pointers to other data types (obviously), so I wonder what other pattern will we use access data A in div Foo from div Bar. Like buttons for example. When we click on a div button we expect it to trigger something that will change other divs. How do we define which div precisely should change? I solved this by using a tree-like structure hierarchy and by using paths. I'm curious about other takes on this problem.

Logic

Also, just to voice a concern, we have an amazing ECS ecosystem, but web frameworks are not ECS based, so I hope we won't gravitate too much to "non-ECS" concepts. I think I stroke a nice balance with Bevy-Lunex in this regard, where you can use Bevy Systems to query for button entities and add any logic desired. An example can be to enlarge the button image by 1.2 if the cursor hovers over it. For any button that I want to do, I just add the component to it with no changes required. I would like to see this get adopted here in some form.

Closing

I will leave the more technical stuff to be debated by more qualified people than me. I don't have that much experience in that regard. I just rode with everything that seemed to work :D
I am very excited that we are finally getting to start on a proper UI solution. Keep up the good work :D

[iiYese]

Something I want to be a part of this discussion is relation based UI. I'm still working on some changes for the next release of aery that enables this but this is just some dogfood code. The main thing to notice here is that it's an extension of how current bevy_ui works. Where current ECS can only take advantage of row polymorphism with components this takes advantage of sibling polymorphism by analyzing the hierarchy structure.

// This would be part of a UI lib. 
// Can be made completely generic and a write once use everywhere thing with individual widget logic delegated to systems callbacks with one shot systems.
// Otherwise currently requires a marker component + system pairing. 

#[derive(Component)]
pub struct ScrollSpan {
    pub lower_bound: f32,
    pub upper_bound: f32,
    pub view: f32,
    pub pos: f32,
}

#[derive(Component)]
pub struct VerticalScroll;

#[rustfmt::skip]
impl VerticalScroll {
    pub fn update(
        focus: Res<Focus>,
        mut wheel_evr: EventReader<MouseWheel>,
        mut spans: Query<&mut ScrollSpan, With<Self>>,
        widgets: Query<((), Relations<Widget>)>,
        scroll_areas: Query<(), With<ScrollArea>>,
    ) {
        let Some(scroll) = wheel_evr.iter().last() else { return };

        widgets
            .ops()
            .join::<Widget>(&mut spans)
            .join::<Widget>(&scroll_areas)
            .traverse_targets::<Widget>([focus.entity])
            .for_each(|_, _, (mut span, _)| {
                span.pos = (-scroll.y * (span.view / 4.).max(1.))
                    .pipe(|shift| shift + span.pos)
                    .clamp(span.lower_bound, span.upper_bound);
                ControlFlow::Exit
            });
    }

    pub fn draw(
        spans: Query<(&Rect, &ScrollSpan, &Paint, &MainColor, Option<&AltColor>), With<Self>>,
        mut painter: ShapePainter,
    ) {
        for (rect, span, paint, main, alt) in spans.iter() {
            let pill = (span.upper_bound - span.lower_bound)
                .pipe(|denom| (span.view / denom, span.pos / denom, rect.sizes()))
                .pipe(|(t_view, t_pos, sizes)| ((sizes.y - sizes.y * t_view), t_pos))
                .pipe(|(view, t_pos)| (*rect).tap_mut(|rect| {
                    rect.north_east.y -= view * t_pos;
                    rect.south_west.y += view * (1. - t_pos);
                    rect.z += 0.5;
                }));

            for (rect, color) in alt
                .map(|color| (pill, **color))
                .into_iter()
                .chain(std::iter::once((*rect, **main)))
            {
                painter.color = color;
                painter.corner_radii = Vec4::splat(paint.roundness);
                painter.set_translation(rect.center().extend(rect.z));
                painter.rect(rect.sizes());
                painter.reset();
            }
        }
    }
}

// The spawn API can be turned into a UI DSL with some cleanup work
// This is tile based so rect layout is handled by a tiling algorithm but other styling is done via regular components.

commands
    .spawn((Tiling::H, Rect::default(), MainColor(theme.mantle)))
    .scope::<Widget>(|mut rect| rect // Add children of type `Widget` in this scope
        .add(ScrollArea)
        .add((
            VerticalScroll,
            ScrollSpan { upper_bound: 256., view: 16., ..default() },
            MainColor(theme.surface0),
            AltColor(theme.overlay0),
            Paint::default(),
            Rect::default(), 
        ))
        .add((
            HorizontalSCroll,
            ScrollSpan { upper_bound: 512., view: 32., ..default() }
        ))
        .add(ScrollSpan { upper_bound: 512., view: 32., ..default() })
    });

I've modeled a scroll area with this that can represent arbitrary tensor data. You can scroll in as many dimensions as you like by adding more and more children. For my use case I currently only need input for vertical and horizontal scrolling, scroll bar drawing for vertical scrolling & 1-2 more dimensions for some complex widgets.

This becomes even more powerful with fragmenting relations cause they're can be more efficient for queries involving entities that are not immediately local. For UI specifically there's also many benefits to using relations like:

• Being able to create reactivity via queries and change detection (essentially diffing for free).
• Hot reloading assets fitting in much more nicely via assets as entities.

[killercup]

One thing missing from the discussion so far is a mention of the new-ish declarative UI frameworks for mobile, Kotlin Compose and SwiftUI. I'm not sure they match the goals set out above as they are for a different use case, but a lot of thought has gone into their design and implementation. Both use UI code written in the native language, come with data binding primitives, have IDE support with previews, and have inline styling.

Sadly I've also not used either much, so I can't go into detail of what we could learn from them. I'd appreciate it if someone could jump in here and fill in the gaps :)

[nicoburns]

There are also several such frameworks in Rust: Iced, Dioxus, Leptos, etc. IMO you'd want to adapt such approaches to integrate the reactivity with Bevy's ECS, but the core approach of such frameworks seems solid and well thought out to me.

[cart]

I've looked into both of Jetpack Compose and SwiftUI in the past (read through their tutorials), but I don't have any practical experience with them. I'll give them another look!

The Rust frameworks @nicoburns listed are also on my radar.

[databasedav]

there's also Dominator, MoonZoon, and Silkenweb which are all powered by the incredible FRP signals of https://github.com/Pauan/rust-signals

i just started using Bevy and was surprised to learn that the UI is going in the DSL direction when the reactivity + ergonomics of signal semantics can be accomplished without one (even without macros), see the example from the MoonZoon README:

use zoon::*;

fn main() {
    start_app("app", root);
}

#[static_ref]
fn counter() -> &'static Mutable<i32> {
    Mutable::new(0)
}

fn root() -> impl Element {
    Row::new()
        .s(Align::center())
        .s(Gap::new().x(15))
        .item(counter_button("-", -1))
        .item_signal(counter().signal())
        .item(counter_button("+", 1))
}

fn counter_button(label: &str, step: i32) -> impl Element {
    let (hovered, hovered_signal) = Mutable::new_and_signal(false);
    Button::new()
        .s(Width::exact(45))
        .s(RoundedCorners::all_max())
        .s(Background::new()
            .color_signal(hovered_signal.map_bool(|| hsluv!(300, 75, 85), || hsluv!(300, 75, 75))))
        .on_hovered_change(move |is_hovered| hovered.set(is_hovered))
        .label(label)
        .on_press(move || *counter().lock_mut() += step)
}

there has been prior discussion about integrating FRP signals into Bevy here Pauan/rust-signals#31

note i'm completely new to commenting on this level of design, so maybe i'm just inflating the weight of maintaining a DSL or maybe there's some broader game dev benefits of a scene spec that can be defined statically (etc.), but just wanted to mention the rust frontend frameworks that utilize FRP signals given the others listed

[therealbnut]

I came here to mention SwiftUI too.

The library rui might be worth looking at for this, it's not bevy, but it demonstrates some basic SwiftUI-like concepts in Rust.

Having used SwiftUI when it came out, and its predecessors, I'd say it is a notable improvement in its platform's UI ecostystem for:

• Replacing previous mutable representations of UIs with a builder-like DSL that produces a data representation of a UI. Conditional code works much the same way as Bevy's run conditions. Builders can be composed with custom builders to produce new convenient ways to make UI. This data-driven approach allows for functional purity, more efficient state management, and easier reasoning.
• It typically aims to have sensible defaults, so you can focus on expressing only what you care about. This is also platform specific, so because you have different interactions on mobile the same builder might default to slightly different UI. A game example of this would be to have different UI depending on whether you have a controller or keyboard, XBox or Playstation.
• It's driven on-demand by bindings. A builder function produces the UI data representation, the function is re-evaluated whenever the data bindings have changes. Because the UI is represented as data it's much easier to evaluate what has changed and you can relatively efficiently update the internal representation. MobX does cool things here too, but I don't think you could do it with idiomatic Rust. Data bindings work similary to bevy's ECS systems, where a binding is sort of like a Query with Changed.

I think all of this has many similarities to what @cart has described in the introduction, so I believe that it's worth a look.

[lee-orr]

Some additional thoughts:

• is an asset file necessary? Bevy prides itself on being "turtles all the way down", and (at least in the web-dev space) the people implementing UI's are developers themselves nowadays. If got library hot-reload existed, would that suffice?

• Could we use a macro to provide a writing experience similar to immediate mode GUIs, and then separate that out into the "Creation" and "Update" portions? Possibly using a leptos-style fine grained approach under the hood?

[cart]

is an asset file necessary?

The asset file is most necessary in the context of visual editor tooling, especially for large game scenes (outside of a UI context). However I've also really enjoyed composing static UIs in Godot's visual scene editor, so I suspect there will be some overlap here. In the UI space I do suspect that most people will prefer the code driven workflows, but I see no reason not to support both (given that we know we want these "asset scenes" for other reasons).

Could we use a macro to provide a writing experience similar to immediate mode GUIs, and then separate that out into the "Creation" and "Update" portions? Possibly using a leptos-style fine grained approach under the hood?

We could, although going the "immediate mode api + retained state internals" approach feels like a worst of both worlds situation (running the "render" code every frame while also diffing all outputs to check for changes).

[lee-orr]

To clarify - I don't think it'd be a matter of running the render code every frame, but rather of providing a single macro that generates two functions: a "setup" function and an "update" function. The update function would utilize changes to the state (i.e. whatever components & resources get queried) to update the display.

[lee-orr]

(I remember a video by Greg Johnston - the creator of Leptos - that is my inspiration for this idea, but don't remember which. This one might be it: https://www.youtube.com/watch?v=GWB3vTWeLd4&t=1931s)

[cart]

I'll look into leptos!

[TmLev]

While I'm no Bevy expert, I'd like to offer a thought for consideration. While fully integrating Leptos might be challenging, I believe incorporating some of its concepts or components for creating and managing UI could be beneficial.

[viridia]

Over the last month I've been doing a lot of prototyping on UI solutions. One thing that has stymied me again and again is that there's a fundamental conflict between React-style reactivity and ECS.

For example, suppose you want to have a React-style "hook" that returns a reference to a resource. Let's use Dioxus-style syntax for this example:

fn example(cx: Cx) {
   let res1 = cx.use_resource::<Res1>();
   let res2 = cx.use_resource_mut::<Res2>();

   // output your UI nodes here
}

This seems simple, but it won't work in Bevy. The reason is because in order for "Cx" to be able to return a mutable reference to a resource, it has to have a mutable borrow on the ECS world. Which in this case produces a borrow conflict. Thus, your UI component can access at most a single mutable resource - which is fine for simple components, but obviously runs in to scaling problems for anything complex.

This would be trivial to solve if, for example, you could pass around a reference to a resource in an Arc<Mutex<>>, but that is not how resources work.

Note that interior mutability of Cx won't save you here, because all of the techniques for interior mutability limit the lifetime of the borrow.

You can also try to do something like dependency injection, like one-shot systems:

fn example(cx: Cx, res1: UseRes<Res1>, res2: UseResMut<Res2>) {
   // output your UI nodes here
}

Here I'm assuming that UseRes is like Res except that it also adds reactive dependency tracking. The problem here is that while this may work for resources, it doesn't work for components or anything that is looked up via a runtime value:

fn example(cx: Cx, cmp: Component<MyComponent(e)>) { // e is not a constant!
   // output your UI nodes here
}

However, even if you manage to solve these problems, there's another problem: event handlers:

fn example(cx: Cx) {
   let res = cx.use_resource_mut::<Res>();
   
   let elt = /* create UI elements */
   elt.on_pointer_down = |ev| {
      /** Do something with 'res' */
   }
}

The problem here is that the event handler (and I mean this generically, whether you are using bevy_mod_picking or some other system) lives longer than the function that creates the UI. (Unless we're talking pure immediate mode, which has other problems). Thus, you can't pass around res because it doesn't live long enough. Also, because (if we're using something like bevy_eventlistener) there's no guarantee that the event handler will be called on the same thread as the code that sets it up.

You can of course design an event handler system in which the handler accesses the resource independently (not using a closure), but now you lose the reactive tracking relationship between the render function and the handler.

Why do I say that there's a "fundamental conflict"? You have to think about what life was like before React. One of the key advancements of React and others like is is the idea that UI components are self-contained and modular. A UI component isn't just about rendering, it's also about state and interaction, and all those things are contained within the component and can be instantiated many times. Unlike, say, Qt or Java Swing, a React component doesn't need to define a bunch of data types and functions at the global level. Any state that it needs can be shared via closures with its event handlers, it all comes together as a package, you don't have to create/register a separate state object every time you want to re-use a widget.

ECS is just the opposite: the ECS philosophy is that everything gets shattered into individual fragments (systems, components, resources) all of which exist at a global level. Complex widgets are no longer self-contained packages. Everything needs to be visible to the scheduler so that it can work its parallelism magic.

Frameworks like Dioxus or Iced don't have these problems because their states are portable: you can pass around state from one widget to another, up and down the hierarchy, without having to pass some mutable borrow of a World.

[nicoburns]

ECS forces us to turn what would otherwise be internal details of a widget into a globally registered system. It means that widgets are no longer "self-contained" but rather atomized into a bunch of global definitions: custom types, custom systems, custom components, setup registration and so on. This is fine if your widgets are simple. The problem comes when you have hierarchies: the behavior of the "OK" button for the item picker dialog is completely different than the behavior of the "OK" button for the quit game dialog. Now you need to invent and register a new ECS component for every different behavior that a button can have.

@viridia I wonder if you could solve this with a single component WidgetState that contains a Box<dyn Any> that is downcast by the framework before being passed to the component? You could probably put a Default bound on it too so that the framework could construct prior to the first render. This would then be accessed by the widget using something like a State<&mut MyStateType> parameter.

[IronGremlin]

Now you need to invent and register a new ECS component for every different behavior that a button can have.

Dumb question maybe but if this paradigm scales perfectly fine for writing a whole ****ing video game why are we convinced that it will not scale well for writing some UI for that video game?

[viridia]

@IronGremlin My previous game + editor, written in TypeScript, consists of 70k lines of code. Scanning the source code, there are over 3,000 places where a UI component is instantiated.

Now imagine that every widget takes 5x as much effort as before. This isn't a problem for a game, because game UIs generally aren't that complex. It's a big problem for tools, though.

[IronGremlin]

But that is not what you're talking about.

You are talking about special case behavior dispatch and managing the state of related entities.

Neither of those have anything to do with the lines of code in your project or the number of times you instantiated a widget.

For special case behavior dispatch, we just got one-off systems, and for managing the state of related entities, you store the state in components on those entities, and then you query for them. If you have common states or structures with minor variations typically you make a custom command or a function.

Like I might get behind the idea that some of these solutions aren't especially ergonomic for UI development because X, but it kind of feels like we aren't even trying to use the same problem solving toolset we'd use for a complex ECS problem, it feels like we're trying to use the same problem solving toolset we'd use for a complex web UI - which is confusing because this isn't a web UI, we don't have the same primitives, and I shudder to think that anyone would actually WANT them.

[CooCooCaCha]

I’d also like to remind that this isn’t just UI. It’s a new scene/prefab system with reactivity.

[vultix]

For a while now I've been considering if adding for-each systems may be a big ergonomics boost, specifically for event handling and bsn!

What are for-each systems?

Today, a typical bevy system may look like this:

fn apply_velocity(
    mut query: Query<(&mut Transform, &Velocity), Without<Disabled>>,
    time: Res<Time>,
) {
    for (mut transform, velocity) in &mut query {
        transform.translation.x += velocity.x * time.delta_seconds();
        transform.translation.y += velocity.y * time.delta_seconds();
    }
}

// Register the system
app.add_systems(FixedUpdate, apply_velocity)

A for-each system automatically applies the for loop for you:

fn apply_velocity(transform: &mut Transform, velocity: &Velocity, time: Res<Time>) {
    transform.translation.x += velocity.x * time.delta_seconds();
    transform.translation.y += velocity.y * time.delta_seconds();
}

// Register the system
app.add_systems(FixedUpdate, apply_velocity.for_each::<Without<Disabled>())

Additionally, there should be some way to run a for-each system on a single entity:

world.run_system(apply_velocity.on_entity(my_entity))

How does this improve event handling ergonomics?

Currently, in event handlers, we have to write a query to modify or read components of the entity handling the event. This feels unintuitive and unergonomic, as we already know there is exactly one entity handling the event, meaning we need to do query.get(entity).unwrap(). A for-each system would allow us to simplify this.

Example: bevy_eventlistener goblin example

Consider the minimal example of the excellent bevy_eventlistener library. It spawns a "Goblin" entity with 3 pieces of armor.

commands
        .spawn((
            Name::new("Goblin"),
            HitPoints(50),
            On::<Attack>::run(take_damage),
        ))
        .with_children(|parent| {
            parent.spawn((
                Name::new("Helmet"),
                Armor(5),
                On::<Attack>::run(block_attack),
            ));
            parent.spawn((
                Name::new("Socks"),
                Armor(10),
                On::<Attack>::run(block_attack),
            ));
            parent.spawn((
                Name::new("Shirt"),
                Armor(15),
                On::<Attack>::run(block_attack),
            ));
        });

Each piece of armor can receive an "Attack" event. The armor reduces the damage of the attack, potentially preventing the event from propagating up to the Goblin.

/// A callback placed on [`Armor`], checking if it absorbed all the [`Attack`] damage.
fn block_attack(mut attack: ListenerMut<Attack>, armor: Query<(&Armor, &Name)>) {
    let (armor, armor_name) = armor.get(attack.target).unwrap();
    // Handle the attack
}

/// A callback on the armor wearer, triggered when a piece of armor is not able to block an attack,
/// or the wearer is attacked directly.
fn take_damage(
    attack: Listener<Attack>,
    mut hp: Query<(&mut HitPoints, &Name)>,
    mut commands: Commands,
    mut app_exit: EventWriter<bevy::app::AppExit>,
) {
    let (mut hp, name) = hp.get_mut(attack.listener()).unwrap();
    // Take damage, subtracting health from hp
}

Notice that in both event handlers, we need to do query.get[_mut](handler_entity).unwrap(). I find it to pretty much always be the case that a handler system needs to extract or modify information of the entity handling the event. There's only ever one such entity, so a for-each system running on just that entity simplifies things:

fn block_attack(armor: &Armor, name: &Name, mut attack: ListenerMut<Attack>) {
    // Handle the attack
}

fn take_damage(
    hp: &mut HitPoints,
    name: &Name,
    attack: Listener<Attack>,
    mut commands: Commands,
    mut app_exit: EventWriter<bevy::app::AppExit>,
) {
    // Take damage, subtracting health from hp
}

Example 2: A Counter UI widget

The place where I think this really shines is when using bsn! to build scenes and UI components.

Here's what a simple Counter UI widget might look like without for-each systems:

#[derive(Default, Component)]
struct CounterWidget {
    pub count: isize,
}

fn increment_counter(mut query: Query<&mut CounterWidget>, event: Listener<Click>, time: Res<Time<Real>>) {
    let mut widget = query.get_mut(event.listener()).unwrap();
    widget.count += 1;

    // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
    println!("Increment clicked at time: {}", time.elapsed())
}

fn decrement_counter(mut query: Query<&mut CounterWidget>, event: Listener<Click>) {
    let mut widget = query.get_mut(event.listener()).unwrap();
    widget.count -= 1;
}

impl Scene for CounterWidget {
    fn setup() -> Bsn {
        bsn![
            Button {text="-" on:click={decrement_counter}}
            Label {bind:text={CounterWidget.count}}
            Button {text="+" on:click={increment_counter}}
        ]
    }
}

This isn't terribly, but feels clunky. The actions of incrementing and decrementing feel decoupled from the widget itself, and there's a lot of extra code just to get access to the widget handling the events.

Here's what the same example could look like with for-each systems:

#[derive(Default, Component)]
struct CounterWidget {
    pub count: isize,
}

impl CounterWidget {
    fn increment(&mut self, time: Res<Time<Real>>) {
        self.count += 1;
        
        // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
        println!("Increment clicked at time: {}", time.elapsed())
    }

    fn decrement(&mut self) {
        self.count -= 1;
    }
}

impl Scene for CounterWidget {
    fn setup() -> Bsn {
        bsn![
            Button {text="-" on:click={CounterWidget::decrement}}
            Label {bind:text={CounterWidget.count}}
            Button {text="+" on:click={CounterWidget::increment}}
        ]
    }
}

The magic here is that Counter::increment and Counter::decrement are valid for-each systems! This gives us unparalleled ergonomics for very common use cases.

Note

I imagine we'd support both &mut self and mut self: Mut<Self> in for-each systems, the former automatically marking the component as changed.

Does this make introducing for-each systems worth it?

Bevy used to have for-each systems and they were removed for good reason. I want to revisit those reasons and consider how this may change things.

1. For-each systems were fundamentally limited in a number of ways. They couldn't iterate removed components, filter, control iteration, or use multiple queries at the same time. This meant they needed to be converted to "query systems" as soon as those features were needed.

I'm hoping we can lift some of these limitations. Specifically, we'd need a way for for-each systems to have other queries, making things like this possible:

impl CounterWidget {
    fn increment(&mut self, event: Listener<Click>, buttons: Query<&Name, With<Button>) {
        self.count += 1;

        // A bit contrived, but shows potential use cases
        let button_name = buttons.get(event.target).unwrap();
        println!("Button {button_name} clicked!")
    }
}

2. Bevy should generally have "one way to do things". For-each systems were a slightly more ergonomic way to define a small subset of system types. This forced people to make a "design decision" when they shouldn't need to. It also made examples and tutorials inconsistent according to people's preferences for one or the other.

This argument definitely still applies, to such an extent that I'd very much consider disallowing for-each systems anywhere except event listeners like this.

3. There were a number of "gotchas" for newcomers that constantly come up in our support forums and confused newbies:
   • users expect &mut T queries to work in foreach systems (ex: fn system(a: &mut A) {}). These can't work because we require Mut<T> tracking pointers to ensure change tracking always works as expected. The equivalent Query<&mut A> works because we can return the tracking pointer when iterating the Query.
   • A "run this for-each system on some criteria" bug that was common enough that we had to cover it in the Bevy Book.

In the case of event handlers, I find it very important that &mut T queries still work, and always mark the component as changed. If we can get that to work I'd consider this an acceptable tradeoff.

4. They increased compile times. Removing for-each systems saved me about ~5 seconds on clean Bevy compiles)
5. Their internal implementation required a complicated macro. This affected maintainability.

I'm unsure to what extent these still apply. At the very least this will be more code to maintain, meaning more bugs, slower compilation time, and larger maintenance burden.

Conclusion

I don't yet have enough data to know if this is a worthwhile tradeoff. I'd love to hear everyone's thoughts on the idea!

More examples of real-world event handling would also be great. I'm curious to see if there'd be considerable improvements in these situations.

[Diddykonga]

What if the event handlers/Listener<T> were provided an EntityMut, giving you direct access to the Entity in question, no longer having a need for extra Queries, unless you wanted them for other purposes.

[gamedev-pnc]

If inheriting from multiple scenes, properties should "cascade" across scenes in the order the scenes are referenced

Hi! Interesting discussion, but I'm confused by the idea of multiple inheritance of scenes. Wouldn't this complicate the scene analysis? In this approach: object's property can be A) cascaded from one of the upper objects in the scene tree or B) inherited from one of the parent scenes, and all those sources can also inherit recursively. It's important to minimize possible count of sources, so initial value assignment to the property could be easily found (instead of overriding it in the child object - it often becomes a mess on practice).

There are reasons why modern programming languages refuse multiple inheritance.

Worth mentioning that style and structure separation (i.e., CSS vs. clean HTML) is very good practice.

[Type1J]

Scene inheritance is more like composition than typical inheritance.

[Diddykonga]

Multiple inheritance/inheritance in general wont be introduced until we get Relations, and implement the IsA Relation, which is a whole other can of worms further down the road.

[mrgzi]

Does anyone interested in implementing Ribir to Bevy? They have their own WGPU backend and there are ready-made widgets. I think it might fit Bevy well.

[mrgzi]

Can you elaborate? How JMS55 integrated bevy and dioxus then? https://github.com/JMS55/bevy_dioxus

[nicoburns]

Dioxus has an abstracted dioxus-core layer that can be used with any widget backend. It's possible that Ribir may have something similar, but it isn't mentioned in the readme and I suspect it may not as most rust ui frameworks do not.

[Veritius]

I'm making a multiplayer game that has both a dedicated server and client application. The client has the full Bevy rendering pipeline and such, but the dedicated client doesn't - the feature is disabled.

I want to be able to use the same scene file for both applications. If I had to maintain two, it'd be a massive pain, since it'd be a larger workload and introduces potential issues where each app has different components in the same scene.

I'll give this example in some pseudo-YAML definition.

components:
     - type: Handle<Mesh>
       path: "../mdls/hello.obj"
     - type: Transform

On the client, both the mesh handle and transform components would be loaded. But on the server, since it doesn't even know what a mesh is, I want it to simply ignore the Handle<Mesh> component.

[matthunz]

Hey everyone 👋 I have proposal in my crate bevy-compose for reactivity using the react/Xilem pattern. This uses a "medium-grain" reactive approach that has a coarse reactive system like react, but finer control with Arc::make_mut like Xilem.

Counter app example:

fn ui(count: Res<Count>) -> impl Compose {
    flex((
        format!("High five count: {}", count.0),
        flex("Up high!").on_click(|mut count: ResMut<Count>| count.0 += 1),
        flex("Down low!").on_click(|mut count: ResMut<Count>| count.0 -= 1),
    ))
}

More advanced counter example, using lazy to create a component:

fn ui() -> impl Compose {
    lazy(|count: Res<Count>| {
        memo(
            count.0,
            flex((
                format!("High five count: {}", count.0),
                flex("Up high!")
                    .on_click(|mut count: ResMut<Count>| count.0 += 1)
                    .on_hover(|| {
                        dbg!("Hover!");
                    }),
                flex("Down low!").on_click(|mut count: ResMut<Count>| count.0 -= 1),
                if count.0 == 2 {
                    Some("The number 2!")
                } else {
                    None
                },
            )),
        )
    })
}

[aschrijver]

I am not in the loop with Bevy project state and see a lotta great evolution going here. I just wanted to mention recently watching a jaw-dropping presentation by Rik Arends of Makepad who create this whole UI framework in Rust where UI is in a Figma-like format you can adapt at runtime. And they also built this great editor for doing so.. using Makepad: https://makepad.dev

[viridia]

Since I last posted here, I've been doing a ton of Bevy UI-related work (R&D mostly), written a lot of code, and learned many things. My earlier comments in this thread seem naive to me now :) If could get readers in this thread to take away just one idea, it would be the need to solve problems in the right order: don't build the penthouse before you've laid the foundation.

More specifically, I would want to move the focus away from both widgets and template syntax:

• Implementing widgets at this point is premature - until we have a framework for declarative updates (explained below), widgets are just technical debt.
• Template syntax is a "nice to have" - I've demonstrated to my own satisfaction that pretty much any UI you can build with a template, you can also do with fluent syntax or builder patterns. While it might not be quite as succinct, the advantage is that these patterns can evolve rapidly as your UI framework develops. This is important so that you don't get locked in to a particular design approach too early.

The really hard problem to solve is "declarative incremental updating", a term which I will need to unpack:

• "declarative" in the sense that we can define our UI hierarchy declaratively (rather than having to imperatively write "add child widget to parent")
• "incremental" in the sense that we're not re-constructing the UI every time there's a change, we're able to apply patches to the UI hierarchy based on changes in app state.
• "updating" in this case refers to the fact that the same code is used for both the initial creation of the hierarchy, and for applying changes to it. This is critical, because it's the key productivity win for UIs in the last 20 years. It's why React/SwiftUI/Elm/Svelte/Vue and all the rest are superior to Java AWT and MFC.

The key reason why bevy_ui as it exists today is so tedious and unergonomic is not because it lacks a templating language, but because creation and updating are separate activities. When these activities are split, it means that the developer has to fetch and carry a lot of state between them, which in turn entails a lot of extra coding. By itself, a templating/macro language won't solve this problem, and can potentially make it worse unless the templating language is built on top of a robust updating solution.

Unfortunately, as I have found, an incremental updating framework can't be just bolted on to ECS, at least not in a straightforward way. "Widgets" can't be "systems" because the runtime scheduling is completely different. Widgets also can't be systems because the same widget appears in many places, whereas re-registering a system doesn't cause it to get run multiple times.

An updating solution may or may not be "reactive": immediate-mode systems like egui are both declarative and unify construction and updating, but they are not reactive in the sense of React, Solid or MobX.

In the last six months I've built two different reactive UI frameworks for bevy: quill and bevy_reactor. The former was a coarse-grained approach modeled after React and Dioxus, and latter a fine-grained approach modeled after Solid. I've convinced myself that I can build a UI framework that's good enough to build an editor; however, I also recognize that my approach has a number of weaknesses that I can't solve by myself.

One weakness is that these frameworks operate in exclusive mode. I know we've discussed this before, and while the general feedback I've gotten is that it's OK for a UI widget to require world access, I'm not sold on this. My reason has nothing to do with performance, but simply because coding anything complicated with exclusive world access means you have a lot more problems with the Rust borrow checker. It means that relatively simple algorithms turn into puzzles.

The other problem is that even though my framework is built on top of ECS, the style of coding required to use it is not very ECS-like. That is, after all the effort we've spent educating people how to rethink their game architecture in terms of entities and components, then we get to UI and say "Oh, and you'll have to learn an entirely new approach when it comes to doing UI."

Part of the tension here is due to a mismatch between what UI needs, and what ECS provides. For example:

• Systems and queries are optimized to operate on relatively flat data, or at least data that can be processed homogeneously, whereas UI hierarchies tend to be heterogenous and have deep hierarchies;
• Systems allow representing state either completely globally (entities and resources) or strictly local (the Local injection), but UIs generally need "semi-local" state, that is, state that is shared between a parent widget and its children, but neither global nor unique (in the sense that there may be multiple constellations of the same parent/child type, each with their own semi-local state).
• Systems allow injection of arbitrary data from the world, but only if the type is known at compile time, and only at the top level (meaning that if you have a system that calls other functions, injection only happens at the top); A complex dialog box or preferences panel won't always know what resources and entities are being used by it's children, so it can't inject what is needed without de-encapsulating the entire widget hierarchy - that is, making the state dependencies of every leaf widget visible to the root of the tree.

The question I keep asking myself is, is it possible to build a widget that looks more like a system? One which supports dependency injection, non-exclusive access, and so on? And every time I start to think about this I run into a wall.

In any case, what I have now is a framework that is powerful, in the sense that a small number of lines of code lets you do a lot. In that sense it's productive and ergonomic. However, it's also occasionally tricky and surprising, sometimes you have to jump through odd hoops to solve what ought to be a straightforward problem.

(I should mention as an aside that while I have doubts about parts of my framework, there are other parts that I am completely happy with, such as style builders. More generally, it's mainly the small, inner reactive kernel that troubles me; most of what I have built on top of it I am fine with.)

[matthunz]

Really interesting points 👀 totally agree here!

I'm curious how well the new Bevy observers will ease this problem. In my experience making a UI widget with components like FontSize and having the widget react to changes has been pretty straightforward already. I feel like maybe with observers we'll finally have an easier way of working with linear-style reactivity, rather than in parallel

[JMS55]

Very much agree with your list of problems and questions @viridia. A lot of these are much the same issues I brought up 10 months ago in a different thread in this discussion. I think it's valuable to read that over again.

These are also the same problems (minus UI <-> ECS interaction) that other UI frameworks are solving or have already solved, both Rust-based and in other languages. I know it's contentious, but I'd rather start with a reactive UI system that reuses an existing UI framework, rather than programming our own with much less developer time spent on it (given all the other parts of a game engine we spend time on). Likely Dioxus or Leptos. More asset/art based, less technically complex UI systems could be built with an alternate crate focused around hot-reloading BSN assets.

For UI <-> ECS interactions, I do think exclusive world access makes sense. Sure it's not "clean", but it honestly works well enough. It's the pattern I follow in bevy_dioxus, and I'm happy enough with it.

[dannymcgee]

I've also been working a lot with Bevy UI lately (including user-interactable 3D elements, which is something that's not often considered part of the "UI" question but does suffer from a lot of the same pain points). I broadly agree with your assessment here, but I do have one point of contention, and a couple of patterns I've found that may be helpful.

The key reason why bevy_ui as it exists today is so tedious and unergonomic is not because it lacks a templating language, but because creation and updating are separate activities.

I think I would actually flip the priority order here, but for largely the same reasons you point out. The biggest difficulty I have defining re-usable UI abstractions currently is the fact that describing a UI hierarchy with more than one level of depth requires access to the Commands buffer (and often other resources as well, like ResMut<Assets<_>> and Res<AssetServer> for fonts, materials and the like). That means that your widget-spawning functions need to be systems, which means that you need exclusive &mut World access to invoke them.

In my mind, having a way to statically describe a hierarchy without needing access to those resources is kind of the foundational problem — solving that would go quite a long way toward making UI immediately less painful, and reactive systems could be layered on top of it. That is basically the problem that's solved by the BSN format (and the Scene system more generally).

Systems allow representing state either completely globally (entities and resources) or strictly local (the Local injection), but UIs generally need "semi-local" state, that is, state that is shared between a parent widget and its children, but neither global nor unique (in the sense that there may be multiple constellations of the same parent/child type, each with their own semi-local state).

100% agreed, though I don't think this is necessarily an intractable problem for ECS. I'm sure this isn't a new idea to you, but a pattern I've found myself using a lot is defining a widget-specific State component that lives at the root of a particular widget's hierarchy, and using Query<&Parent>::iter_ancestors and Query<&Children>::iter_descendants to look up and act upon those state components in Update or event-listener systems.

I built a basic checkbox widget recently which is composed of three entities:

(
  CheckboxField,
  CheckboxState(Interaction),
  On<Pointer<Click>>(on_checkbox_click),
  Children([
    (Checkbox(bool), /* various visual stuff */),
    (CheckboxLabel, Text),
  ])
)

The on_checkbox_click system is defined like this:

fn on_checkbox_click(
    event: Listener<Pointer<Click>>,
    mut ew_changes: EventWriter<CheckboxChange>,
    mut q_checkbox: Query<&mut Checkbox>,
    q_children: Query<&Children>,
    q_parents: Query<&Parent>,
    q_root: Query<Entity, With<CheckboxField>>,
) {
    let e_checkbox = if q_checkbox.contains(event.target) {
        Some(event.target)
    } else {
        let e_root = if q_root.contains(event.target) {
            Some(event.target)
        } else {
            q_parents
                .iter_ancestors(event.target)
                .find(|e| q_root.contains(*e))
        };

        let Some(e_root) = e_root else {
            bevy::log::error!("Failed to find Checkbox Field root");
            return;
        };

        q_children
            .iter_descendants(e_root)
            .find(|e| q_checkbox.contains(*e))
    };

    if let Some(e_checkbox) = e_checkbox {
        let mut checked = q_checkbox.get_mut(e_checkbox).unwrap();
        **checked ^= true;
        ew_changes.send(CheckboxChange {
            target: e_checkbox,
            checked: **checked,
        });
    } else {
        bevy::log::error!("Failed to find Checkbox")
    }
}

The CheckboxState(Interaction) stores the highest-priority of the three individual Interaction states so that all three entities can be treated as hovered/pressed if any of them are hovered/pressed. The system for managing that is so long I don't even want to paste it here.

That said, I don't think it would be a huge stretch to design some kind of "hierarchical dependency injection" abstraction that allows for easy and efficient lookups of the nearest instance of a given component among the ancestors of a given entity. This would be similar in practice to React's Context API. You could even write an abstraction like that today with a custom SystemParam, though it wouldn't solve the dubious performance characteristics, nor the fact that, architecturally, this is basically object-oriented programming with extra steps.

[alice-i-cecile]

flecs has good tools for "hierarchical dependency injection". Making that fast and easy is a big part of the value in relations.

[Fidius-jko]

Is this something like gizmos?

[Cedar-81]

With regards to referencing nested entity paths I think it would be nice to enforce that every entity has unique names or maybe an optional id property that if specified is used to reference that particular entity.

Also the syntax does need improvement(I know this is still experimental) but I think an xml like syntax will be more approachable.

[Type1J]

I came here to see the current and planned state of Bevy templating and updating. I quickly realized that this issue is TL, so I DR. Can someone in the know please recap? If it's too tentative at this point, then maybe just a solution candidate outline would be nice.

[alice-i-cecile]

Broad phases:

1. Use required components to automatically insert components that "belong with" the component added.
2. Rewrite a custom, low-boilerplate scene format called BSN. Add a bsn! macro for Rust code that matches the syntax exactly.
3. Add a reactive solution to the ECS and integrate it with the scene format.

Right now, stage 1 is complete, stage 2 is in development, and stage 3 is in the R&D phase.

[HyperCodec]

Broad phases:

1. Use required components to automatically insert components that "belong with" the component added.

2. Rewrite a custom, low-boilerplate scene format called BSN. Add a bsn! macro for Rust code that matches the syntax exactly.

3. Add a reactive solution to the ECS and integrate it with the scene format.

Right now, stage 1 is complete, stage 2 is in development, and stage 3 is in the R&D phase.

Any ETA for stage 2?

[alice-i-cecile]

The hope is Bevy 0.16 :)

[Type1J]

Is there some branch in some repo with an example of bsn in its current state?

[viridia]

@cart I think that what I would like to see from you next is more detailed requirements around reactivity. Here are some options (mostly things that I have either prototyped or have been discussed) that can be used as a starting point.

The first group of options is "kinds of reactions":

• conditional sub-trees
• iteration / foreach
• use of game states as conditions
• case/switch for complex branching
• "computed" children, allowing children to be generated by complex algorithms such as match.
• dynamically computed styles
• dynamically computed components
• dynamically computed text
• conditionally-inserted components like Disabled (where the component properties aren't dynamic, but its presence or absence is).
• generic mutation effects (run this callback on the entity when something changes).

The second set of options relates to "kinds of data sources we can depend on"

• components
• resources
• assets
• queries
• derived / computed functions (signals that are transformations of other signals)
  • whether these will be memoized
• external data from outside the Bevy world

A third set of requirements are more general, not necessarily related to reactivity:

• Whether there will be support for a optional, non-template fluent API, or third-party ui libraries that offer alternative construction experiences
• Whether we want to support non-world-exclusive construction.
  • Reactions will almost certainly require world access.
  • However, the code that actually sets up the reactions and spawns the UI may not need world access, if we are willing to wait until the next command flush for everything to be properly initialized. (Not making promises here.)
• How child widgets are passed as parameters between templates.

The child widgets thing is trickier than it sounds. Take for example, a dialog box: the entity hierarchy only exists when the dialog box is open, or when it's in the middle of it's opening or closing transition animation. So the child entities of the dialog box shouldn't exist unless the dialog logic says that they should. But you want to be able to build a generic dialog box template that takes the contents as a parameter. This is easy if the contents parameter is a generating closure, hard if the child entities are created in advance.

The easy answer is "we want all the things", but some of these things rely on particular design choices, and may be easier or harder if those design choices change. At the very least, there should be a sense of which are the most important.

[matthunz]

Hey 👋 just wanted to share my design here: https://github.com/matthunz/bevy-compose

My thinking is if we go the route of course-grained reactivity we can keep the systems running in parallel.
With my crate you can define templates with traditional systems:

let zombie = Template::new(
    // Spawning a Zombie will spawn the following components:
    Zombie,
    (
        // This only runs once.
        || Health(100),
        // This runs every time a `Health` component is updated,
        // and it's guraranteed to run before other systems using the `Damage` component.
        |entity: In<Entity>, health_query: Query<&Health>, ...| {
            let health = health_query.get(*entity).unwrap();
            Damage(**health * 2)
        },
    ),
);

The systems can then be scheduled like normal systems:

App::new().add_plugins(TemplatePlugin::default().add(zombie))

And the bundles spawned with:

fn setup(mut commands: Commands) {
    commands.spawn(Zombie);
}

The systems are then automatically scheduled to run writers after readers (systems that return a component are guaranteed to run before systems that read that component). Systems are also reactive, and only run if items in their params have changed (this is somewhat inefficient now, but can eventually use DetectChanges and Changed)

I think this approach has serious benefits, one of which being we don't need to block the world. The downside of course is fine-grained reactivity always has better potential, but I've really come to like this pattern.

[kvfreedom]

PanGui looks great and is worth learning from. HTML and CSS technology stacks should be thrown into the garbage dump of history.

[Type1J]

I'm sure that the design of Bevy UI has been well informed by many UI frameworks at this point. A strong ECS focus will be needed to be viable, and I believe that the closest to ideal UI framework has been Lunex.

[ilyas-taouaou]

I hope the new bevy UI would get inspiration from Kotlin Android Compose UI framework. it's one of the best UI systems I've experienced so far.
I think using old tech like HTML, CSS and XML would be a disaster for a modern tool like bevy.

[matthunz]

I've been working on a project for more than Bevy inspired by React/Compose https://github.com/actuate-rs/bevy_mod_actuate

[useyourfeelings]

Coding will change heavily in new ui?

Should I wait for new ui system for a project with complex ui?