Mesh slicing

[MarcusLoppe]

I might have misunderstood some parts of the encoding & decoding so correct me if I'm wrong.

Current the SageConv takes all the faces in a F x init_dim size, and outputs F x 576 , when the face count gets high enough it will run out of memory very fast. Since the encoder needs all the data for all the other faces it cannot do this in batches.
The decoder have the same type of issues.

The current state of doing things I don't foresee any high face count meshes being generated or trained on even if there is a x10 improvements. The issue remains; the entire mesh needs to be processed at once.

I got a kinda unfinished idea but since it's a discussion page I was hoping to discuss it :)

Instead of processing all at once, you use a fixed width input for the encoder & decoder.

So lets say it needs it inputs the faces that is marked in yellow, you take these face features and input it into a fixed padded space.
Then for extra information about the rest of the shape you take the face features from the nearest edges of the faces that needs processing (green) and input as well in a fixed padded space.

With the rest of the faces you create a embedding e.g face feature summarizer. If the shape is a ball then it might not need the full mesh to get a understanding of the shape but a embedding that describe the shape.

Hopefully this will let you scale to 15k poly count and beyond. Since people want these high poly counts and not simple shapes; I think slicing it is the only way.

Here is a beautiful image that will try to describe it.

[lucidrains]

if you get something to work, you can write a follow up paper

something something hierarchical embedding

[lucidrains]

@MarcusLoppe the reason is because i don't see myself being able to open source indefinitely and there are other projects i need to make headway on

[lucidrains]

in research, you can spend a lot of time trying to come up with some solution, only to come up empty handed half a year later.

[lucidrains]

that said, hope the repository empowers you (and others) to make the next improvement

[MarcusLoppe]

@lucidrains I understand, the curve for improvements isn't very linear and dead-ends are far too common.

Just by estimating, how much work would it require to instead of feeding the entire mesh that you limit it to 100 faces + a embedding of the rest of the faces that is lets say, 200x576?

So there would be two SageConv networks, one with that creates a fixed embedding for the rest of the faces plus the one with the faces the ResNet needs to predict.
So the input for the ResNet would be: [ Overview_Embedding(200x576) + F x 576 ]

[lucidrains]

@MarcusLoppe it cannot be estimated as this is still an open research question, afaik