How do I use this framework for single image multi objects reconstruction?

[sainatarajan]

Firstly, thank you all for this great library.

I have already explored a few models (Pixel2Mesh, Sphere-FC, Sphere-GCN) using Pytorch3D to reconstruct a single object from a single image. However, I'm not sure how to proceed with reconstructing many objects from a single image.

I have a pre-trained Object Detector that will give me the number of objects in the image. What I'm trying to do now, is to create multiple initial spherical meshes and wrap them in a Meshes() object, and pass it on to the network. For example: if my Obj. Det tool detects 4 objects, I create 4 spheres and input them into the network for training. I also have corresponding 4 .obj files that I use for the loss calculation.

Can someone let me know if this procedure is the right way to approach the problem?

I'm happy to provide further information if needed.

[nikhilaravi]

Hi @sainatarajan apologies for the late response. This would be a good question for the Discussions section of the repo: https://github.com/facebookresearch/pytorch3d/discussions.

Are you trying to use only 2D supervision (i.e. train the model using differentiable rendering) or use full 3D supervision as you have the obj files for each object?

[sainatarajan]

@nikhilaravi As of now, I'm using the full 3D supervision. But I would pretty much convert it into 2D-based supervision via Diff. Rendering.