UniDepthV2 is a better metric model than DepthPro

[AIVFI]

Is it possible to add it to iw3? What do you guys think about this?

DepthCrafter does not do well with facial geometry and DepthPro is much better here:
Tencent/DepthCrafter#32

The best metric of how a model performs in this situation is the F-score, which is described in detail here:
From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction

UniDepthV2 has a better F-score than DepthPro in every test. Even the UniDepthV2 base model is better than DepthPro in Sintel animation:

MPI Sintel: F-score>=0.4

RK	Model Links:          Venue   Repository	F-score ↑   {Input fr.} UD2
1	UniDepthV2-Base	0.518 {1}
2	Depth Pro	0.497 {1}
3	MASt3R	0.415 {2}
4	UniDepth	0.402 {1}

[nagadomi]

It is not an open-source licensed project, so it will not be added to iw3.

---

Regarding Metric Depth:
Fundamentally, iw3 does not handle metric depth correctly.
This is because the results of all models are min-max normalized once and go through many common processing steps.
I intend to solve this problem, but it’s not done yet.

If you are particular about the metric depth, using https://github.com/calledit/metric_depth_video_toolbox
might be a good choice.
I've never used it myself, but it seems like they are very particular about metric depth and camera parameters. And it supports UniDepth.

[AIVFI]

Thank you for your answer.

However, I'd like to know your opinion on how you rate the quality of the UniDepthV2 depth maps compared to Depth Pro, which was test-implemented in iw3, although the license wasn't perfect either. I'm referring specifically to situations where Depth Pro or DAv2 made obvious errors in estimating the relative position of objects in space. You probably have such test images or video scenes.

Did UniDepthV2 come a little closer to perfect than Depth Pro or DAv2?

[nagadomi]

I am not a critic and I am not working in the depth estimation field, so I don’t really care about evaluating models. But I think MoGe is excellent for recovering 3D shapes. This is just my guess, not based on tests. Therefore, MoGe is currently a high priority for me.

[AIVFI]

I am not a critic and I am not working in the depth estimation field, so I don’t really care about evaluating models.

That is why I am asking you for your opinion. If you worked in the depth estimation field you would not be objective. I, like you, care about the best quality 2D to 3D video conversion. I mean exactly what you wrote next:

But I think MoGe is excellent for recovering 3D shapes. This is just my guess, not based on tests. Therefore, MoGe is currently a high priority for me.

I agree with you that MoGe is excellent for recovering 3D shapes. What you write about can very easily be tested on Hugging Face, where the MoGe demo has a very useful 3D Viewer in addition to the depth map. Unfortunately, the shortcomings of their model, i.e. segmentation problems and lack of fine detail, are very well known to us as well as the MoGe developers. We have been waiting for a fixed model since December:

We've recently identified some issues in our training data that have led to blurriness and missing details in particular patterns, and we're actively working on retraining the model with corrected data.
microsoft/MoGe#33

If anyone is reading this discussion and can and has the ability to compare the quality of UniDepthV2 with MoGe, Depth Pro and Depth Anything V2 and the sense of adding it to iw3 then I would be very curious.

The first version of UniDepth had significant drawbacks: lack of fine detail and no sky segmentation: Figure 5. However, in the most popular relative depth ranking it was the only one to come close to MoGe:

NYU-Depth V2: AbsRel<=0.0424 (relative depth)

📝 Note: The ranking order is determined in the first instance by a direct comparison of the scores of two models in the same paper. If there is no such direct comparison in any paper or there is a disagreement in different papers, the ranking order is determined by the best score of the compared two models in all papers that are shown in the columns as data sources. The Metric3D v2 ViT-Large rank is not based on a score of 0.134, which is probably just an anomaly.

RK	Model Links:          Venue   Repository	AbsRel ↓   {Input fr.} MoGe	AbsRel ↓   {Input fr.} BD	AbsRel ↓    {Input fr.} M3D v2	AbsRel ↓   {Input fr.} DA	AbsRel ↓     {Input fr.} DA V2
1	MoGe	0.0341 {1}	-	-	-	-
2	UniDepth	0.0380 {1}	-	-	-	-
3-5	BetterDepth	-	0.042 {1}	-	-	-
3-5	Depth Anything V2 Large	0.0420 {1}	-	-	-	0.045 {1}
3-5	Metric3D v2 ViT-Large	0.134 {1}	-	0.042 {1}	-	-
6	Depth Anything Large	0.0424 {1}	0.043 {1}	0.043 {1}	0.043 {1}	0.043 {1}