found a way to get 1000+depth res quality out of 518 any v2 model maps without upscaling! 5 depth no artifacts!

[donrikk]

SO! @nagadomi i finally figured out how to properly use my own custom depth maps in ur program! apperently if u export ur video in iw3 as export disparity it obviously creates a rgb and depth folder with their frames the audio file and last the YML file. i figured out if u use the yml file instead of the video source u can just replace the maps in the depth folder with a customize and enhanced version of those maps in as long as the amount of frames matches the amount in the rgb frames folder it will convert!

what i did was i have taken the maps fro mthe depth folder, i created a script (that i will be providing u here along with examples and comparisons of 518 maps out of iw3 straight vs enhanced with no upscale both still 518 maps) that allows me to drag that depth folder on top of the script and convert it to a video with my specified fps in this methods case the fps of the source video ur doing.

once that video is created i take it and drop it into davinci and add some specific editing nodes for the map to clean it up givingit basically a bit of a wipe down before the enhancement, this node web i used in davinci resolve will also be available in the zip ill be sharing with u and u simply drag and drop the setting file on the grid in the fusion tab of davinci. then u export out at the resolution of the map in this case 910x518 and itll be cleaned up a bit and ready for the oven, the oven being topaz video enhance AI. i use the proteus enhance low to medium quality model and i bump improve detail and fix compresiion to 100 max and dehalo to 20 with a deblur of 100 and thn proceed to render that out of topaz.

once that step is done and the video is exported u then want to take that video and go back to the folder with the scripts i provided, this time use the one that turns video into image sequence, u just drag that video on to that script and it will start to create a new image sequence. once thats done u go into that folder and delete the last frame in the list of the frames for some reason the dpeth maps and rg are always off by 1 causing error code on final convert.

once u delete that frame rename the folder depth, go back into the folder where u have ur original iw3 disparity export and delete the original depth folder and replace it with the new folder and frames uve created.

then u open iw3 set ur settings, and for source u pick that YML file in that disparity folder, from there u just click start and it should use your custom maps. i can get a near zero artifacting result at 5 depth just by doingthis method. and agian the craziest part is technically the map is still only at 518 resolution! so i have achieved 1000+map res quality only by using a 518 depth map!

if ude like to know more and watch the tutorial on how i do this this file will give u the exmaples, tutorial, scripts, and node tree. i really hope u take a look and maybe in the future u could find a way to make this more ease of use kind of feature or if maybe there is a open source AI enhancer out there that does the same job as proteus in topaz it could be implimented and used on map durin the start of conversions as a first step in the conversion say a option to click on or off that says like 'map enhancement' or 'pseudo high resolution maps' then every time it processes a depth frame it enhances it while the conversion goes on, and since the map is only 518 the render time on enhancment like proteus takes 10 fps render speed so going frame by frame in the conversion process with something like that wouldnt really hinder wait time to aggressively! and it would be a way to get "higher resolution" maps out of the depth anything V2 model!

https://drive.google.com/file/d/1Mrfz2RyhVgc14-Zk4AyC4PViPnf1r5pT/view?usp=sharing

download that zip file from my drive with everything in it ready for u to take a look at my man! hope to hear from you soon!

[donrikk]

also the zip contains a full length avatar Way of Water beginning scene, its an example of the method in action and what it can produce, all the metadata is in the title i believe its an example that with this method i can go to 3 depth comfortably without artifacts and still only use 518 maps. its a beautiful result and i think itll blow u away like it did everyone else!

[loawizard]

It's very interesting what you've done over here, Donrikk, would love to know what @nagadomi thinks about it.

[donrikk]

sorry for my bad typing i got the AC on and its frigid i just turned it off but my hands are icicles XD

[nagadomi]

"enhance" probably means restoration. It may include denoising, deblurring, or something else that targets video degradation.
waifu2x has denoising feature without upscaling, but it only targets image degradation such as JPEG compression artifact.

but I cannot think of any logical reason why that would improve depth.
If the source video has compression artifact or other problems, it may be better to "enhance" the source video and then generate depth.

[nagadomi]

but I cannot think of any logical reason

basically, if an AI image model only processes depth without referring to rgb, other problems such as edge misalignment may occur (if the change is significant).

[donrikk]

very interesting! and in my conversion so far using proteus ive come across maybe 1 kinda cross talky artifact out of a 3 minute clip. i had the same notion as u going into it i was like i hope proteus doesnt change the map and just enhances it, i did multiple tests and it seems at least with that model that it doesnt mess anything up. i was full on beside myself ide take the original rgb video and the depth map video i created with the enhanced frames and i put an opacity on it and stacked them on eachother in davinci to see and they aligned perfectly to the source rgb even without the source rgb as reference! this blew my mind as i didnt think it would work it was a shot in the dark thing and it seems to have paid off so far im doing a whole movie conversion right now with the method ive produced to see what the effects are on a feature length film to really get a solid idea of the consistency of this method. but so far i havent seen any discrepancies in my results. ill totally keep u updated on it if ude like.

and also about the depth thing, so it doesnt enhance the depth by enhancing the map of course, but what it does is proteous kinda dds the pixels the image is missing thus increasing the fideleity of the image thats already ther so it only fills in the gaps of whats there it doesnt technically add anything new just adjusts what is currently present. at least thats what the proteus model seems to be doing. so basically the 518 results come out looking naturally like 518 res with banding and so on which is normal for that res, but the proteus model cleans that map to a point where it basically has the fidelity and over all visual quality of a higher res alternative. basically its faking a higher map resolution by cleaning the map up so much there isnt a distinguishable difference in the pixels when the final sbs result is produced allowing the depth map to cohere more to the source image by keeping cleaner tighter edges and improved less bandy contrasts. thats what i visually see from the enhancement anyways. i only do fix compression option and enhance detail option with a slight dehalo and a max deblur on the proteus settings so no sharpening or denoising or anything just those few options are being used.

[donrikk]

and i have a slight suspicion that proteus is trained on depthmaps somewhere in its training model cuz it handles depth maps like a charm for some reason i havent used any model that comes close to it other than the iris model that does a better job with faces at a distance in frames.