[Stable Diffusion] Enable image upscaling for external calls

[NineBallo]

Summary

Add automatic post-generation upscaling to the /v1/images/generations, /v1/images/edits, and /v1/images/variations endpoints. When a model has a pixel_upscaler recipe option (or the caller passes pixel_upscaler per-request), the generated image is automatically upscaled before returning. Both sd-cpp (Real-ESRGAN) and thenoise upscalers are supported — the correct backend CLI is selected from the upscaler model's recipe.

This also introduces refine (latent-space 2x refine for thenoise during generation) and skip_implicit_upscaling (per-request opt-out of auto-upscale). The /v1/images/upscale endpoint gains a pixel_upscaler alias for the model field. (just to keep things consistent)

Scope

• This PR addresses one clear issue or change.
• I reviewed the full diff myself before submitting.
• I removed unrelated local changes.
• I kept refactoring separate unless it is required for this change.

Testing

• Code builds without errors locally.
• I tested this change locally.
• I described the testing performed below.

Testing details:

• Generated images with and without pixel_upscaler set on the model recipe
• Verified skip_implicit_upscaling=true bypasses auto-upscale
• Verified refine=true forwards upscale=true to thenoise backends only
• Verified pixel_upscaler per-request override works for all three image endpoints
• Verified /v1/images/upscale accepts pixel_upscaler as alias for model
• Verified failed upscale returns the original image without erroring the request
• Verified response includes upscaled, width, height fields when auto-upscale occurs

Documentation

• Documentation is not affected by this change.
• Documentation is affected and has been updated.
• Documentation is affected but will be handled in a separate PR or issue.

Breaking Changes

• This PR does not introduce breaking changes.

AI-assisted contribution

Please select one:

• I used AI tools for this PR.
• I did not use AI tools for this PR.

If AI tools were used:

• I verified that I understand the changes.
• I checked for hallucinated APIs, unrelated changes, and incorrect assumptions.

[NineBallo]

This one has a few possibly polarizing decisions, opening for review but just lmk where you want to go with it. I think it makes a lot of sense but idk.

[jeremyfowers]

[AI-assisted review] Automated pre-review from repo-manager — flags for the human reviewer, not a replacement for one.

Attention level: Elevated — any reviewer can take it, but the to-dos below need resolving before approval (2 documentation gap(s))

Documentation gaps

• Add upscale_model to the sd-cpp parameter table in the /v1/load endpoint section of docs/api/lemonade.md.
  • Gap: The new upscale_model recipe option for sd-cpp is not documented alongside the other sd-cpp parameters.
  • Where: docs/api/lemonade.md
  • Why: Documentation for a new or changed feature belongs in the same PR as the code.
• Add upscale_model to the sd-cpp image_defaults example in docs/guide/configuration/custom-models.md.
  • Gap: The image_defaults example for sd-cpp models does not include the new upscale_model option.
  • Where: docs/guide/configuration/custom-models.md
  • Why: Documentation for a new or changed feature belongs in the same PR as the code.

Scope

minor — The PR extends the existing sd-cpp recipe with a new optional upscale_model parameter and hooks it into three existing image endpoints. It does not introduce a new subsystem or a new user-facing surface — the endpoints already existed and the upscaler is opt-in via a new GUI dropdown within an existing modal.

Suggested reviewers

• danielholanda (GUI) — The PR adds a new dropdown control to ModelOptionsModal, a GUI component under the GUI maintainer area.
• kenvandine (Image AI) — The PR adds backend upscaling logic for sd-cpp image endpoints, which falls under the Image AI maintainer area.

Reviewed at head 5707574 by repo-manager pr-review. Regenerate with repo-manager review-pr 2842.

[jeremyfowers]

@NineBallo please see TODOs in the comment above

@ramkrishna2910 I think you should review

[jeremyfowers]

Also @NineBallo please be aware that we have a feature freeze on gui2, and the gui changes will need to go into GUI3. If you can remove the gui features from this PR and rebase them into gui3 in a future PR that would be great.

[jeremyfowers]

see comment

[NineBallo]

@jeremyfowers Oops totally forgot about the docs! Some of these pr's won't be directly usable in that case but I'll strip ui changes out and open up seperate prs.

GUI PR questions:

1. Should I wait or target gui3-staging?
2. Do you want a pr for each, or one for all of my sd changes? (Figure all but just to confirm)

[jeremyfowers]

@jeremyfowers Oops totally forgot about the docs! Some of these pr's won't be directly usable in that case but I'll strip ui changes out and open up seperate prs.

GUI PR questions:

1. Should I wait or target gui3-staging?

Wait for gui3 to merge to main first

2. Do you want a pr for each, or one for all of my sd changes? (Figure all but just to confirm)

@kpoineal - heads up there's 3 PRs adding some minor UI elements to the image gen UI.

addressed in comments

[bconsolvo]

Thanks for picking this up @NineBallo — the gap in #2836 is real, and I agree the do_upscale() extraction is a nice cleanup regardless of where we land on the rest.

Some context that's relevant here: an earlier iteration of the upscaling work did have exactly this design — an sd-cpp_upscale_model recipe option that applied automatically as post-processing. I removed it in favor of the explicit two-step /images/upscale endpoint. The reason for the split was that upscaling shells out to sd-cli as a separate process, and that startup cost is significant. Making it a distinct request meant the UI could show accurate "Generating..." then "Upscaling..." stages with real timing, and could display the original and upscaled images side by side. Folding it back into /images/generations doesn't remove that cost — it hides it as unexplained latency on an endpoint the caller thinks is just doing diffusion.

Concerns I'd like to talk through before this lands:

1. Response contract. A client that requests 512x512 now gets back 2048x2048 with nothing in the response indicating why. That's surprising for an OpenAI-compatible endpoint, and it can break callers that size their UI or downstream processing off the request parameters.

2. The original is unrecoverable. apply_upscale_if_configured() overwrites data[0].b64_json in place, so there's no way to get the pre-upscale image back. That's the capability the two-step flow was specifically built to preserve.

3. No per-request control. The setting is sticky per model, so a caller can't opt out for a single request. Would an optional request-level parameter (defaulting to the model-level setting) cover your use case? That would keep the "just works for external clients" property while leaving an escape hatch.

4. Error handling looks incorrect. do_upscale() signals failure by returning an empty string, and the callers use res.status == 0 / res.status != 0 to decide whether an error response was already written. cpp-httplib initializes Response::status to -1, not 0. If that's right, two things follow:

   • In apply_upscale_if_configured(), a failed upscale_via_cli() returns res.status != 0 → true, so the endpoint silently returns the un-upscaled image as if nothing went wrong.
   • In handle_image_upscale(), the if (res.status == 0) guard never fires, so the existing "ESRGAN upscale failed" 500 is no longer returned. That's a regression on the current endpoint.

   Worth verifying against the vendored httplib header, but I think both paths need an explicit success/failure signal rather than inferring it from res.status.

5. The --backend fix. Can you share the repro? The current code already selects the per-backend sd-cli build via get_backend_binary_path(), so I'd like to understand what was still falling back to CPU before we add a second mechanism on top.

I want us to make that tradeoff on purpose, and to not lose the progress reporting and side-by-side comparison in the process.

[NineBallo]

@bconsolvo

The goal of this is to be optional, opt-in, and per-model, only for external calls. So you can still have the same flow in the webui chat if you'd like; you could still generate an image and call /images/upscale if you'd like; but you can also now use upscaling for chat apps that don't call /images/upscale (most of em). Given that this project is targeting self-hosters I think this makes a lot of sense, there is no case where the latency is unexplained because they set it up. People with true multi-step image workflows will be using comfyui or something anyway, this directly targets the main usecase for this endpoint. Giving users the choice to get higher quality images in generic chat apps without the exponential cost of generating the base at a higher resolution (especially with consumer hardware).

1. Given this would be opt-in I think it's the right path. I see this as an improvement for chat workflows where you might want to iterate on an image. Chat seems to be the target for lemonade and a lot of them don't have a native /image/upscale pathway. The alternative would be to track the upscaling ratio of each model (currently only x4) then generate at res/4. I don't like this as much, because if you called the endpoint wanting a 512x512 image, you want at least the quality of a 512x512 image. Currently we preserve the quality, though that could be switched to resolution. I'm open to discussion there, and do see that as a viable solution if you feel strongly on it.

2. The idea is to preserve the 2 step workflow, this just adds the option to combine them. You can select no upscaler for that sd model (leave it defaulted), then run the /images/upscale as you normally would if that's what you want. This just makes it possible to have a basic image gen pipeline for external chat applications like openwebui that don't expect any of that.

3. Honestly I just noticed the huge gap in general usability for "normal" chat applications. I don't generate a ton of images but it's occasionally nice to be able to make a quick edit to an image here or there, and this is very helpful there. I like the models being setup like presets though, similar to how we have the llm side is handled with per-model settings. We could add an optional setting: upscale=off to disable but I think it's better to use the proposed solution in 1 instead of that.

4. This was a bit rushed cause I've been real busy IRL. The error handling could totally be wrong I'll take a look when I get a chance lol.

5. I spun this all up in an lxc and messed up my gpu passthrough. Found that without --backend it would silently fall back to the cpu if it didn't see a gpu. Given that I told it to be on the GPU I prefer it erroring to silently continuing and adding like 2 mins to the latency. though I did change a few things between tests so I'll go back and re-test if it's needed. It was a mangled setup and a normal user shouldn't have that problem so it honestly may be.

---

Next Steps:

1. I'll run a QA sweep of the old pipeline/workflow since that is a significant concern and ensure everything still works. 2.Fix the error handling.
2. Review the --backend setting.
3. Just gonna wait for how we want to tackle the endpoint compliance

[ramkrishna2910]

Thanks @NineBallo and @bconsolvo — good discussion. Here's where I'd like to land this.

Direction: accept the per-model opt-in, with conditions. The gap is real — OpenAI-compatible clients only call /images/generations and can't discover /images/upscale, so a server-side, operator-configured option is the right shape for external callers. But it must be a convenience layer over the existing flow, not a replacement:

1. /images/upscale stays the primitive. The webui keeps the two-step flow (staged progress, side-by-side, original preserved). Both paths share the do_upscale() extraction.
2. Per-request override (bconsolvo's Support for FIM completions #3): add an optional request-level parameter defaulting to the model-level setting, so a caller can opt out (or in) per request.
3. Response transparency (Files moves and removals for v7 #1): when auto-upscale fires, the response must say so — an extension field on the image object (e.g. "upscaled": true plus the final dimensions) so callers sizing UI off request params have something to check. Surprising 2048² from a 512² request with no signal isn't acceptable for an OpenAI-compatible surface.
4. Error handling must be fixed (Please support non-english system language #4): confirmed — our vendored cpp-httplib initializes Response::status to -1, so the status == 0 checks never fire. As written, a failed upscale silently returns the un-upscaled image on /images/generations, and /images/upscale loses its existing "ESRGAN upscale failed" 500 — that's a regression. Use an explicit success/failure signal from do_upscale() rather than inferring from res.status.
5. On Release Lemonade v7 #2 (original unrecoverable): acceptable on the auto path — it's opt-in per model, and callers who need the original keep the two-step endpoint. Worth one line in the docs making that tradeoff explicit.
6. --backend fix: please share the repro before we add a second selection mechanism on top of get_backend_binary_path() — if it's not reproducible, drop that piece from this PR.

Process items before this is approvable:

• GUI changes must come out of this PR. We're freezing gui2 feature additions — GUI3 is landing shortly, so please strip the ModelOptionsModal changes and resubmit them against GUI3 after it merges to main (per @jeremyfowers above). The recipe option itself is still settable via /v1/load / config, so the server-side feature stands alone fine.
• Address the two doc gaps flagged in the pre-review (including upscale_model in the sd-cpp parameter table for /v1/load).
• Rebase — the branch is behind main.

Happy to re-review once those are in.

[NineBallo]

@ramkrishna2910 Fixed some stuff up, if you could take another peek it'd be appreciated :)

Two notes:

• If upscaling fails it will return the original un-upscaled image... Ideally this doesn't happen but figured this path was better then just outright burning whatever work had been done.

• I made the skip_upscale opt-out only. I think an opt-in path is better handled for a future PR where we un-hide upscaling models from the UI and let users manage what ones are downloaded.

[ramkrishna2910]

Thanks for the rework @NineBallo — I went back through my August 3 list item by item, and substantively all six are met. Recording that explicitly so it doesn't get relitigated:

1. /images/upscale stays the primitive, both paths share do_upscale() — done.
2. Per-request override — done via skip_upscale. It's opt-out only rather than the two-way override I described, which I'm fine with: opting in per request without a configured model has no sensible target.
3. Response transparency — implemented (upscaled / width / height), with one defect below.
4. Error handling — done properly. do_upscale() returns std::optional instead of inferring from res.status, which sidesteps the vendored cpp-httplib -1 initialization entirely, and /images/upscale keeps its "ESRGAN upscale failed" 500 via the non-null res path.
5. Docs line on the original being unrecoverable — present, verbatim, in the upscale_model row.
6. --backend — dropped, as asked.

Process items: GUI changes are out, and the doc gaps are closed — upscale_model is in the /v1/load table in docs/api/lemonade.md, skip_upscale is documented on all three endpoints, and there's a new response-format section. The documentation on this PR is genuinely good.

Three things still blocking.

1. CI has never run on this branch.

There are zero check runs against the head commit. This is a cross-repo PR, so the workflows need a maintainer to approve them — I'm kicking that off now, so this one isn't on you. Flagging it because nothing here has been verified by CI yet, and the branch is also 3 commits behind main and will need an update before it can merge.

2. The upscaled flag is coupled to PNG header parsing.

In apply_upscale_if_configured, the image is replaced first, then:

item["b64_json"] = upscaled.value();
std::string raw_header = utils::JsonUtils::base64_decode(upscaled.value().substr(0, 32));
if (auto dims = lemon::utils::get_png_dimensions(raw_header)) {
    item["upscaled"] = true;
    item["width"]  = std::get<0>(*dims);
    item["height"] = std::get<1>(*dims);
}

upscaled is only set when get_png_dimensions succeeds — but the substitution already happened unconditionally. Any output that isn't a PNG with a well-formed IHDR (or a truncated first chunk) yields an upscaled image carrying no signal at all, which is exactly the "surprising 2048² from a 512² request with no way to detect it" case that transparency requirement existed to prevent. Your own docs make it load-bearing: "Callers should check this field to determine actual image dimensions."

This is latent rather than live today, since both upscaler CLIs emit PNG — but it's a one-line fix and the contract is already written down. Set upscaled unconditionally when the image is replaced, and attach width/height only when the header parses.

3. do_upscale dispatches on the label alone, which breaks once #3327 lands.

The gate is has_label(info.labels, "upscaling"), after which the code hardcodes try_get_spec_for_recipe("sd-cpp") and calls SDServer::upscale_via_cli. Nothing checks the model's actual recipe.

#3327 (in review now) adds five upscalers with recipe: "thenoise", all carrying the upscaling label and all marked suggested: true. Once both land, a user setting upscale_model: "LSDIR-4x-TheNoise" passes your label check, and a TheNoise .safetensors gets handed to sd-cli. It fails — and on the auto path the failure is swallowed by design, so the caller gets an un-upscaled image, no upscaled field, and only a server-side WARNING. Silent wrong behavior from a setting the UI actively suggests.

Your docs already state the requirement ("must be an sd-cpp recipe model with the upscaling label"); the code just doesn't enforce it. #3327 adds recipe dispatch to handle_image_upscale for precisely this reason, so I'd like that PR to land first and this one to route do_upscale through the same dispatch — which turns this from "add a recipe guard" into "reuse what's already there," and gets TheNoise upscalers working on the auto path for free. If #3327 slips, a minimal recipe check with a clear error is fine as a stopgap.

One housekeeping item: the PR description is now stale. It lists ModelOptionsModal.tsx, recipeOptionsConfig.ts, and sdcpp_server.cpp in its file table, none of which are still in the diff, and it doesn't mention the new image_sniff.h. Since the description becomes the squashed commit message, please refresh it to match what actually ships.

Happy to re-review as soon as CI is green and 2 and 3 are addressed — the substance is close.

[bitgamma]

to better coordinate, I'd like add a few more notes about upscaling with TheNoise, since this could be relevant to this PR, especially in trying to keep naming/behavior consistent (or at least non-conflicting) between backends.

With #3327 the same mechanism used by sdcpp (cli-based upscaling) will be available (as already mentioned).

Additionally, TheNoise already supports (also through lemonade since request parameters are passed down as-is) the generate-then-upscale flow without any additional steps/calls, just by passing specific parameters. This breaks down as:

1. A high-quality 2x upscale without any additional model by either passing upscale=True or upscale_factor=2.0 to the generation request. This does latent upscaling via SesquiLSR + 1 step refine using the generation model at the final target resolution, which yields a much higher quality than ESRGAN.

2. An ESRGAN upscale which can be chained to the refiner step by passing pixel_upscaler=esrgan_model_name and an upscale_factor > 2. This brings the total range up to 8x when using a 4x-ESRGAN and up to 4x when using a 2x-ESRGAN (which introduces less artifacts btw). The model file will be searched in the directory configured via thenoise.upscaler_dir, the .safetensors extension is added automatically

[NineBallo]

@bitgamma
Most of the naming on my pr is going to need to change or it wont make much sense. What do you think about this?

Current idea is:

• Migrate my skip_upscale to !upscale in the request params. Having both is contradictory. Then hardcode a default x2 esrgan where it's not pre-defined on the sd side. I don't love the backend difference of latent vs pixel on the same param but that's the best middleground we have under this implementation.
• I'll map pixel_upscaler on the sd side to hit the respective upscale model through that.

This does leave a gap where: upscale on theNoise paired with a pixel upscale chains; whereas the sd side wont. Given they are on a unified api I think we should figure something out.

---

IMO an ideal solution would be:

upscale -> used a a global request level toggle to either skip all upscaling steps (latent or pixel) or hit a default pixel upscaler (2x esrgan). It will have worse quality on theNoise's side but then they are at least unified.

refine/prescale/preupscale/latent-upscale -> used as a "theNoise" specific path to hit the latent upscaler first.

pixel_upscaler -> model picker, so any request can hit any valid upscale model on either backend.

[bitgamma]

@NineBallo maybe we can simplify it further:

• a 'refine' boolean that is only supported on TheNoise which simply adds "upscale=true" to the request
• a 'pixel_upscaler' string which is the model name of an esrgan model configured in lemonade. This will work exactly the same for TheNoise and sdcpp as originally intended by this PR.

either of these options can be set independent of each other, so the "stacking" becomes an explicit user choice and not cause a mismatch between the two.

In addition, a skip_implicit_upscaling request parameter will be treated for both backends as an instruction to not apply any configured upscaling logic to this request. This parameter will not be forwarded down to the backend but will be handled/filtered by lemonade.

WDYT?

[NineBallo]

@bitgamma
Awesome, thanks for being quick I can work with that.

Only thing is skip_implicit_upscaling is real long, and would skip implicit and explicit upscaling anyways. Thinking we keep the 'skip_upscaling' naming already on this branch since that cleared already.

Any objections or can I send it like this?

[bitgamma]

@NineBallo I won't strongly oppose to the naming, but technically with TheNoise you can send all upscale parameters directly in the request (this doesn't need to change and works even today since parameters are passed through as-is) so 'skip_upscaler' is a bit of a misnomer (but something I can live with). Maybe 'no_implict_upscaler' is a tiny bit shorter?

I'll leave this up to you though, as long as functionality matches I am fine with it.

[NineBallo]

@bitgamma

Last ping for now I promise haha

Since this would strip all upscaling on the sd side, would it make sense to just catch&drop any upscaling params when it's set for theNoise? Then we match behavior on both. Only concern is maintenance burden depending on how many future params theNoise is planning to add.

[bitgamma]

@NineBallo no, let's not strip params. That's why I wanted to add "implicit" to the name, because all explicit parameters need to be left alone.

The behavior still matches on both, no-Lemonade-driven upscale, but what the backend does internally is not our concern

[NineBallo]

@bitgamma
Implemented how we described, what do you think about this?

[bitgamma]

this doesn't match what I think we agreed on. I made my comments on the documentation items but of course the corresponding code needs to be changed as well.

[bitgamma]

refine and pixel_upscaler should only be at recipe options, not per-request. At least that's how I understood it. The entire goal was to enable upscaling for openai-compatible endpoints that aren't aware of these options, wasn't it?

the way you implemented it intentionally breaks clients that want to take advantage of TheNoise's embedded handling for no reason at all.

upscale should be just passed through instead of just being deleted. It has never been defined before so there is no "backwards compatibility" to speak of

If you really want to to make the pixel_upscaler overridable per-request than please rename it to something else (also in the recipe) so that it doesn't clash with TheNoise naming. refine is fine as it doesn't clash

[NineBallo]

Yeah that's fair.

A better solution imo is to just pass through the option in the initial generation when TheNoise is used for both gen/upscaling. Though for now I'll go back to the old upscale_model param I was previously using. Trying to unify the api's this much is out of scope and I want to get the bulk of this cleared.

EDIT: Having two identical api params is bugging me, sketching up the solution that preserves both here.

[NineBallo]

@bitgamma Do you hate this logic for pixel_upscaler (sd.cpp@5267)? If so I'll just revert to having upscaling_model and pixel_upscaler but it seems redundant since one is a subset of the other. Idea is that for some settings if we don't have a known recipe for another backend then it falls through to the requested one. This ideally gives us the best of both worlds.

(this has been rather loosely tested, want to check if you like the direction before I do a full sweep)

[bitgamma]

@NineBallo this is even less unified, because in one case (sdcpp) the pixel_upscaler needs to be a model configured in lemonade, while for TheNoise it needs to be an upscaler that the user put in thenoise.upscaler_dir. I think this is too confusing.

The only thing that needed to be done for the API to be unified was to keep the original logic for both sdcpp and thenoise, just with the old upscaling_model name so that it doesn't clash with thenoise non-standard API. If someone want to use thenoise proprietary API they can still pass pixel_upscaler explicitly and lemonade doesn't need to know anything about it, since everything is just passed through untouched.

Basically, from the beginning, my only ask was NOT to use field names used by thenoise (to avoid conflict) and to apply the same logic to both backends since they now have a rather unified dispatch. The refine option is not 100% mandatory but I think it is nice to have (I see you already implemented it). There is no special handling needed for pixel_upscaler, literally not writing this string anywhere in the code is enough for supporting it!