FXC-3769 inverse design seminar notebooks

[yaugenst-flex]

No description provided.

[github-actions]

Spell check passed successfully for 7 notebook(s).
Generated by GitHub Action run: https://github.com/flexcompute/tidy3d-notebooks/actions/runs/19333897371

[tylerflex]

These are a great walkthrough of the process for doing robust gradient-based design. It's very clear. I did a read through and had a few high level comments:

Do want to add these to the TOC / rst files so they show up in the docs? and do they deserve their own section maybe?

The sensitivity values probably should have units? or is it ok? maybe some mention because people might not immediately get that these are literally just units of [objective fn return units] / [parameter unit]

The robust Monte Carlo adjoint actually didnt seem to work that well from the plots?

I wonder if this is something we can improve with a different example? or trying different parameters? or is it just not expected to work that well?

When you quantify the robust vs nominal designs in section 5, do you sample the same fab variations used in the adjoint optimization? or is it a new sample? is there a risk of overfitting to the specific sample?

It seems the measurement calibration didn’t recover the actual structure? can this be explained with the same issue we saw in the example I did regarding the lack of invertibility? Should we comment on it?

[yaugenst-flex]

Do want to add these to the TOC / rst files so they show up in the docs? and do they deserve their own section maybe?

Yeah I didn't do it mainly because I didn't know how this ends up interacting with the web examples 😄 The notebooks do need to be in their own section I think, especially since they all share the same setup code. @daquinteroflex can I just add this to the TOC or do we have to take care of anything else?

The sensitivity values probably should have units? or is it ok? maybe some mention because people might not immediately get that these are literally just units of [objective fn return units] / [parameter unit]

Yeah good idea. In this case the parameters are all in um so that should be quite intuitive.

The robust Monte Carlo adjoint actually didnt seem to work that well from the plots?
I wonder if this is something we can improve with a different example? or trying different parameters? or is it just not expected to work that well?

Yes from just the spectra the difference is not obvious. One of them is that this is all single-frequency, so you really have to look at the difference only at the central wavelength, everything else kind of doesn't matter. There are a couple of other factors that also play into this, which I discuss at the end of 05_robust_comparison.ipynb - do you think that covers these concerns? In my opinion effect is large enough for an educational demo. You can get a stronger effect depending on the starting design, which depends on the seed in the Bayesian optimization. I could play around with that and try to get a bit luckier but I'm not super convinced of that approach 😄

When you quantify the robust vs nominal designs in section 5, do you sample the same fab variations used in the adjoint optimization? or is it a new sample? is there a risk of overfitting to the specific sample?

The robust adjoint here is actually not stochastic, we sample 3 fixed scenarios:

scenarios = {
    "nominal": params,
    "over": apply_bias(params, -ETCH_BIAS),
    "under": apply_bias(params, ETCH_BIAS),
}

So I don't think overfitting to the distribution is a concern. I did this to keep things simple and deterministic (and not to run too many simulations). I was planning on expanding on these considerations a bit if we publish this as some sort of tutorial/blog series.

It seems the measurement calibration didn’t recover the actual structure? can this be explained with the same issue we saw in the example I did regarding the lack of invertibility? Should we comment on it?

I think it works quite well? Maybe it's an issue with the presentation? The green line starts out at the nominal design and converges toward the orange line, which it is trying to fit. The remaining difference between green and orange here is on the order of the noise injected into the spectrum, so I think that's expected. I chose only a single degree of freedom here to illustrate the concept and avoid problems with nonuniqueness. Granted it's a bit contrived but my goal was just to demonstrate the concept.

[tylerflex]

You can get a stronger effect depending on the starting design, which depends on the seed in the Bayesian optimization. I could play around with that and try to get a bit luckier but I'm not super convinced of that approach 😄

I'm mostly wondering if tweaking the meta parameters of the robust optimization can improve things. eg more simulations, more variation in the sims.

Also I'm not suggesting removing this, because it's useful for educational aid but one question is whether the bayesian optimization part is useful here. vs just doing adjoint from the start? It uses a lot of sims.

A related question is I wonder if some of this (bayes, also Monte Carlo) would be good to do using the design plugin, just for more examples of that.

I think it works quite well? Maybe it's an issue with the presentation? The green line starts out at the nominal design and converges toward the orange line, which it is trying to fit.

It converges towards but it doesn't ever reach it. I noticed the same thing in my own study and ultimately ended up fixing it through changing parameters. I just anticipate this is a question people might have and it would probably be nicer if we could get it to find the true values?

The remaining difference between green and orange here is on the order of the noise injected into the spectrum, so I think that's expected.

Hm, but the noise injected is not systematic? to me the read of the plot is that it systematically underestimates the tooth sizes? I'm not sure I understand why but unless I'm missing something I'd expect the noise would just mean some green points would be above or below the orange, not all the same amount below them.

I chose only a single degree of freedom here to illustrate the concept and avoid problems with nonuniqueness. Granted it's a bit contrived but my goal was just to demonstrate the concept.

Yea this is fair, I just think it's possible we can augment these examples a bit to make them even stronger, but they demonstrate the concept very well.

[daquinteroflex]

Why don't you convert this to a rst file, add the toctree in here for each notebook and then it's relatively self contained to its own section in the docs. Then you just need to add this file to a notebook section toctree at whichever section hierarchy you want?

[daquinteroflex]

Basically you can move this file to docs/ but change the toctree to the notebook folder path in this directory

[daquinteroflex]

Actually, the bigger problem will be the way that this will get represented in the commercial website notebooks. That's why they're all at the same top level of this repo.

[yaugenst-flex]

could you double-check if it look alright now? here is the corresponding PR on tidy3d (without the submodule update yet): flexcompute/tidy3d#2998

[tylerflex]

Looks good! the only comment is that I think the measurement calibration notebook could use more description. I think it would be useful to state up front that we can also use gradient based optimization for this, just like we did for design but now on our fabrication corners. and that this similarly can scale to any number of parameters because we're also using adjoint.

It just seemed a little sparse on detail and I felt like someone skimming these might not get the point from the text as written.

[daquinteroflex]

Looks good!

[daquinteroflex]

Is this intended to be linked and shown to the user? How are you linking it to the main toctree?