[FEATURE] Expose API corresponding to `cocoindex setup` and `cocoindex drop`

[badmonster0]

Currently cocoindex setup and cocoindex drop is the only way to apply setup changes / drop setup for flows. There's a bunch of related functions under cocoindex/setup.py which CLI calls, but they have the following limitations:

• After applying setup change, data processing has to happen in a separate process. This is because AllSetupState and FlowContext needs to be rebuilt after setup change, and they're only built once and maintained as global in-process states now.

• setup work for all loaded flows, but don't work for a single flow. This was a simplification, given there're resource shared among different flows (i.e. the CocoIndex Metadata Table). This makes more sense for a CLI interface (which usually involves human interaction), but seems cumbersome if we expose an API over that.

Ideally, we may want to expose API to be used in this way:

flow1.setup()  # Apply setup updates for flow1. It's noop if already up-to-date.
flow1.drop()   # Drop the setup for flow1

# We also support doing it for all
cocoindex.setup_all()
cocoindex.drop_all()

and users should be able to run update immediately after setup within the same process:

flow1.setup()
flow1.update()

---

❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.

[badmonster0]

Note: After this, the update CLI can also provide a --setup option, which automatically push setup changes in one CLI run.

[lemorage]

If not mistaken, we need to update setup-related logic in Rust codebase to work for a single flow like drop, so we can use that in our exposed API.
Basically, after this issue, users can choose two options to setup/update/drop flows, either by CLI, or using exposed API directly in code (update already has its API).

cocoindex/src/py/mod.rs

Lines 412 to 429 in 9319618

	#[pyfunction]
	fn drop_setup(py: Python<'_>, flow_names: Vec<String>) -> PyResult<SetupStatus> {
	let lib_context = get_lib_context().into_py_result()?;
	let all_setup_states = lib_context
	.require_all_setup_states()
	.into_py_result()?
	.read()
	.unwrap();
	py.allow_threads(|| {
	get_runtime()
	.block_on(async {
	let setup_status = setup::drop_setup(flow_names, &all_setup_states).await?;
	anyhow::Ok(SetupStatus(setup_status))
	})
	.into_py_result()
	})
	}

[badmonster0]

@lemorage yes, but note that there's more change needs than making it support single flow: i.e. the first bullet in my original post. Specifically, FlowSetupState is needed in analyzer::analyze_flow(), so all flows need to be analyzed again and FlowContext needs to be updated.

So actually we can split this issue into two steps

• Support cocoindex.setup_all() and cocoindex.drop_all(). And allow users to call flow1.update() after cocoindex.setup_all().
• Support setup on a single flow, e.g. flow1.setup().