Update dependencies to make dlt-init-openapi compatible with dlt versions >= 0.5.1 (preferably >= 1.2.0)

[jordan-bulka-sh]

When using poetry, I run into dependency conflicts if I want to use the more recent stable versions of dlt. I would like to use the filesystem and Snowflake destinations with dlt-init-openapi and the rest_api source. This is preventing my team from adopting dlt.

Steps to reproduce:

My pyproject.toml file initially looks like this:

[tool.poetry.dependencies]
python = ">=3.11,<3.13"

Running poetry add dlt; poetry add dlt-init-openapi; results in the following dependency conflict:

Because no versions of dlt match >0.4.12,<0.4.13a0 || >0.4.13a0,<0.5.0
 and dlt (0.4.13a0) depends on dlt (0.4.13a0), dlt (>0.4.12,<0.5.0) requires dlt (0.4.13a0).
And because dlt (0.4.12) depends on dlt (0.4.12), dlt (>=0.4.12,<0.5.0) requires dlt (0.4.12 || 0.4.13a0).
Because no versions of dlt-init-openapi match >0.1.0,<0.2.0
 and dlt-init-openapi (0.1.0) depends on dlt (>=0.4.12,<0.5.0), dlt-init-openapi (>=0.1.0,<0.2.0) requires dlt (>=0.4.12,<0.5.0).
Thus, dlt-init-openapi (>=0.1.0,<0.2.0) requires dlt (0.4.12 || 0.4.13a0).
So, because <repo-name> depends on both dlt (^1.2.0) and dlt-init-openapi (^0.1.0), version solving failed.

After pinning the dlt version to 0.4.12, I can successfully add dlt-init-openapi as a dependency. However, if I try to initialize a simple pipeline using dlt init chess snowflake, the console warns me about my dlt installation needing to be updated:

WARNING: This pipeline requires a newer version of dlt than your installed version (0.4.12). Pipeline requires 'dlt>=0.5.1'

I receive a similar warning if I change the destination to filesystem or the source to rest_api.

However, if I try to pin the dlt version to 0.5.1 with poetry add dlt==0.5.1, I again get a dependency conflict:

Because no versions of dlt match >0.4.12,<0.4.13a0 || >0.4.13a0,<0.5.0
 and dlt (0.4.13a0) depends on dlt (0.4.13a0), dlt (>0.4.12,<0.5.0) requires dlt (0.4.13a0).
And because dlt (0.4.12) depends on dlt (0.4.12), dlt (>=0.4.12,<0.5.0) requires dlt (0.4.12 || 0.4.13a0).
Because no versions of dlt-init-openapi match >0.1.0,<0.2.0
 and dlt-init-openapi (0.1.0) depends on dlt (>=0.4.12,<0.5.0), dlt-init-openapi (>=0.1.0,<0.2.0) requires dlt (>=0.4.12,<0.5.0).
Thus, dlt-init-openapi (>=0.1.0,<0.2.0) requires dlt (0.4.12 || 0.4.13a0).
So, because <repo-name> depends on both dlt-init-openapi (^0.1.0) and dlt (0.5.1), version solving failed.

[joscha]

same when trying to upgrade the dependences with pip:

pip install --upgrade --force-reinstall -r requirements.txt

yields

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dlt-init-openapi 0.1.0 requires dlt[duckdb]<0.5.0,>=0.4.12, but you have dlt 1.4.0 which is incompatible.

Uninstalling the generator after using it works however:

pip uninstall dlt-init-openapi

and then pip install --upgrade --force-reinstall -r requirements.txt succeeds. Given that we don't need the generator anymore after executing it, that works for the upgrade, however it seems the generated code is also not compatible with newever versions of dlt, as the code afterwards fails with:

Traceback (most recent call last):
  File "/Users/joscha/dev/dlt-test/affinity_pipeline/affinity_pipeline.py", line 3, in <module>
    from affinity import affinity_source
  File "/Users/joscha/dev/dlt-test/affinity_pipeline/affinity/__init__.py", line 5, in <module>
    from rest_api import rest_api_source
  File "/Users/joscha/dev/dlt-test/affinity_pipeline/rest_api/__init__.py", line 484, in <module>
    _register_source(rest_api_source)
  File "/Users/joscha/dev/dlt-test/affinity_pipeline/rest_api/__init__.py", line 473, in _register_source
    from dlt.common.source import _SOURCES, SourceInfo
ModuleNotFoundError: No module named 'dlt.common.source'

so I think not only the dependency closure needs updating, but also the generator itself.

[joscha]

dlt.common.source seems to be gone >=1, so the only working version is:

dlt>=0.5.2, <1

[gregorywaynepower]

@sh-rp @rudolfix I wanted to check in to see if there were plans on updating this to be compatible with more recent versions of dlt?