Merge branch 'doris' of https://github.com/xinge-ji/sqlmesh into doris

Author: xinge-ji

docs/integrations/dbt.md

@@ -219,7 +219,7 @@ This section describes how to adapt dbt's incremental models to run on sqlmesh a
 SQLMesh supports two approaches to implement [idempotent](../concepts/glossary.md#idempotency) incremental loads:
 
 * Using merge (with the sqlmesh [`INCREMENTAL_BY_UNIQUE_KEY` model kind](../concepts/models/model_kinds.md#incremental_by_unique_key))
-* Using insert-overwrite/delete+insert (with the sqlmesh [`INCREMENTAL_BY_TIME_RANGE` model kind](../concepts/models/model_kinds.md#incremental_by_time_range))
+* Using [`INCREMENTAL_BY_TIME_RANGE` model kind](../concepts/models/model_kinds.md#incremental_by_time_range)
 
 #### Incremental by unique key
 
@@ -233,28 +233,22 @@ To enable incremental_by_unique_key incrementality, the model configuration shou
 
 #### Incremental by time range
 
-To enable incremental_by_time_range incrementality, the model configuration should contain:
+To enable incremental_by_time_range incrementality, the model configuration must contain:
 
-* The `time_column` key with the model's time column field name as the value (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
 * The `materialized` key with value `'incremental'`
-* Either:
-    * The `incremental_strategy` key with value `'insert_overwrite'` or
-    * The `incremental_strategy` key with value `'delete+insert'`
-    * Note: in this context, these two strategies are synonyms. Regardless of which one is specified SQLMesh will use the [`best incremental strategy`](../concepts/models/model_kinds.md#materialization-strategy) for the target engine.
+* The `incremental_strategy` key with the value `incremental_by_time_range`
+* The `time_column` key with the model's time column field name as the value (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
 
 ### Incremental logic
 
-SQLMesh requires a new jinja block gated by `{% if sqlmesh_incremental is defined %}`. The new block should supersede the existing `{% if is_incremental() %}` block and contain the `WHERE` clause selecting the time interval.
+Unlike dbt incremental strategies, SQLMesh does not require the use of `is_incremental` jinja blocks to implement incremental logic. 
+Instead, SQLMesh provides predefined time macro variables that can be used in the model's SQL to filter data based on the time column.
 
 For example, the SQL `WHERE` clause with the "ds" column goes in a new jinja block gated by `{% if sqlmesh_incremental is defined %}` as follows:
 
 ```bash
-> {% if sqlmesh_incremental is defined %}
 >   WHERE
 >     ds BETWEEN '{{ start_ds }}' AND '{{ end_ds }}'
-> {% elif is_incremental() %}
->   ; < your existing is_incremental block >
-> {% endif %}
 ```
 
 `{{ start_ds }}` and `{{ end_ds }}` are the jinja equivalents of SQLMesh's `@start_ds` and `@end_ds` predefined time macro variables. See all [predefined time variables](../concepts/macros/macro_variables.md) available in jinja.
@@ -263,13 +257,11 @@ For example, the SQL `WHERE` clause with the "ds" column goes in a new jinja blo
 
 SQLMesh provides configuration parameters that enable control over how incremental computations occur. These parameters are set in the model's `config` block.
 
-The [`batch_size` parameter](../concepts/models/overview.md#batch_size) determines the maximum number of time intervals to run in a single job.
-
-The [`lookback` parameter](../concepts/models/overview.md#lookback) is used to capture late arriving data. It sets the number of units of late arriving data the model should expect and must be a positive integer.
+See [Incremental Model Properties](../concepts/models/overview.md#incremental-model-properties) for the full list of incremental model configuration parameters.
 
 **Note:** By default, all incremental dbt models are configured to be [forward-only](../concepts/plans.md#forward-only-plans). However, you can change this behavior by setting the `forward_only: false` setting either in the configuration of an individual model or globally for all models in the `dbt_project.yaml` file. The [forward-only](../concepts/plans.md#forward-only-plans) mode aligns more closely with the typical operation of dbt and therefore better meets user's expectations.
 
-Similarly, the [allow_partials](../concepts/models/overview.md#allow_partials) parameter is set to `true` by default for incremental dbt models unless the time column is specified, or the `allow_partials` parameter is explicitly set to `false` in the model configuration.
+Similarly, the [allow_partials](../concepts/models/overview.md#allow_partials) parameter is set to `true` by default unless the `allow_partials` parameter is explicitly set to `false` in the model configuration.
 
 #### on_schema_change
 

examples/sushi_dbt/models/customer_revenue_by_day.sql

@@ -1,7 +1,7 @@
 {{
   config(
     materialized='incremental',
-    incremental_strategy='delete+insert',
+    incremental_strategy='incremental_by_time_range',
     cluster_by=['ds'],
     time_column='ds',
   )

examples/sushi_dbt/models/waiter_as_customer_by_day.sql

@@ -1,7 +1,7 @@
 {{
   config(
     materialized='incremental',
-    incremental_strategy='delete+insert',
+    incremental_strategy='incremental_by_time_range',
     cluster_by=['ds'],
     time_column='ds',
   )

examples/sushi_dbt/models/waiter_revenue_by_day.sql

@@ -1,7 +1,7 @@
 {{
   config(
     materialized='incremental',
-    incremental_strategy='delete+insert',
+    incremental_strategy='incremental_by_time_range',
     cluster_by=['ds'],
     time_column='ds',
   )

examples/sushi_dbt/models/waiter_revenue_by_day_v1.sql

@@ -1,7 +1,7 @@
 {{
   config(
     materialized='incremental',
-    incremental_strategy='delete+insert',
+    incremental_strategy='incremental_by_time_range',
     cluster_by=['ds'],
     time_column='ds',
   )

pyproject.toml

@@ -24,7 +24,7 @@ dependencies = [
     "requests",
     "rich[jupyter]",
     "ruamel.yaml",
-    "sqlglot[rs]~=27.12.0",
+    "sqlglot[rs]~=27.13.2",
     "tenacity",
     "time-machine",
     "json-stream"
@@ -61,7 +61,10 @@ dev = [
     "dbt-bigquery",
     "dbt-core",
     "dbt-duckdb>=1.7.1",
-    "dbt-snowflake",
+    # version 1.10.1 of dbt-snowflake declares that it's compatible with dbt-adapters>=1.16 but in reality
+    # it depends on the 'InvalidCatalogIntegrationConfigError' class that only exists as of dbt-adapters==1.16.6
+    # so we exclude it to prevent failures and hope that upstream releases a new version with the correct constraint
+    "dbt-snowflake!=1.10.1",
     "dbt-athena-community",
     "dbt-clickhouse",
     "dbt-databricks",
@@ -108,7 +111,6 @@ duckdb = []
 fabric = ["pyodbc>=5.0.0"]
 gcppostgres = ["cloud-sql-python-connector[pg8000]>=1.8.0"]
 github = ["PyGithub>=2.6.0"]
-llm = ["langchain", "openai"]
 motherduck = ["duckdb>=1.2.0"]
 mssql = ["pymssql"]
 mssql-odbc = ["pyodbc>=5.0.0"]
@@ -214,7 +216,6 @@ module = [
     "pymssql.*",
     "pyodbc.*",
     "psycopg2.*",
-    "langchain.*",
     "pytest_lazyfixture.*",
     "dbt.adapters.*",
     "slack_sdk.*",

sqlmesh/cli/main.py

@@ -1079,50 +1079,6 @@ def rewrite(obj: Context, sql: str, read: str = "", write: str = "") -> None:
     )
 
 
-@cli.command("prompt")
-@click.argument("prompt")
-@click.option(
-    "-e",
-    "--evaluate",
-    is_flag=True,
-    help="Evaluate the generated SQL query and display the results.",
-)
-@click.option(
-    "-t",
-    "--temperature",
-    type=float,
-    help="Sampling temperature. 0.0 - precise and predictable, 0.5 - balanced, 1.0 - creative. Default: 0.7",
-    default=0.7,
-)
-@opt.verbose
-@click.pass_context
-@error_handler
-@cli_analytics
-def prompt(
-    ctx: click.Context,
-    prompt: str,
-    evaluate: bool,
-    temperature: float,
-    verbose: int,
-) -> None:
-    """Uses LLM to generate a SQL query from a prompt."""
-    from sqlmesh.integrations.llm import LLMIntegration
-
-    context = ctx.obj
-
-    llm_integration = LLMIntegration(
-        context.models.values(),
-        context.engine_adapter.dialect,
-        temperature=temperature,
-        verbosity=Verbosity(verbose),
-    )
-    query = llm_integration.query(prompt)
-
-    context.console.log_status_update(query)
-    if evaluate:
-        context.console.log_success(context.fetchdf(query))
-
-
 @cli.command("clean")
 @click.pass_obj
 @error_handler

sqlmesh/cli/project_init.py

@@ -114,6 +114,7 @@ def _gen_config(
   rules:
     - ambiguousorinvalidcolumn
     - invalidselectstarexpansion
+    - noambiguousprojections
 """,
         ProjectTemplate.DBT: f"""# --- Virtual Data Environment Mode ---
 # Enable Virtual Data Environments (VDE) for *development* environments.

sqlmesh/core/context.py

@@ -587,7 +587,8 @@ def state_sync(self) -> StateSync:
             self._state_sync = self._new_state_sync()
 
             if self._state_sync.get_versions(validate=False).schema_version == 0:
-                self._state_sync.migrate(default_catalog=self.default_catalog)
+                self.console.log_status_update("Initializing new project state...")
+                self._state_sync.migrate()
             self._state_sync.get_versions()
             self._state_sync = CachingStateSync(self._state_sync)  # type: ignore
         return self._state_sync
@@ -2355,7 +2356,6 @@ def migrate(self) -> None:
         self._load_materializations()
         try:
             self._new_state_sync().migrate(
-                default_catalog=self.default_catalog,
                 promoted_snapshots_only=self.config.migration.promoted_snapshots_only,
             )
         except Exception as e:
@@ -3130,7 +3130,9 @@ def lint_models(
         found_error = False
 
         model_list = (
-            list(self.get_model(model) for model in models) if models else self.models.values()
+            list(self.get_model(model, raise_if_missing=True) for model in models)
+            if models
+            else self.models.values()
         )
         all_violations = []
         for model in model_list:

sqlmesh/core/engine_adapter/fabric.py

@@ -121,10 +121,19 @@ def _create_catalog(self, catalog_name: exp.Identifier) -> None:
     def _drop_catalog(self, catalog_name: exp.Identifier) -> None:
         """Drop a catalog (warehouse) in Microsoft Fabric via REST API."""
         warehouse_name = catalog_name.sql(dialect=self.dialect, identify=False)
+        current_catalog = self.get_current_catalog()
 
         logger.info(f"Deleting Fabric warehouse: {warehouse_name}")
         self.api_client.delete_warehouse(warehouse_name)
 
+        if warehouse_name == current_catalog:
+            # Somewhere around 2025-09-08, Fabric started validating the "Database=" connection argument and throwing 'Authentication failed' if the database doesnt exist
+            # In addition, set_current_catalog() is implemented using a threadlocal variable "target_catalog"
+            # So, when we drop a warehouse, and there are still threads with "target_catalog" set to reference it, any operations on those threads
+            # that use an either use an existing connection pointing to this warehouse or trigger a new connection
+            # will fail with an 'Authentication Failed' error unless we close all connections here, which also clears all the threadlocal data
+            self.close()
+
     def set_current_catalog(self, catalog_name: str) -> None:
         """
         Set the current catalog for Microsoft Fabric connections.