fix(table): ducklake needs extra commit betwen 2 alter table rename

[AntoineGiraud]

cf. issue #646
otherwise, we had an error on dbt build -s my_table on 2nd run (drop / create table)
and we already did a backup on line 16

08:05:32    Runtime Error in model stg_commande (models/hypermarche/stg/stg_commande.sql)
  Binder Error: Cannot rename table stg_commande__dbt_tmp to stg_commande, since stg_commande__dbt_tmp already exists.

[jwills]

@AntoineGiraud ah so the idea is that in ducklake we can't do the rename op first and then backup later? I think the issue with this as-written is that it will break the backup behavior for any other table definition, so if this is what ducklake needs we need to think through how to special-case it

[AntoineGiraud]

Hi @jwills,

Thanks for your replay, indeed, my edit wasn't the right one !

i took more time with a colleague, to repeat / translate our dbt run -s my_model steps
In native DuckLake, (connected to a postgresql or duckdb for the catalog)

-----------------------------------------------------------
-- init ducklake with 1st table load (as in dbt-duckdb)
-----------------------------------------------------------
ATTACH 'ducklake:~/my_ducklake.ducklake' AS my_ducklake;
USE my_ducklake;
create schema if not exists stg;
show tables from my_ducklake.stg;

-----------------------------------------------------------
-- 1st time `dbt run -s coucou`
-----------------------------------------------------------
begin;
create table stg.coucou__dbt_tmp as select 'coucou' nom;
alter table stg.coucou__dbt_tmp rename to coucou;
commit;

Nxt up, 2nd time dbt run -s coucou failing with

Cannot rename table stg_commande__dbt_tmp to stg_commande, since stg_commande__dbt_tmp already exists.
It's like alter DDL won't apply until commit ... !

-----------------------------------------------------------
-- 2nd time `dbt run -s coucou`
-----------------------------------------------------------
begin transaction; 
create table stg.coucou__dbt_tmp as select 'coucou' nom;
alter table stg.coucou rename to coucou__dbt_backup ;
alter table stg.coucou__dbt_tmp rename to coucou;
commit;
drop table stg.coucou__dbt_backup ;

Adding commit between 2 alter rename works :

-----------------------------------------------------------
-- 2nd time `dbt run -s coucou`
-----------------------------------------------------------
begin TRANSACTION; 
create table stg.coucou__dbt_tmp as select 'coucou' nom;
alter table stg.coucou rename to coucou__dbt_backup;
commit;
------------
begin TRANSACTION; 
alter table stg.coucou__dbt_tmp rename to coucou;
commit;
drop table stg.coucou__dbt_backup ;

I'll start an issue on DuckLake side to see what there thinking on it ...
& addding the above comment in dbt-duckdb issue #646
& do the follow up :)

[AntoineGiraud]

Here is the update / status from ducklake side !

• This issue was spotted in ducklake#509
• and fixed in ducklake PR #510

I've installed the latest ducklake extension (force INSTALL ducklake FROM core_nightly;)
the fix shall ship with duckdb v1.4.2 expeted around the 10th of nov. (cf. release calendar)

My previous dbt failure went away :)

• 2nd dbt run -s my_model went fine
• dbt run -s my_incremental_model --full-refresh went also fine !

I'm canceling the PR & closing the linked issue