dlt 1.29.1 Release Notes

Core Library

• Add instance requirement spec for jobs (#4262 @tetelio) — Jobs can now declare runner instance requirements via require.instance (an open Dict[str, Any], e.g. {"size": "medium"}). The legacy machine key is deprecated — it still works and serializes, but now emits a DltDeprecationWarning pointing to instance. Fully backward compatible.
• Fix: use case-sensitive identifiers in sqlglot schema (#4269 @rudolfix) — sqlglot 30.13.0 began case-folding identifiers in more cases; case folding is now explicitly prevented when building the sqlglot schema.
• Fix: preserve REST paginator stop conditions on has_more=true (#4227 @mattfaltyn) — An API returning has_more=true can no longer re-enable a paginator that already hit a stop condition (maximum_offset/maximum_page, response total, or a missing cursor), preventing requests past a hard limit or without a valid cursor. Fixes #4225.
• Fix: handle non-scalar total in REST range paginator (#4186 @anxkhn) — A total_path resolving to a JSON object or array now raises the paginator's clear "not an integer" ValueError instead of escaping as an opaque TypeError.
• Fix: allow JWT auth without scopes (#4235 @mattfaltyn) — OAuthJWTAuth with the default scopes=None no longer raises TypeError; the scope claim is omitted when no scopes are configured. Fixes #4234.
• Fix: pass load id column name to remove_columns as a sequence (#4241 @chuenchen309) — add_dlt_load_id_column no longer drops RecordBatch columns whose name is a substring of _dlt_load_id (e.g. id, load), which previously produced silent NULLs or a crash. Fixes #4240.

Docs

• Replace dead playground link in README with the dlt + Hugging Face marimo notebook demo (#4267 @elviskahoro)
• Remove an outdated IP address from the docs (#4238 @VioletM)

Chores

• fix(ci): preserve dependency resolution across test jobs (#4264 @Travior) — Propagates matrix resolution settings via UV_SYNC_ARGS, prevents uv run from resyncing prepared CI environments (UV_NO_SYNC), bumps the GitPython minimum, and keeps the SQLGlot compatibility test green across versions.
• test(ci): rerun remote destinations on transient connection errors (#4243 @burnash) — Widens pytest-rerunfailures filters so transient ODBC/Azure SQL and Databricks connection errors get retried instead of failing the run, while benign assertion failures still fail fast.

New Contributors

• @chuenchen309 made their first contribution in #4241
• @anxkhn made their first contribution in #4186

dlt 1.29.0 Release Notes

Highlights

• ClickHouse staging-optimized replace strategy (#3927 @filipesilva) — ClickHouse now supports the staging-optimized replace strategy, performing atomic table swaps via EXCHANGE TABLES so a full replace is instantaneous and never leaves the destination in a half-loaded state.
• AWS Secret Manager config provider (#4162 @rudolfix) — dlt can now resolve config and secrets directly from AWS Secret Manager, matching the existing Google Secret Manager provider. Also stops re-creating the Google secrets client on every lookup.
• Explicit joins in Relation.join() (#3868 @burnash) — The dataset relation API gains explicit join support, giving you control over join conditions when composing relations.
• Cross-destination join compatibility (#3905 @Travior) — Destination configs now expose a physical_location() accessor and can_join_with rules, so dlt knows when relations that live on different destinations (or datasets) can actually be joined together.

Core Library

Features

• BigQuery atomic replace (#4130 @oystersuki) — New opt-in enable_atomic_replace config flag makes the truncate-and-insert strategy do a single-job, metadata-preserving WRITE_TRUNCATE_DATA load from GCS staging, giving BigQuery an atomic, transactional full replace.
• DuckDB SQLAlchemy destination (#4161 @rudolfix) — Fully tests and fixes the SQLAlchemy destination running on DuckDB.
• Expose Parquet compression codec (#4183 @Travior) — Configure the Parquet compression codec via DATA_WRITER__COMPRESSION or ParquetFormatConfiguration(compression=...).
• Cross-batch schema evolution in ArrowToParquetWriter (#3896 @AyushPatel101) — Safe Arrow type promotions (e.g. float32 → float64) now work across flush batches instead of crashing once data spans multiple batches.
• Additional Redshift COPY options (#4088 @timH6502) — New Redshift config to append extra options (e.g. FILLRECORD) to the generated COPY command for staged loads.
• Cross-destination join compatibility (#3905 @Travior) — See Highlights.
• Disable dlt→dlthub command proxying in workspace context (#4037 @rudolfix) — Instead of silently running a dlthub command, dlt now shows a note and points to the correct dlthub counterpart, avoiding misleading behavior.

Fixes

• Fix: Normalize crash on empty Arrow tables with add_dlt_id (#4187 @nizar-zerrad) — Adding _dlt_id to an empty Arrow table no longer fails with ArrowInvalid.
• Fix: correct table/resource metric aggregation (#4155 @tetelio) — Writer metrics are now aggregated correctly when jobs for the same table interleave across normalize workers or file rotation (previously undercounted items_count).
• Fix: Airflow parallel staging truncation (#4079 @DRACULA1729) — Auto-disables staging truncation for decompose="parallel" / parallel-isolated runs so concurrent tasks no longer truncate each other's staging files.
• Fix: move misplaced merge subqueries to post-qualify (#4169 @rudolfix) — Corrects merge SQL generation where pre-qualify subquery collapse could not work correctly.
• Fix: switch replace strategy correctly (#4164 @rudolfix)
• Fix: use truncate mechanism for empty "replace" tables (#4153 @rudolfix) — Empty "replace" resources are truncated via load-package state rather than empty files, avoiding hint side-effects.
• Fix: use empty files to truncate top-level tables (#4168 @rudolfix) — Partially reverts #4153 for optimized replace strategies, where truncation happens upfront rather than inside the job transaction.
• Fix: reuse job client when truncating staging dataset (#4207 @rudolfix)
• Fix: keep the pipeline selector when no pipeline is selected (#4096 @lis365b) — Dashboard keeps the pipeline dropdown visible and shows a clearer empty-state message when the selection is cleared.
• Fix: logging more frequent than log_period (#3962 @cbility) — Stops resetting the log timer on every counter update, so logs respect the configured log_period.
• Bump jsonpath-ng and improve test coverage (#4105 @Travior) — Upgrades to jsonpath-ng 1.8.0, dropping the archived ply transitive dependency flagged by security scanners (CVE).

Docs

• Add Tigris to S3-compatible storage options for the filesystem destination (#4166 @davidmyriel)
• Add secrets management page to dltHub docs (#4124 @ShreyasGS)
• Add "Orchestrate with dltHub" cookbook entry (#4077 @ShreyasGS)
• Add Slack and email notifications guide (#4081 @ShreyasGS)
• Add playground workspace page (#4150 @AstrakhantsevaAA)
• Add playground destination page (#4148 @AstrakhantsevaAA)
• Clarify soft deletes description for MSSQL source (#4165 @AstrakhantsevaAA)

Chores

• Add git commit-message rule, hook, and /commit skill (#4098 @lis365b) — Claude Code tooling to keep commit messages Conventional-Commit clean (no footers/emojis).
• Refactor ibis handover — create ibis backend on the Destination factory (#4170 @rudolfix)
• Relax dlthub-client deps (#4202 @rudolfix)
• Fix Windows path for griffe tests (#4104 @rudolfix)

New Contributors

• @filipesilva made their first contribution in #3927
• @cbility made their first contribution in #3962
• @DRACULA1729 made their first contribution in #4079
• @oystersuki made their first contribution in #4130
• @davidmyriel made their first contribution in #4166
• @nizar-zerrad made their first contribution in #4187

Core Library

This is a patch release that allows 1.28.x dlt to use future versions of dlthub-client

dlt 1.28.1 Release Notes

Python 3.9 EOL

1. Dropped Python 3.9 support (#4074 @Travior) — Python 3.9 reached end-of-life on 2025-10-31. dlt no longer tests against or advertises support for 3.9. The ruff/mypy target versions were bumped accordingly. Users still on 3.9 must upgrade to 3.10+ to use this release.

Highlights

• Dataset browser shown by default in the dltHub dashboard (#4068 @lis365b) — The dashboard now opens directly on the dataset browser (no entry page, no toggle), auto-selecting the most recently used pipeline. Empty-state copy was made generic so it reads correctly both locally and in the dltHub runtime. Resolves product_backlog#153.

Core Library

• Dataset browser by default in the dashboard (#4068 @lis365b) — see Highlights.
• Fix: connectorx temporal-column precision on newer connectorx (#3996 @tetelio) — Newer connectorx returns timestamps as timestamp[ns] and times as time64[ns] (via the arrow_stream return type) instead of date64 (ms). dlt now normalizes all connectorx temporal columns to microsecond precision (the renamed cast_connectorx_temporal_columns), truncating ns→us losslessly since source DBs carry no nanoseconds.
• Fix: ISO week/year in detect_datetime_format so week-date cursors round-trip (#4061 @yashs33244) — Week-date formats (YYYY-Www) were detected as %Y-W%W, which disagrees with ISO weeks at year boundaries. "2026-W01" parsed to 2025-12-29 and re-rendered as "2025-W52", silently corrupting saved incremental cursors. Now emits %G-W%V / %G-W%V-%u.
• Fix: remove 0x0 characters from postgres INSERT strings (#4086 @rudolfix) — Postgres escaping allowed null (0x0) characters through; they are now stripped in the escape function. INSERT statements are also split by \n only.
• Fix: metadata caching in sql_database (#4067 @rudolfix) — User-passed metadata is now correctly consulted for cached tables in both eager and deferred reflection. Fixes #4066.
• Fix: cap codex skill descriptions at 1024 chars on ai init (#4072 @lis365b) — Codex silently drops skills whose description frontmatter exceeds 1024 chars. The Codex install path now truncates the installed SKILL.md description to the cap (with a warning) so workbench skills like dlthub-router load correctly.

Docs

• Add EU static egress IPs for the dltHub platform (#4085 @tetelio)
• Add EU proxy/egress IPs for the dltHub platform (#4084 @tetelio)
• Recommend dlthub-init vs dlthub-start consistently in onboarding docs (#4080 @bjoaquinc)
• Clean up hub getting-started/installation wording and remove outdated commands (#4070 @ShreyasGS)
• Document the managed playground destination (#4075 @tetelio)

Chores

• Faster CI testing + Python 3.14 on CI (#3996 @tetelio) — Splits monolithic test_common.yml into 27 parallel focused jobs (3 per OS/Python combo) so wall-clock is roughly max(suite) instead of sum(suites), moves runners to blacksmith-* variants, and enables Python 3.14 in CI (running the full suite, with 3.14-gated deps snowflake-connector-python>=4.4.0 and cffi>=1.18). Also includes job/workflow renames; one functional reduction — lance_s3 non-essential tests now run only on full-suite triggers.
• Fix: pin airflow smoke-test installs with the airflow constraints file (#4069 @Travior) — Adds a script that fetches the latest airflow version and builds the matching constraint file, as airflow recommends for application installs.

New Contributors

• @yashs33244 made their first contribution in #4061

dlt 1.28.0 Release Notes

Breaking Changes

1. refresh="drop_data" on Delta and persistent-catalog Iceberg no longer frees storage (#4051 @rudolfix) — Truncation is now a transactional delete that keeps the table, schema, version history, and data files (retained for time travel until vacuum). Previously the files were deleted. This corrects prior erroneous behavior, but pipelines relying on drop_data to reclaim disk space will no longer see storage freed without an explicit vacuum.
2. replace now fully truncates empty and orphaned tables (#4010 @rudolfix) — Tables belonging to a replace resource that receive no data in a run (including nested tables, dynamic-name and variant tables) are now consistently truncated. Previously these tables could be left orphaned with stale rows surviving the reload, leaving the dataset in an inconsistent state. Pipelines that implicitly relied on that leftover data will now see those tables emptied.

Highlights

• Lance destination write optimizations (#4051 @rudolfix) — Namespace/session pooling shares one LanceNamespace + lance.Session across job clients; atomic single-commit-per-table writes uncommitted fragments in parallel and commits them in one version (Append/Overwrite/upsert). replace is now a single Overwrite commit so readers never see a partially-replaced table, and the namespace pool rebuilds handles on credential rotation to avoid ExpiredToken on long loads. Also fixes #3800 (Iceberg 409 Table already exists after drop_sources).
• Reliable replace / refresh truncation (#4010 @rudolfix) — replace resources now consistently truncate all participating tables even when a load carries no data (nested tables, dynamic names, variants included), and drop_data refresh truncates correctly on append and survives non-existent tables. refresh is now the recommended way to do a full refresh — the replace switch is deprecated. Also fixes #3998 and #4017.
• Configurable CSV encoding (#4045 @AstrakhantsevaAA) — New write_encoding option lets you choose the encoding of CSV files dlt writes (default utf-8), e.g. utf-8-sig for Excel BOM or latin-1/cp1252 for legacy importers. Set via [normalize.data_writer] write_encoding="latin-1".
• Refreshable cloud credentials for long-running loads (#4056 @tderk) — Default credential-chain credentials are now passed to external consumers (fsspec, rust crates, fileio) consistently and as refreshable where supported, instead of being frozen once. Fixes ExpiredToken failures on long-held connections (#4003).

Core Library

Features

• Prune duplicate deps from launcher groups (#4044 @rudolfix) — Deps already present in user requirements are eliminated; numpy/pandas removed from rows→arrow conversion and dashboard deps. Row conversion uses a pure-arrow fast path (up to ~2x faster) or a Python zip path as good as pandas.
• CDN marimo launcher (#4049 @tetelio) — Serve marimo frontend assets from jsDelivr CDN via configurable --asset-url. Launcher path resolution uses find_spec instead of import_module so notebooks aren't executed before marimo run / streamlit run (~1–1.5s faster port readiness).

Fixes

• Fix: duckdb refreshes credential_chain secrets to survive temp-token expiry (#4021 @0ywfe) — Adds REFRESH auto so long-held sql_client connections no longer die with ExpiredToken once temporary AWS tokens rotate. Fixes #3987.
• Bump duckdb to 1.5.3, ducklake to 1.0 (#4055 @rudolfix)
• Fix: Retry-After: 0 no longer triggers an immediate retry loop (#4043 @AstrakhantsevaAA) — Values ≤ 0 are treated as no actionable hint, letting tenacity's exponential backoff take over. Fixes #4036.
• Fix: clickhouse insert file quoting (#4018 @rudolfix) — Also bumps the driver version. Fixes #4014.
• Fix: incremental merge truncates destination on no-data runs (#4000 @burnash) — Port of the 1.27.2 hotfix (#3998) to devel with a regression test.
• Fix: databricks emits foreign key only when create_indexes is enabled (#4011 @burnash) — Avoids Unity Catalog UC_REFERENTIAL_CONSTRAINT_DOES_NOT_EXIST failures when the matching primary/unique key isn't created.
• Fix: DuckLake DuckDB-backed catalog attach incorrectly applied META_TYPE 'sqlite' (#3871 @Analect) — Splits the duckdb/sqlite branch so a duckdb:///catalog.duckdb catalog URI attaches cleanly instead of failing on PRAGMA journal_mode=WAL.
• Fix: mssql ingests parquet row-groups individually to bound ADBC driver memory (#3947 @wtfashwin) — Prevents OOM on parquet files larger than available memory. Closes #3915.

Docs

• Streamlit cookbook entry + Cookbook sidebar restructure into dlt / dlthub subcategories (#4050 @ShreyasGS)
• Langfuse cookbook (#4038 @zilto)
• Apply review feedback on dltHub docs (datasets, data-quality API) (#3975 @ShreyasGS)
• Remove memory warning from delta-rs docs now that the upstream issue is resolved (#4060 @tetelio)
• Restore orphaned content after docs restructuring + add orphan-detection lint step (#4028 @rudolfix)
• Restore orphaned docs on master (#4029 @rudolfix)
• High-contrast dltHub docs button (#4040 @zilto)
• Fix outdated dashboard description (commands + broken link) (#4053 @AstrakhantsevaAA)
• Escape __ in generated CLI docs to prevent markdown bold (#3993 @burnash)
• Escape characters to correct __deployment__.py file name in CLI docs (#3988 @nuetu)
• Fix typo in egress IPs list (#3992 @tetelio)
• Remove the Star Wars gif (#3994 @AstrakhantsevaAA)

Chores

• Retry flaky databricks and motherduck remote tests (#4046 @burnash) — Adds pytest-rerunfailures, scoped to just those two transient destinations via --only-rerun.
• Fix Arrow string-width assumptions in pandas 3 CI (#4025 @Travior) — Updates deltalake/filesystem reader tests for pandas 3's Arrow-backed string columns. Fixes #4024.
• Increase marimo cell re-render timeout in dashboard tests (#3991 @burnash) — Bumps 15s → 30s to stop test_multi_schema_selection flaking on slow CI.
• master → devel merge after hotfixes (#4019 @rudolfix)

New Contributors

• @0ywfe made their first contribution in #4021
• @Analect made their first contribution in #3871
• @wtfashwin made their first contribution in #3947
• @nuetu made their first contribution in #3988
• @tderk made their first contribution in #4056

Hotfix: fixes #3998 (merge with empty data after replace on incremental truncates the destination table).
Upgrade for anyone on 1.27.0/1.27.1.

dlt 1.27.0 Release Notes

Breaking Changes

1. workspace extra removed and dlthub command split out (#3929 @rudolfix) — The workspace extra is gone; users should install marimo, pyarrow, ibis, fastmcp, and other dependencies directly. Part of dev tooling moved to a plugin: dlt dashboard, dlt pipeline ... show, dlt pipeline ... mcp now require pip install dlt[hub]. dlt ai was moved to dlthub ai.

Highlights

• Native Polars DataFrame and LazyFrame support in resources (#3837 @AyushPatel101) — Polars DataFrames and LazyFrames can now be yielded directly from @dlt.resource without manual conversion. Auto-detected and routed through the Arrow extraction pipeline; LazyFrames are auto-collected before conversion.
• Databricks Zerobus loading (#3904 @jorritsandbrink) — New databricks_adapter(my_resource, insert_api="zerobus") enables loading via the Databricks Zerobus SDK into Delta tables. API mirrors bigquery_adapter; supported for append write disposition.
• Incremental filtering for dlt.Relation (#3889 @burnash) — Apply incremental filters directly on dlt.Relation, enabling efficient incremental reads from datasets.
• dlthub command split (#3929 @rudolfix) — Reorganizes _workspace modules and splits dev tooling (dashboard, mcp, ai) into a dedicated dlt[hub] plugin. See Breaking Changes above.

Core Library

• Native Polars DataFrame / LazyFrame support (#3837 @AyushPatel101) — See Highlights.
• Databricks Zerobus loading (#3904 @jorritsandbrink) — See Highlights.
• Incremental filtering for dlt.Relation (#3889 @burnash) — See Highlights.
• dlthub command split (#3929 @rudolfix) — See Highlights / Breaking Changes.
• lance destination REST Namespace support (experimental) (#3908 @jorritsandbrink) — Adds experimental REST Namespace support to the lance destination, currently only validated against an ephemeral in-memory Lance REST Namespace proxy.
• -y / --yes flag to bypass non-interactive prompts (#3910 @anuunchin) — New flag auto-accepts all confirm() prompts via a dedicated ALWAYS_CONFIRM flag in echo.py, without leaking into prompt() or text_input(). Resolves #3592.
• Control dashboard section opening from URL (#3686 @djudjuu) — Lets external tools / agents pre-open a specific section of the dashboard via URL, useful for guiding users to particular pipeline state.
• Fix: Pydantic annotation metadata handling across 2.10–2.13 (#3863 @Travior) — Restores discriminated RootModel handling and preserves root-level Annotated[...] metadata after Pydantic 2.13 moved the discriminator off root_field.annotation.
• Fix: correctly materialize empty tables for multi-table resources (#3901 @anuunchin) — Tracks empty tables (not resources) so multi empty-table materialization works correctly. Resolves #3840.
• Fix: scd2 unmapped insert for nested tables (#3812 @deschman) — Generates an explicit column list (excluding SCD2 from/to metadata) for the staging-to-destination insert, fixing DATATYPE_MISMATCH errors on Spark/Databricks ANSI mode when schemas drift. Fixes #3811.
• Fix(clickhouse): replica-safe merge delete temp tables (#3824 @RedZapdos123) — Detects the destination table engine and uses ReplicatedMergeTree / SharedMergeTree for merge delete temp tables, preventing duplicates on replicated/shared MergeTree deployments. Closes #3797.
• Fix(databricks): replace deprecated _user_agent_entry (#3935 @xodn348) — Switches DatabricksCredentials.to_connector_params() to the non-prefixed user_agent_entry, silencing the deprecation warning printed on every connection. Closes #3934.
• Fix: with_load_id breaking on non-root sibling branches (#3878 @Travior)
• Fix: schema naming convention respects max_length param (#3826 @aditypan) — Removes spurious length appending in schema naming when max_length is provided. Fixes #3816.
• Fix: propagate dataset_name to job metrics in load step (#3808 @rudolfix)

Docs

• Docs structure overhaul (#3884 @zilto) — Reorganizes the documentation site structure.
• OSS / paid split documentation (#3745 @VioletM)
• Cookbook: Arize Phoenix example (#3914 @zilto)
• Improve docs around join usage (#3902 @Travior)
• Misc docs fixes (#3909 @zilto)

Chores

• CI from fork (#3897 @anuunchin) — Splits CI so secret-requiring tests run via a new fork_tests_with_secrets.yml workflow gated by the ci from fork label and a maintainer approval in the fork-ci environment. Resolves #3850.
• Workspace tests setup fixes + CLI docs refresh (#3948 @rudolfix)
• Scope fruit_pipeline fixture to module to avoid duckdb lock race (#3932 @burnash) — Stabilizes the dashboard e2e suite.

New Contributors

• @xodn348 made their first contribution in #3935
• @RedZapdos123 made their first contribution in #3824
• @deschman made their first contribution in #3812

dlt 1.26.0 Release Notes

Breaking Changes

1. Incremental external scheduler now raises instead of silently warning (#3877 @rudolfix) — Untyped/non-coercible cursor values now raise JoinSchedulerError; missing intervals raise ExternalSchedulerNotAvailable. Resources with allow_external_schedulers=True that previously fell back to dlt state will now fail. This is a bugfix that corrects previously incorrect behavior.

Highlights

• dlt.Relation.join(...) (#3590 @Travior) — Adds a join() method on dlt.Relation based on the normalizer and table references, enabling fluent relational composition over datasets.
• Extended Snowflake query tags (#3759 @Travior) — Snowflake query tagging is extended beyond load jobs to broader dlt operations: storage setup, schema/state reads, schema updates, load execution, load completion, and table drops. TJobQueryTags is generalized to TQueryTags with a new operation field (with a compatibility export).
• Time interval context for incrementals (#3877 @rudolfix) — New dlt.current.interval() returns the active (start, end) interval or None, backed by an injectable TimeIntervalContext with optional allow_external_schedulers override and auto-detection from env vars / Airflow.

Core Library

• dlt.Relation.join(...) (#3590 @Travior) — see Highlights.
• Extended Snowflake query tags beyond load jobs (#3759 @Travior) — see Highlights.
• Time interval context for incrementals (#3877 @rudolfix) — see Highlights.
• Destination-aware incremental SQL filter (#3877 @rudolfix) — New dlt/extract/incremental/sql.py (to_sqlglot_filter) honors timestamp_precision, supports_tz_aware_datetime_in_cast, and sqlite quirks; works on bound and unbound incrementals.
• start_value persisted in incremental state (#3877 @rudolfix) — Only written when rows actually arrive, so it is no longer advanced silently on empty runs.
• uuid_to_string PyArrow fast path (#3877 @rudolfix) — Numpy-vectorized with a pure-Python fallback; pyarrow ≥ 24 arrow.uuid extension arrays are coerced to canonical strings, and UUID columns under pyarrow < 24 also take the fast path.
• Custom metrics emitted even when a resource produces no data (#3877 @rudolfix) — dlt.current.resource_metrics() counters are no longer dropped when every item is filtered out.
• TypedDict validator honors NotRequired[T] (#3877 @rudolfix) — Via __required_keys__.
• sqlglot "dremio" dialect literal (#3877 @rudolfix) — Added to TSqlGlotDialect.
• Opt-in naming-convention check in Schema.unify_schemas() (#3898 @burnash) — The naming-convention check in Schema.unify_schemas() is now opt-in; also drops the max_length tests workaround.
• Fix(clickhouse): correct metadata sort keys to avoid full scans (#3851 @anuunchin) — Resolves #3806 by correcting metadata sort keys so ClickHouse no longer performs full scans.
• Fix(pyarrow): preserve string encoding for UUID columns under pyarrow 24+ (#3894 @burnash) — Fixes #3893 so UUID columns keep their string encoding under pyarrow 24+.
• Fix: clearer error from dlt.attach() when pipeline cannot be restored (#3890 @bjoaquinc) — Rewrites CannotRestorePipelineException messages to name required inputs, show a concrete dlt.attach(...) example, and offer dlt.dataset() as a lighter alternative; suppresses a redundant inner exception in tracebacks.
• Fix: closed-pipe race in iter_std (#3877 @rudolfix) — Reader threads swallow ValueError/OSError and always close the queue.
• Fix: active profile shown in ConfigFieldMissingException (#3877 @rudolfix)

Docs

• Improved dashboard docs (#3484 @VioletM)
• Updated documentation for pg_replication (#3900 @dat-a-man) — Adds an "Alternative: CDC with Debezium" section linking to the Debezium + dlt demo.
• Snowflake marketplace listing link (#3888 @kaliole)
• Quality-checker fixes to pipeline.md (#3885 @ShreyasGS) — Tense, contractions, and grammar cleanups via Harper + Vale Google Developer Docs style.
• Remove playground page and associated CI (#3883 @zilto) — Streamlines the "Getting Started" section; interactive material remains on Google Colab and marimo molab.
• Pydantic Logfire export example (#3882 @zilto)

New Contributors

• @bjoaquinc made their first contribution in #3890

dlt 1.25.0 Release Notes

Breaking Changes

1. Multischema datasets (#3770 @burnash) — Datasets can now hold multiple schemas. The main benefit is to be able to see tables from all source in multi-source pipelines. This is a new default behavior.
   Users can pass a list of schemas to dataset() method and still go back to single-schema dataset by providing pipeline.default_schema when creating dataset.

Highlights

• lance destination (#3810 @jorritsandbrink) — New destination for the Lance table format with optional vector embedding generation via lancedb. Supports local storage and s3/az/gs, uses the Lance Directory Namespace V2 spec, and supports branching. Complements the existing lancedb destination (which targets LanceDB Cloud).
• Multischema datasets (#3770 @burnash) — See Breaking Changes above. Enables sidecar schemas (e.g. data-quality quarantine tables) to live alongside the primary schema in a single dataset.
• Improved progress and load metrics (#3768 @rudolfix) — Load metrics now persist across restarts, normalizer metrics are updated via update files, and the follow-up job graph is saved into the trace. Closes the long-standing #853.

Core Library

• lance destination (#3810 @jorritsandbrink) — See Highlights.
• Multischema datasets (#3770 @burnash) — See Highlights.
• Improved progress and load metrics (#3768 @rudolfix) — See Highlights.
• ducklake: metadata_schema ATTACH option (#3763 @sangwookWoo) — Adds metadata_schema to DuckLakeCredentials so the DuckLake metadata schema can be configured independently from ducklake_name.
• Fix: preserve credential chain in AWS credentials (#3798 @rudolfix) — Default credential mixing applied correctly, STS scoped to Databricks only. Closes #3115.
• Fix: replay state transitions after crash (#3767 @rudolfix) — Writes a pending state-transition marker right after the DB commit so an interrupted load no longer leaves the load package in an inconsistent state.
• Fix: create all eligible tables on staging dataset (#3765 @rudolfix) — Closes #2862.
• Fix: normalize pool workers skip __main__ in orchestrators (#3784 @rudolfix) — Closes #3586.
• Fix(clickhouse): lightweight DELETE for single-table merge (#3783 @rudolfix) — Removes the _dlt_id requirement when merging arrow tables without nested tables on ClickHouse.
• Fix(clickhouse): pass aws_session_token to staging s3() table function (#3769 @anuunchin) — Temporary AWS credentials now work for ClickHouse staging.
• Fix: avoid leaking PUA markers in nested fields (#3760 @serl) — Fixes Pydantic nested-model PUA-marker leak. Closes #3755.
• Fix: deepcopy paginator in child resource (#3779 @anuunchin) — Prevents paginator state corruption across child-resource invocations. Closes #3772.
• Fix: honor explicit non-utf8 encoding in filesystem read_csv (#3743 @biefan) — File is opened with the requested encoding so SFTP/paramiko stacks no longer pre-decode as UTF-8.
• Fix: don't filter out trace steps with exceptions (#3843 @anuunchin) — trace.asdict() now retains pipelines that fail in the sync step before extract.
• Fix: check duckdb version when installing lance extension (#3773 @zilto) — Handles the lance extension promotion to built-in in duckdb 1.5.
• Fix: transient Windows file-lock PermissionError in rename_tree (#3853 @burnash) — Resolves intermittent Windows CI failures during normalize→loaded rename.
• Fix: deprecation warnings across supported package versions (#3831 @anuunchin) — Closes #3785, #3807, #3787, #3794.

Docs

• Cookbook section (#3860 @zilto) — Tested examples moved to a dedicated top-level tab; dlt tab added for navigation back; UI cleanups.
• Same-domain docs button (#3859 @zilto) — Avoids full page reload when navigating.
• Explore-and-transform page (#3782 @hibajamal) — New page covering data-exploration and transformations workbench toolkits.
• Expand handover-to-other-toolkits section (#3737 @njaltran) — Expands data-exploration toolkit coverage in llm-native-workflow.md.
• Add EAI instructions (#3803 @kaliole)
• Update name to dlt Connector App (#3857 @kaliole) — Snowflake Native App docs renamed.
• Update source count to 8,000+ (#3830 @Pawansingh3889) — Closes #3761.
• Rename dltHub Basic tier to dltHub Pro (#3795 @elviskahoro)
• Fix outdated hf login command (#3781 @julien-c)

Chores

• Move mypy configs to pyproject.toml (#3780 @zilto) — Partially resolves #3346.
• Remove Python 3.9 from CI matrices (#3777 @zilto) — Python 3.9 reached EOL in October 2025. Resolves #3587, #3619.
• Increase Playwright timeout in e2e dashboard test (#3848 @burnash) — Matches the 15s timeout used elsewhere; reduces Windows CI flakiness.
• Silence Airflow 3.2 smoke-test log noise (#3835 @burnash) — Fixes #3834.

New Contributors

• @Pawansingh3889 made their first contribution in #3830
• @biefan made their first contribution in #3743
• @julien-c made their first contribution in #3781
• @serl made their first contribution in #3760
• @sangwookWoo made their first contribution in #3763
• @njaltran made their first contribution in #3737

dlt 1.24.0 Release Notes

Breaking Changes

1. Custom resource metrics now stored as tables (#3718 @rudolfix) — Incremental metrics in the trace are now represented in table format. This changes the location and structure of incremental metrics in the trace object.

Highlights

• Insert-only merge strategy (#3741 @rudolfix, based on #3372 by @OnAzart) — New insert-only merge strategy that performs idempotent, key-based appending: inserts records whose primary key doesn't exist in the destination while silently skipping duplicates. No updates or deletes. Supported across all SQL destinations, Delta Lake, and Iceberg.
• Parallelize all sources in Airflow (#3652 @JustinSobayo) — In parallel and parallel-isolated decompose modes, all source components now fan out concurrently from a shared start node. Previously the first source had to complete before others could begin, adding unnecessary wall-clock time. This release also adds basic Airflow 3 support with smoke tests.
• ClickHouse ReplacingMergeTree support (#3366 @prevostc) — New replacing_merge_tree table engine type for ClickHouse that enables native deduplication and soft deletes via dedup_sort and hard_delete column hints.
• Custom resource metrics as tables (#3718 @rudolfix) — Resources can now emit custom metrics that are stored as tables in the trace, enabling richer observability for pipelines.

Core Library

• Insert-only merge strategy (#3741 @rudolfix, based on #3372 by @OnAzart) — See Highlights.
• ClickHouse ReplacingMergeTree support (#3366 @prevostc) — See Highlights.
• Parallelize all sources in Airflow (#3652 @JustinSobayo) — See Highlights.
• Custom resource metrics as tables (#3718 @rudolfix) — See Highlights.
• Configurable Arrow table concatenation promote_options (#3701 @AyushPatel101) — arrow_concat_promote_options can now be set to "default" or "permissive" instead of the hardcoded "none", enabling automatic type promotion when yielding multiple Arrow tables with slightly different inferred types.
• Fix: CLI info/show fails on custom destinations (#3676 @anuunchin) — dlt pipeline info/show no longer crashes with UnknownDestinationModule on pipelines using @dlt.destination.
• Fix: Primary key assignment for incremental resources (#3679 @shnhdan) — Passing primary_key=() to Incremental to disable deduplication is no longer silently overwritten by the resource's own primary key.
• Fix: MotherDuck missing catalog validation (#3723 @YuF-9468) — Connection strings that omit the catalog/database name (e.g. bare md:) now raise a clear configuration error instead of a confusing connection failure.
• Fix: BigQuery infinite loop on internal error (#3732 @aditypan) — BigQuery jobs that encounter an internal error no longer cause an infinite retry loop.
• Fix: SCD2 column order mismatch in SQLAlchemy destinations (#3733 @anuunchin) — SCD2 validity column insert jobs now match the column order of existing tables in SQLAlchemy destinations.
• Fix: Timezone mapping in SQL timestamp datatype (#3735 @aditypan) — Timezone is now correctly set for timestamp/datetime column datatypes.

Docs

• Realistic closure-based data masking example (#3617 @veeceey) — Replaced the hardcoded example with a reusable mask_columns() function supporting all sql_database backends.
• Redirects for removed pages (#3688 @djudjuu)
• AI workbench license info (#3729 @lis365b)
• Minor doc fixes (#3734 @anuunchin)

Chores

• Bumps npm docs deps (#3728 @rudolfix)
• Switch lancedb example from Spotify to PodcastIndex (#3736 @Travior)
• Adds CLI docs check to docs CI workflow (#3739 @rudolfix)
• Moves render CLI docs command to a separate tool (#3740 @rudolfix)

New Contributors

• @prevostc made their first contribution in #3366
• @veeceey made their first contribution in #3617
• @JustinSobayo made their first contribution in #3652
• @shnhdan made their first contribution in #3679
• @YuF-9468 made their first contribution in #3723