Metadata-Version: 2.4
Name: vd-dlt
Version: 0.1.7
Summary: Core DLT ingestion framework for VibeData pipelines
Project-URL: Homepage, https://github.com/accelerate-data/vd-dlt-connectors
Project-URL: Repository, https://github.com/accelerate-data/vd-dlt-connectors
Author-email: VibeData <info@vibedata.dev>
License-Expression: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.9
Requires-Dist: pyyaml>=6.0
Provides-Extra: notion
Requires-Dist: vd-dlt-notion; extra == 'notion'
Provides-Extra: notion-schema
Requires-Dist: vd-dlt-notion-schema; extra == 'notion-schema'
Provides-Extra: pipeline
Requires-Dist: dlt[deltalake,filesystem]; extra == 'pipeline'
Requires-Dist: pyarrow>=17.0.0; extra == 'pipeline'
Provides-Extra: salesforce
Requires-Dist: vd-dlt-salesforce; extra == 'salesforce'
Provides-Extra: salesforce-schema
Requires-Dist: vd-dlt-salesforce-schema; extra == 'salesforce-schema'
Description-Content-Type: text/markdown

# vd-dlt

Core DLT ingestion framework for VibeData pipelines. Provides config resolution, credential management (Azure Key Vault), pipeline execution, and observability.

## Installation

```bash
# Core only
pip install vd-dlt

# With pipeline dependencies (dlt, pyarrow)
pip install vd-dlt[pipeline]

# With Notion connector
pip install vd-dlt[notion]

# With Notion schema (defaults, docs)
pip install vd-dlt[notion-schema]

# Everything for Notion
pip install vd-dlt[pipeline,notion,notion-schema]
```

## Quick Start

```python
from vd_dlt import PipelineRunner

runner = PipelineRunner(
    vault_url="https://my-vault.vault.azure.net/",
)
result = runner.run("my_source_name")
print(f"Loaded {result.total_rows_loaded} rows")
```

## Architecture

The framework uses a 4-level config hierarchy (most specific wins):

1. **Connector Defaults** - from connector schema package
2. **Source Config** - from source YAML (including `default_sync`)
3. **Group Config** - optional grouping within source
4. **Resource Config** - per-table overrides
