Integration · dbt

    Your dbt project visible to whoever uses the figures.

    Datahub reads the artifacts of your dbt runs and shows models, lineage and test results. Read-only: Datahub does not run dbt and changes nothing in your project.

    erp.orderserp.customerscrm.dealsstg_ordersstg_customersfct_revenuedim_customerTest failsReportSourcesStagingMarts
    • Lineage

      What feeds what

      Per model you see what goes in and what depends on it, straight from your manifest.

    • Tests

      The outcome of every run

      Per test the status, the number of failures and the run time, recorded at every sync.

    • Read-only

      Your project stays yours

      Datahub does not run dbt, manage projects or schedule jobs.

    100+ hours
    In practice

    of finance work a week now runs automatically every night

    Hudig & Veder works with Databricks and dbt.

    Hudig & Veder
    What you see

    Answers to the questions every data team gets.

    • “Where does this figure in the report come from?”

    • “Which tests failed in the last run?”

    • “What depends on this source?”

    • “What does the SQL look like after compiling?”

    • “Which columns does this model have and how is it stored?”

    • “Which models carry this tag?”

    How it works

    Upload first, then sync.

    After your dbt run the artifacts go to a container in Azure Blob Storage. Then Datahub syncs, with a click or as the last step in your CI/CD. Datahub has no scheduler of its own, so your pipeline sets the pace.

    manifestRequiredcatalogOptionalrun_resultsOptionalBlobSync nowAfter every dbt run, from your CI/CD

    The manifest is enough to start.

    • What it reads

      Three files

      From the target folder of your dbt run.

      • manifest.json, required
      • catalog.json, optional
      • run_results.json, optional
    • What you see

      Every part of your project

      Models, sources, seeds and snapshots, each with its own page.

      • Schema, materialisation, tags and columns
      • Raw and compiled SQL
      • Upstream and downstream lineage
    • Not yet

      In the data catalog

      The dbt data is not yet in Datahub's data catalog. You browse it in the dbt explorer.

    Where you use it

    One connection, three answers.

    • Lineage

      Where a figure comes from

      Follow a model back to its sources and forward to what depends on it.

    • Quality

      Whether the figure can be trusted

      See per run which tests passed, warned or failed.

    • SQL

      How it is calculated

      Read the SQL as written and as it runs after compiling.

    How to connect dbt

    From dbt run to lineage in four steps.

    1. 01

      Upload the artifacts

      After your dbt run, put the files from target in an Azure Blob container, with the Azure CLI or your CI/CD.

    2. 02

      Set up sign-in

      One app registration with read rights on the container, once. The secret is kept in Azure Key Vault.

    3. 03

      Create the connection

      Enter the storage account, the container and an optional prefix, and test the connection.

    4. 04

      Sync

      Click Sync now, or call the API as the last step of your pipeline.

    Frequently asked questions

    What data teams ask before they connect.

    Next step

    Show where every figure comes from.

    An intro call maps how your dbt project runs and what it takes to connect it.