Data catalogs and lineage: practical tips and trade-offs

You're three months into a data platform migration, and someone asks: "Can we use this revenue table from the finance team?" You have no idea who maintains it, where it pulls from, or whether it's been validated. That's when you realize data catalogs and lineage tracking aren't luxuries—they're survival tools. In this article, we'll explore data catalogs and lineage: practical tips and trade-offs that help teams actually know where data comes from.

Why data catalogs matter in production

A data catalog is a centralized inventory of your data assets. It answers three critical questions: What data do we have? Who owns it? Is it safe to use? Without one, teams waste time hunting for the right tables, duplicating work, and building pipelines on unstable foundations.

We've seen teams lose hours because three different pipelines pulled from three different "customer" tables—all slightly different, all untested. A catalog with proper metadata prevents that chaos. It's also your first line of defense against compliance issues: you can't govern data you can't find.

The tradeoff is real, though. Adding a catalog means upfront investment in tooling, metadata standards, and team discipline to keep it current. But the cost of not having one—rework, outages, compliance violations—typically exceeds the cost of building one.

Unity Catalog, Glue, or DataHub: choosing a platform

Three platforms dominate production environments: Unity Catalog (Databricks), AWS Glue Data Catalog, and the open-source DataHub. Each has different strengths.

Unity Catalog is tight integration with Databricks. If you're already on Databricks SQL Warehouse, Notebooks, or Delta Live Tables, it's the path of least resistance. It handles table ownership, access control, and basic lineage natively. You define a catalog, then schemas, then tables:

-- Unity Catalog structure
CREATE CATALOG finance;
CREATE SCHEMA finance.production;
CREATE TABLE finance.production.revenue (
  date DATE,
  amount DECIMAL(10, 2),
  owner STRING COMMENT "Data owner: finance-team@company.com"
);

-- Assign owner
ALTER TABLE finance.production.revenue SET OWNER finance_admins;

The downside: if you're multi-cloud or use non-Databricks tools (Snowflake, BigQuery, custom Python), Unity Catalog doesn't span those systems. You're locked into the Databricks ecosystem.

AWS Glue Data Catalog integrates with S3, Athena, Redshift, EMR, and Lambda. It's agnostic to your compute layer, making it a good fit for heterogeneous environments. You crawl S3 paths or connect to Redshift, and Glue auto-discovers schemas and builds a searchable catalog.

import boto3

glue = boto3.client('glue')

# Manually register a table
glue.create_table(
    CatalogId='123456789012',
    DatabaseName='finance',
    TableInput={
        'Name': 'revenue',
        'Owner': 'finance-team@company.com',
        'StorageDescriptor': {
            'Columns': [
                {'Name': 'date', 'Type': 'date'},
                {'Name': 'amount', 'Type': 'decimal(10,2)'},
            ],
            'Location': 's3://data-lake/finance/revenue/',
            'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
        },
    }
)

Glue's lineage tracking is basic—it doesn't automatically trace column-level dependencies across jobs. You'll need to manually annotate pipelines or use third-party tools. The upside is cost and flexibility; it works anywhere AWS services run.

DataHub is open-source and database-agnostic. It connects to Spark, Snowflake, Postgres, Kafka, and more. Ideal if you want to avoid vendor lock-in or run on-premises. DataHub also has the strongest lineage engine of the three—it automatically traces Spark and Airflow DAGs back to source tables.

from datahub.ingestion.run import run
from datahub.ingestion.source.snowflake import SnowflakeSource

# Ingest Snowflake metadata and lineage
config = {
    'source': {
        'type': 'snowflake',
        'config': {
            'account_id': 'xy12345.us-east-1',
            'warehouse': 'compute_wh',
            'username': 'datahub_user',
            'password': 'secret',
            'database': 'analytics',
        }
    },
    'sink': {
        'type': 'datahub_rest',
        'config': {
            'server': 'http://datahub:8080',
        }
    }
}

run(config)

The tradeoff: DataHub requires more operational overhead. You host and maintain it, write custom connectors, and debug ingestion pipelines. But you get full control and zero vendor dependency.

Track table owners and metadata

A catalog with no owners is a graveyard. Three months later, nobody remembers who maintains the dim_customer table, whether it's SLA'd, or if someone should be notified when it breaks.

Assign owners at the table level and tag contact channels. In Databricks, that's a OWNER clause. In Glue, it's a property. In DataHub, use custom properties or a custom field:

-- Databricks example
ALTER TABLE finance.revenue OWNER finance_data_team;

-- Annotate with contact info
COMMENT ON TABLE finance.revenue IS 'Daily revenue snapshot. Owner: finance-data-team@company.com. SLA: 6am UTC. Slack: #data-finance';

In Airflow, register table ownership in your DAG:

from airflow import DAG
from airflow.operators.spark_sql_operator import SparkSqlOperator
from datetime import datetime

dag = DAG(
    'revenue_etl',
    owner='finance-team',
    start_date=datetime(2024, 1, 1),
)

task = SparkSqlOperator(
    task_id='load_revenue',
    sql='INSERT INTO finance.revenue SELECT ...',
    dag=dag,
)

# Add custom metadata
task.doc_md = """
Revenue ETL pipeline.
Owner: finance-team@company.com
Slack channel: #data-revenue
Contact for issues: @john (on-call)
"""

Don't rely on GitHub or email threads as a source of truth. Your catalog is the single source. Update it when ownership changes, test that notifications work, and validate quarterly that contacts are still valid.

Column-level lineage: where does each field come from?

Table-level lineage tells you "revenue comes from the orders table." Column-level lineage tells you "revenue.amount = orders.total - orders.discounts." That's the difference between "I know this table exists" and "I can debug a calculation bug."

Tracing column lineage by hand is fragile. A SQL join breaks, an engineer forgets to update documentation, and your lineage graph is now wrong. Automated column lineage is better.

DataHub extracts column lineage from Spark DataFrames and Airflow DAGs automatically. If you run Spark SQL:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('revenue_etl').getOrCreate()

df_orders = spark.read.table('raw.orders')
df_discounts = spark.read.table('raw.discounts')

df_revenue = df_orders.join(df_discounts, 'order_id') \
    .selectExpr(
        'order_id',
        'total - discount_amount as amount',
        'order_date'
    )

df_revenue.write.mode('overwrite').saveAsTable('analytics.revenue')

DataHub will trace that analytics.revenue.amount depends on raw.orders.total and raw.discounts.discount_amount. Glue won't do this automatically; you'd need to write custom metadata during the job:

import json
from awsglue.context import GlueContext

glue_context = GlueContext(spark_session)

# Write lineage metadata to Glue
lineage = {
    'output_table': 'analytics.revenue',
    'columns': {
        'amount': {
            'sources': [
                {'table': 'raw.orders', 'column': 'total'},
                {'table': 'raw.discounts', 'column': 'discount_amount'},
            ],
            'transformation': 'total - discount_amount',
        }
    }
}

# Store in Glue job bookmarks or external metadata system
print(json.dumps(lineage, indent=2))

For Unity Catalog + Databricks, enable Lineage in the workspace settings. Databricks SQL and Notebooks will automatically track column dependencies. For legacy Hive or custom ETL, you'll need manual registration or a custom ingestion layer.

Column-level lineage pays off when a downstream report breaks. Instead of asking "which table?" you ask "which transformation?" and fix it in minutes, not hours.

Search for trusted datasets

A catalog without discovery is just metadata in a database. Teams should be able to search "customer" and immediately see which tables exist, who owns them, and which ones are production-tested.

Add trust signals to your catalog. Tag tables as "certified," "experimental," or "deprecated." Document quality metrics: what's the freshness SLA? How often does it break? How many downstream consumers depend on it?

In DataHub, add custom properties:

from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import GlobalTagsClass, TagAssociationClass

# Tag a table as certified
emitter = DatahubRestEmitter('http://datahub:8080')

tag = TagAssociationClass(tag='urn:li:tag:Certified')
tags = GlobalTagsClass(tags=[tag])

# Send tag via API (simplified)
emitter.emit_mce({
    'proposedSnapshot': {
        'com.linkedin.pegasus2avsc.generated.EntitySnapshot': {
            'datahubTable': {
                'urn': 'urn:li:table:(urn:li:dataPlatform:spark,analytics.revenue,prod)',
                'globalTags': tags,
            }
        }
    }
})

In Glue or Unity Catalog, use tags natively:

-- Unity Catalog
ALTER TABLE finance.revenue SET TAGS (
  'trustLevel' = 'certified',
  'refreshSLA' = '6am UTC daily',
  'slackChannel' = '#data-finance'
);

Train your team: "Use certified tables when possible. If an experimental table meets your needs, document the risk." Over time, this creates a self-reinforcing culture where high-quality, well-maintained datasets become the default.

Common mistakes

• Fire and forget: Build a catalog, populate it once, then let it rot. Lineage and ownership data becomes stale within weeks. Assign someone to review and update it quarterly, or integrate catalog updates into your deployment pipeline.
• Over-cataloging: Trying to catalog every intermediate table, log file, and cache. Start with production tables and key staging layers. Expand gradually. A catalog of thousands of irrelevant assets isn't better than a catalog of a hundred trusted ones.
• No governance enforcement: A catalog is voluntary without teeth. If you don't prevent unauthorized tables from being created, or don't enforce tagging, adoption stalls. Pair your catalog with access control and naming conventions.
• Ignoring lineage staleness: Automatic lineage is only good if you run jobs through the same system consistently. If some teams use Spark, others write custom Python, and others use dbt, you'll have blind spots. Standardize where possible; document where you can't.
• Treating lineage as read-only: Teams ignore the lineage graph if it's just informational. Make it actionable: alert when a table upstream breaks, show impact when someone deletes a column, block schema changes if consumers exist downstream.

Key takeaways

• Data catalogs and lineage aren't optional for mature teams. They save time, prevent compliance issues, and unblock debugging.
• Choose your platform based on your infrastructure: Unity Catalog for Databricks-only, Glue for AWS-heavy, DataHub for flexibility or multi-cloud.
• Assign table owners and keep contact information current. Include SLAs and Slack channels so people know who to reach when something breaks.
• Automate column-level lineage where possible (Spark, Airflow, dbt) to catch calculation bugs faster. Manual lineage doesn't scale.
• Tag tables with trust levels and keep the catalog actively maintained. A stale catalog is worse than no catalog.
• Pair your catalog with governance enforcement—access control, naming standards, schema change reviews—so adoption is mandatory, not voluntary.