Data governance basics: a practical guide for developers

Data governance basics: a practical guide for developers starts with a simple observation: as soon as your platform touches customer data, you're responsible for protecting it. Governance isn't just compliance theater—it's the infrastructure that keeps your data secure, your team accountable, and your business running. Let's walk through the practical systems we build into production pipelines every day.

Why governance matters in data platforms

Early in a data platform's life, governance feels optional. A few engineers, a small dataset, light regulatory pressure—it all runs on trust and muscle memory. But the moment you scale, share access across teams, or handle regulated data (PII, payment info, health records), you need explicit controls.

We've seen teams burn weeks debugging access violations because nobody documented who could query what. Others faced compliance audits unprepared, unable to prove data lineage or show audit trails. The cost of retrofitting governance is always higher than building it in from the start.

Access control: Table and column permissions

Access control is your first line of defense. It answers the question: who is allowed to read, write, or modify this data?

In production Spark pipelines, we typically implement access control at two levels: table-level (can you query the table at all?) and column-level (can you see this specific field?). Most teams use their data warehouse's native security model—Unity Catalog on Databricks, Ranger on Hadoop, or role-based access in Redshift.

Here's how you'd set up table-level permissions in Unity Catalog:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import sql

client = WorkspaceClient()

# Grant SELECT on a table to a group
client.grants.update(
    securable_type="TABLE",
    name="catalog.schema.customers",
    changes=[
        {
            "principal": "data_analysts",
            "add": ["SELECT"]
        }
    ]
)

# Revoke MODIFY from individual user
client.grants.update(
    securable_type="TABLE",
    name="catalog.schema.customers",
    changes=[
        {
            "principal": "intern_account",
            "remove": ["MODIFY"]
        }
    ]
)

Column-level access is trickier. Not all systems support it natively, so we often use dynamic views that filter columns based on the caller's role:

# In your data warehouse, create a view that filters columns by role
CREATE VIEW customers_safe AS
SELECT
    customer_id,
    email,
    CASE 
        WHEN current_user() IN ('finance_team', 'risk_team') 
        THEN credit_score 
        ELSE NULL 
    END AS credit_score,
    CASE 
        WHEN current_user() IN ('support_team', 'finance_team') 
        THEN phone_number 
        ELSE NULL 
    END AS phone_number,
    created_at
FROM customers_raw;

In practice, we assign users to groups (data_analysts, finance_team, support_team) and grant permissions to groups, not individuals. This scales much better than managing hundreds of individual access rules. Review access quarterly—people move teams, projects end, permissions accumulate.

PII handling: Mask or tokenize sensitive fields

Not everyone who can access a table needs to see raw personally identifiable information. Credit card numbers, social security numbers, email addresses—these need protection even after access control is in place.

We have two main approaches: masking (replace with redacted values) and tokenization (replace with reversible tokens).

Masking is simpler and suits analytics use cases:

from pyspark.sql.functions import concat, lit, substring

# For analytics, mask sensitive fields in the view layer
spark.sql("""
CREATE VIEW users_analytics AS
SELECT
    user_id,
    CONCAT('***', SUBSTRING(email, LENGTH(email) - 4, 5)) AS email_masked,
    CONCAT('****', SUBSTRING(phone, -4)) AS phone_masked,
    country,
    signup_date
FROM users_raw
""")

Tokenization is more complex but reversible—useful when you need to join tokenized data with raw data later:

import hashlib
from pyspark.sql.functions import udf, col
from pyspark.sql.types import StringType

# Simple token: hash the PII value with a secret key
# In production, use a proper tokenization service
def tokenize_email(email, secret_key="your-secret"):
    return hashlib.sha256(f"{email}{secret_key}".encode()).hexdigest()[:16]

tokenize_udf = udf(tokenize_email, StringType())

df_tokenized = df.withColumn("email_token", tokenize_udf(col("email")))
df_tokenized.drop("email").write.mode("overwrite").saveAsTable("users_tokenized")

In production, use a dedicated tokenization service (like AWS Data Key or a third-party provider) rather than implementing it yourself. The key insight: decide at pipeline design time which fields need masking or tokenization, and apply it once in your canonical data layer. Don't let raw PII scatter across different tables.

Retention: Delete data according to policy

Data governance isn't just about who can see what—it's also about not keeping data longer than you should. Regulations like GDPR require you to delete personal data when it's no longer needed. Beyond compliance, retention policies reduce storage costs and security risk.

Define a retention policy for each data asset: how long should it live? When should it be deleted or archived?

from pyspark.sql.functions import col, datediff, current_date
from datetime import timedelta

# Example: Delete user sessions older than 90 days
retention_days = 90

df_sessions = spark.table("sessions")

# Flag rows to delete
df_to_keep = df_sessions.filter(
    datediff(current_date(), col("session_date")) <= retention_days
)

# Overwrite table with retained data only
df_to_keep.write.mode("overwrite").saveAsTable("sessions")

print(f"Deleted sessions older than {retention_days} days")

For large tables, deletion can be expensive. A better pattern is immutable snapshots with versioning:

# Use Delta Lake's time-travel to manage retention
from datetime import datetime, timedelta

# Archive old versions before retention expires
retention_date = (datetime.now() - timedelta(days=90)).isoformat()

# Vacuum removes files older than the retention threshold
spark.sql(f"""
VACUUM table_name RETAIN {90} DAYS
""")

# This physically deletes data older than 90 days

Set up automated jobs to run retention policies monthly. Document your policies in a central registry so teams know what data is kept where and for how long.

Audit: Log who accessed what

Compliance audits require proof: who queried this table? When? What did they see? A solid audit trail is non-negotiable for regulated data.

Most data warehouses (Databricks, Redshift, BigQuery) log query activity natively. Your job is capturing those logs and making them queryable:

# Example: Capture Spark audit logs from a Databricks workspace
# These are available via workspace audit logs API

import requests
from datetime import datetime, timedelta

def fetch_audit_logs(workspace_url, token, days=7):
    """Fetch query audit logs from Databricks"""
    headers = {"Authorization": f"Bearer {token}"}
    
    start_time = int((datetime.now() - timedelta(days=days)).timestamp() * 1000)
    end_time = int(datetime.now().timestamp() * 1000)
    
    response = requests.get(
        f"{workspace_url}/api/2.0/audit/logs",
        headers=headers,
        params={
            "start_time": start_time,
            "end_time": end_time,
            "filter_by_request_type": "sql_queries"
        }
    )
    
    return response.json()

# Store logs in a table for analysis
logs = fetch_audit_logs(workspace_url, token)
df_logs = spark.createDataFrame(logs["audit_logs"])
df_logs.write.mode("append").saveAsTable("audit_logs")

Once audit logs are in a table, query them regularly to spot suspicious patterns:

# Who accessed sensitive tables in the last 7 days?
spark.sql("""
SELECT 
    user_email, 
    table_name, 
    COUNT(*) as query_count,
    MAX(timestamp) as last_accessed
FROM audit_logs
WHERE table_name IN ('customers_raw', 'payments', 'users_pii')
    AND DATE(timestamp) >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY user_email, table_name
ORDER BY query_count DESC
""").show()

# Alert if someone outside finance queries payment data
spark.sql("""
SELECT user_email, COUNT(*) as queries
FROM audit_logs
WHERE table_name = 'payments'
    AND user_email NOT LIKE '%@finance%'
    AND DATE(timestamp) = CURRENT_DATE
GROUP BY user_email
""").show()

Store audit logs separately from operational data (different bucket, longer retention). Make them immutable—nobody should be able to delete or modify audit records after the fact.

Common mistakes

• Applying governance too late. If you wait until you have sensitive data in production, you're retrofitting. Build access control, masking, and audit from day one, even on small datasets.
• Overly restrictive access. The pendulum swings too far the other way—locking down data so tightly that teams can't do their jobs. Strike a balance: enable self-service within clear guardrails, not zero trust.
• Masking but not auditing. Just because data is masked doesn't mean access doesn't matter. Log everything anyway. Someone masking data improperly is still a security event.
• Forgetting about derived tables. You control access to the raw table, but analysts run queries and export results to CSV. Those CSVs now contain sensitive data. Educate teams about downstream data handling.
• No retention review process. Policies sit in documentation and are never enforced. Automate retention jobs and audit them quarterly to make sure they actually ran.
• Ignoring data lineage. When data flows through ten pipelines before landing in a report, it's hard to audit. Track lineage (which tables feed which outputs) so you can answer "where did this number come from?"

Key takeaways

• Build access control early: table-level for coarse permissions, column-level via views for fine-grained masking. Assign permissions to groups, not individuals.
• Choose between masking (for analytics) and tokenization (for reversible PII protection) based on your use case. Apply it once in your canonical layer, not scattered across tables.
• Define explicit retention policies for each data asset. Automate deletion and use Delta Lake's vacuum to physically remove old data. Document your policies centrally.
• Audit everything: capture query logs, store them immutably, and query them for suspicious patterns. Compliance isn't about one-time certification—it's about continuous monitoring.
• Governance is a team activity. Document decisions, educate teams on data handling, and review access quarterly. Treat it like any other operational concern.