Data Architect

Data Architect

Overview

Design data architecture at enterprise and solution levels. This skill covers data mesh, lakehouse, governance, domain-driven design, conceptual/logical/physical data modeling, platform selection, and compliance frameworks. Produce ADRs, data model diagrams, platform comparison matrices, and governance policy templates.

When to Use

• Choosing among warehouse, lake, lakehouse, mesh, or streaming-first patterns
• Creating conceptual, logical, or physical data models and ADRs
• Defining data governance, catalog, quality, and compliance frameworks
• Evaluating data platforms and long-term TCO or vendor trade-offs

When NOT to Use

• Day-to-day pipeline on-call, SLA breaches, or shift handoffs → use data-system-ops-lead
• Single-platform SQL tuning or star-schema implementation detail → use data-warehouse-engineer
• dbt project implementation, mart tests, and analytics CI → use analytics-data-engineer
• Team roadmaps, sprint cadence, or governance operations execution → use data-manager
• OWL/RDF ontologies or knowledge-graph construction → use ontology-engineer
• Application integration patterns and non-data system ADRs → use senior-system-architecture
• LLM/RAG/copilot solution architecture and AI ADRs → use applied-ai-architect-commercial-enterprise

Features

• Architecture decision framework with weighted criteria evaluation
• Progressive data modeling workflow (conceptual → logical → physical)
• Platform selection decision tree for warehouse/lake/lakehouse/mesh/streaming
• Governance pillar planning with tool recommendations
• ADR template generation and stakeholder review processes

Usage

1. Identify the user's data architecture need (platform choice, modeling, governance, or decision framework) 2. Follow the corresponding workflow below 3. Produce structured outputs: ADRs, data model diagrams, platform comparison matrices, or governance policies

Examples

• User: "Should we use a data lake or data warehouse for our analytics?"

Agent: Runs Platform & Technology Selection workflow (Workflow 3), evaluates structured vs raw data needs, recommends warehouse/lake/lakehouse with trade-offs

• User: "We need to model our customer domain"

Agent: Runs Data Modeling Workflow (Workflow 2), starts with conceptual model (entities, relationships), progresses to logical ER diagram, then physical DDL

• User: "How do we set up data governance for GDPR compliance?"

Agent: Runs Governance & Compliance Planning (Workflow 4), maps GDPR requirements to governance pillars, recommends tools and controls

Core Workflows

1. Architecture Decision Framework

Use this 5-step process for any major data architecture decision:

1. Define the decision context

• Business drivers (scale, latency, cost, compliance)
• Constraints (budget, timeline, existing tech, team skills)
• Stakeholders (data engineers, analysts, product, legal)

2. Identify alternatives

• At least 3 options (do nothing, minimal change, transformative)
• Include cloud-native, hybrid, and open-source alternatives

3. Evaluate against criteria

4. Assess risks & mitigation

• Migration risk, talent risk, operational risk
• POC plan for the top 2 options

5. Document the decision

• ADR (Architecture Decision Record) with context, decision, consequences
• Share with stakeholders; revisit quarterly

2. Data Modeling Workflow

Progressive refinement from business to implementation:

Key principles:

• Start with the business question, not the technology
• Use surrogate keys in physical model; natural keys in logical
• Denormalize only when you have a performance requirement

3. Platform & Technology Selection

Decision tree:

• Need structured analytics + BI at scale? → Data Warehouse (Snowflake, BigQuery, Redshift)
• Need raw data + ML + flexible schemas? → Data Lake (S3 + Athena/Spark)
• Need both with ACID guarantees? → Lakehouse (Databricks, Iceberg, Hudi)
• Need domain ownership + federated governance? → Data Mesh (multiple warehouses/lakes per domain)
• Need real-time + low latency? → Streaming-first (Kafka + Flink + materialized views)

4. Governance & Compliance Planning

Governance pillars:

Data Modeling

Conceptual Modeling

Purpose: Align business and technology on "what" data exists

Techniques:

• Business capability mapping: What does the business do? → What data does it need?
• Domain storytelling: Walk through user journeys; identify events and entities
• Event storming: Identify domain events, commands, aggregates

Output:

• Business glossary (terms, definitions, owners)
• High-level entity list
• Context map showing bounded contexts

Logical Modeling

ER Notation (Crow's Foot)

• Entity = rectangle
• Relationship = line with crow's foot (many) or single bar (one)
• Attribute = oval or listed inside entity

Normalization Rules

Practical guidance:

• Normalize to 3NF in logical model
• Denormalize in physical model for read performance
• Document every denormalization with justification

Common Schema Patterns

Star Schema

• Central fact table + surrounding dimension tables
• Simple joins, fast aggregation
• Best for: BI, dashboards, ad-hoc analysis

Snowflake Schema

• Dimensions normalized into sub-dimensions
• Reduced redundancy, more complex joins
• Best for: complex hierarchies, storage optimization

Data Vault 2.0

• Hubs (business keys), Links (relationships), Satellites (attributes)
• Auditability, agility, parallel loading
• Best for: large enterprises, frequent source changes, compliance needs

Anchor Modeling

• Extreme normalization; every attribute is a separate table
• Highly flexible but complex queries
• Best for: domains with rapid schema evolution

Key Design

Recommendation: Use surrogate keys in physical tables; document natural key mapping.

Physical Modeling

Transformation from Logical to Physical

1. Choose platform (see platform_selection.md) 2. Map data types (platform-specific) 3. Add technical columns: created_at, updated_at, loaded_by, batch_id 4. Design partitions: Time-based for event data, hash-based for even distribution 5. Add indexes: Primary, foreign, covering indexes for common queries 6. Apply compression: Columnar for analytics, row for OLTP

Partitioning Strategies

Denormalization Techniques

Enterprise Architecture Patterns

Data Mesh

Core principles (Zhamak Dehghani):

• Domain-oriented decentralized data ownership
• Data as a product (with SLAs, documentation, discoverability)
• Self-serve data infrastructure as a platform
• Federated computational governance

When to adopt:

• Large organization (>500 data consumers)
• Multiple domains with distinct data needs
• Pain points with centralized data team bottleneck

When NOT to adopt:

• Small organization (<50 data consumers)
• Limited platform engineering capacity
• Strong need for centralized compliance control

Implementation phases: 1. Identify domains and domain data product owners 2. Build self-serve infrastructure platform (data catalog, pipeline templates, access control) 3. Define federated governance standards (schema registries, quality SLAs) 4. Gradually migrate domain by domain

Data Lakehouse

Combines data lake flexibility + warehouse reliability:

• Open table formats: Delta Lake, Apache Iceberg, Apache Hudi
• ACID transactions on object storage
• Schema evolution and time travel
• Unified batch + streaming

Architecture layers:

┌─────────────────────────────────────┐
│  Consumption (BI, ML, Analytics)   │
├─────────────────────────────────────┤
│  Semantic/Gold (curated, modeled)   │
├─────────────────────────────────────┤
│  Refined/Silver (cleaned, joined)   │
├─────────────────────────────────────┤
│  Raw/Bronze (ingested as-is)        │
├─────────────────────────────────────┤
│  Ingestion (batch + streaming)       │
└─────────────────────────────────────┘

Medallion architecture guidelines:

• Bronze: Append-only, schema on read, keep raw for 30-90 days
• Silver: Deduplicated, typed, domain-partitioned
• Gold: Business-aggregated, star schema, optimized for query

Domain-Driven Design (DDD) for Data

Bounded contexts map to data domains:

• Each domain owns its data models, pipelines, and quality SLAs
• Shared kernels for cross-domain concepts (customer, product)
• Anti-corruption layer when integrating legacy systems

Ubiquitous language:

• Maintain a business glossary per domain
• Map terms to logical model entities
• Resolve conflicts at enterprise level (e.g., "customer" in Sales vs Support)

Event-Driven Architecture

Patterns:

• Event sourcing: Store state changes as events; reconstruct state
• CQRS: Separate read and write models
• Change Data Capture (CDC): Database → event stream → consumers

When to use:

• Microservices needing data synchronization
• Real-time analytics requirements
• Audit trails and compliance needs

Reference Architectures

Modern Data Platform (Simplified)

Sources → Ingestion → Storage → Processing → Serving → Consumption

Sources:     Apps, APIs, DBs, IoT, SaaS
Ingestion:   Kafka, Fivetran, Airbyte, Debezium
Storage:     S3/ADLS/GCS (lake) + Warehouse (Snowflake/BQ)
Processing:  Spark, dbt, Flink, Airflow
Serving:     Semantic layer (Cube, dbt metrics), Feature store
Consumption: BI (Looker, Tableau), Notebooks, Apps

Hybrid Cloud

• Sensitive data on-premise (HIPAA, financial)
• Analytics in cloud (scalability, ML services)
• Use private connectivity (Direct Connect, ExpressRoute)

Governance & Compliance Frameworks

Data Quality Framework

Dimensions:

Implementation tiers: 1. Reactive: Fix issues after they cause problems 2. Proactive: dbt tests, Great Expectations, data contracts 3. Predictive: Anomaly detection (Monte Carlo, Bigeye, Anomalo)

Data Catalog & Lineage

Capabilities:

• Discovery: Search datasets by name, description, tags, owner
• Metadata: Schema, descriptions, data types, update frequency
• Lineage: Upstream sources → transformations → downstream consumers
• Impact analysis: What breaks if I change this column?

Tools comparison:

Lineage implementation approaches: 1. Query log parsing: Parse SQL to build lineage (automated, incomplete) 2. Orchestrator integration: Airflow/Dagster task dependencies 3. Manual annotation: Human-curated for critical pipelines

Master Data Management (MDM)

When to implement MDM:

• Same entity exists in 3+ systems with conflicting data
• Customer 360 initiatives failing due to fragmented data
• Compliance requires single source of truth

Patterns:

• Registry: Match IDs across systems, no central storage
• Consolidation: Extract, match, merge into golden record database
• Coexistence: Hybrid; updates flow back to source systems
• Transaction hub: Central system becomes system of record

Matching techniques:

• Deterministic: Exact match on email, phone, SSN
• Probabilistic: Fuzzy matching (Levenshtein, Jaro-Winkler)
• ML-based: Classification models for entity resolution

Access Control Architecture

Layered approach:

Network → Identity → Role → Policy → Data

Network: VPC, private links, IP allowlists
Identity: SSO, MFA, service accounts
Role: RBAC groups (analyst, engineer, admin)
Policy: Row-level, column-level, masking
Data: Encryption at rest, in transit

Row-level security (RLS) patterns:

• Filter predicates in views
• Platform-native RLS policies
• Application-level enforcement (last resort)

Compliance Frameworks

GDPR (EU)

Key requirements for data architecture:

• Data minimization: Collect only what's necessary
• Purpose limitation: Define and enforce use cases
• Right to erasure: Technical capability to delete across systems
• Data portability: Export in machine-readable format
• Privacy by design: Pseudonymization, encryption

CCPA/CPRA (California)

Key requirements:

• Consumer rights: Know, delete, opt-out of sale
• Data inventory: Map personal information flows
• Service provider contracts: Ensure downstream compliance

HIPAA (Healthcare)

Key requirements:

• PHI identification and access logging
• Encryption (at rest, in transit)
• Audit trails (who accessed what, when)
• Minimum necessary access principle

SOC 2

Trust services criteria:

• Security: Access controls, encryption
• Availability: Monitoring, incident response
• Processing integrity: Data validation, error handling
• Confidentiality: Classification, retention
• Privacy: Consent, collection limitations

Data Retention & Lifecycle

Policy template:

Technical implementation:

• Partitioning by date enables efficient deletion/archival
• Object storage lifecycle policies (S3 Glacier, GCS Nearline)
• Legal holds override automatic deletion

Platform & Technology Selection

CAP Theorem for Data Systems

In distributed systems, pick two:

• Consistency: All nodes see the same data at the same time
• Availability: Every request receives a response
• Partition tolerance: System continues despite network failures

Data system choices:

Platform Comparison Matrix

TCO Considerations (3-Year)

Cost components: 1. Infrastructure (compute, storage, networking) 2. Licensing / subscription 3. Migration (data movement, schema conversion, testing) 4. Operations (monitoring, tuning, support) 5. Training (team upskilling)

Cost optimization levers:

• Reserved instances / committed use discounts for predictable workloads
• Separate dev/test environments (smaller or spot instances)
• Lifecycle policies: hot → warm → cold → archive
• Query optimization (see data-warehouse-engineer skill for SQL tuning)

Vendor Evaluation Checklist

Technical:

• [ ] Supports required data types and formats
• [ ] Meets latency/throughput SLAs
• [ ] Integrates with existing toolchain
• [ ] Has required security certifications
• [ ] Provides adequate backup/DR capabilities

Commercial:

• [ ] Pricing model aligns with usage patterns
• [ ] Contract terms allow flexibility (scale up/down)
• [ ] Exit strategy defined (data portability, format)

Organizational:

• [ ] Team has skills or can acquire them
• [ ] Vendor support quality assessed
• [ ] Community/ecosystem maturity

Hybrid & Multi-Cloud Strategies

When to go multi-cloud:

• Vendor redundancy requirements
• Best-of-breed services (e.g., BigQuery for analytics + Snowflake for sharing)
• M&A integration (inherited platforms)

Challenges:

• Data egress costs
• Cross-cloud identity management
• Operational complexity
• Consistent tooling across platforms

Mitigation:

• Use open table formats (Iceberg, Delta) for portability
• Centralize orchestration (Airflow, Dagster)
• Abstract storage (S3-compatible APIs)