Senior Data Engineer

Name: Senior Data Engineer
Author: davila7

davila7/claude-code-templates

1.5k installs
29.9k repo stars
Updated July 27, 2026
davila7/claude-code-templates

Senior-level data engineering expertise for building, deploying, and operating scalable production data systems with high availability, security, and cost efficiency.

About

Senior Data Engineer skill provides production-grade expertise in building scalable data pipelines, ETL/ELT systems, and ML infrastructure. Covers data modeling, pipeline orchestration with Airflow and dbt, real-time processing via Kafka, and distributed computing on Spark. Developers use this when designing data architectures, optimizing workflows across PostgreSQL/BigQuery/Snowflake, implementing data quality validation, and deploying ML models with monitoring via MLflow and Prometheus. Includes patterns for horizontal scaling, fault tolerance, A/B testing, and automated retraining with security and compliance built-in.

Orchestrate ETL/ELT pipelines with Airflow, dbt, Spark, and Kafka for batch and real-time processing
Deploy ML models with A/B testing, feature stores, and drift detection; monitor with MLflow and Weights & Biases
Validate data quality, optimize performance to P99 <200ms latency, and handle >1000 req/s throughput
Manage distributed computing, horizontal scaling, fault tolerance, and cost optimization across cloud platforms
Enforce security (encryption, PII handling, GDPR/CCPA), monitoring (Prometheus, Datadog), and automated canary deploymen

Senior Data Engineer by the numbers

1,521 all-time installs (skills.sh)
+32 installs in the week ending Jul 28, 2026 (Skillselion tracking)
Ranked #144 of 2,066 Data Science & ML skills by installs in the Skillselion catalog
Security screen: LOW risk (skills.sh audit)
Data as of Jul 28, 2026 (Skillselion catalog sync)

At a glance

senior-data-engineer capabilities & compatibility

Capabilities: data pipeline design and optimization · etl/elt orchestration and scheduling · real time streaming and batch processing · ml model deployment and monitoring · data quality validation and governance · performance tuning and cost optimization · security and compliance implementation · incident response and observability
Works with: kafka · databricks · snowflake · postgres · datadog · kubernetes · docker · aws · gcp · azure
Use cases: ci cd · data analysis · api development · devops · testing
Runs: Local or remote
Pricing: Free

From the docs

What senior-data-engineer says it does

Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps.

skill listing description

npx skills add https://github.com/davila7/claude-code-templates --skill senior-data-engineer

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/davila7/claude-code-templates/senior-data-engineer.svg)](https://skillselion.com/skills/davila7/claude-code-templates/senior-data-engineer)

Installs	1.5k
repo stars	★ 29.9k
Security audit	3 / 3 scanners passed
Last updated	July 27, 2026
Repository	davila7/claude-code-templates ↗

What it does

Design and operate production data pipelines, ETL systems, and ML infrastructure with monitoring and optimization.

Who is it for?

Teams building enterprise data lakes, real-time analytics platforms, MLOps infrastructure, or multi-terabyte data systems requiring high availability and compliance.

Skip if: Simple CSV analysis, single-server workloads, or exploratory data science without production deployment requirements.

When should I use this skill?

Designing data architecture, optimizing existing pipelines, implementing data governance, deploying ML models to production, or scaling systems beyond 1000 req/s.

What you get

Production data pipelines with 99.9% uptime, <0.1% error rate, distributed fault-tolerant architectures, and automated ML model deployment with monitoring and governance.

data models
pipeline architecture
observability plan

By the numbers

Performance target P99 latency: <200ms
Throughput target: >1000 requests/second
Availability target: 99.9% uptime, <0.1% error rate

Files

SKILL.mdMarkdownGitHub ↗

Senior Data Engineer

World-class senior data engineer skill for production-grade AI/ML/Data systems.

Quick Start

Main Capabilities

# Core Tool 1
python scripts/pipeline_orchestrator.py --input data/ --output results/

# Core Tool 2  
python scripts/data_quality_validator.py --target project/ --analyze

# Core Tool 3
python scripts/etl_performance_optimizer.py --config config.yaml --deploy

Core Expertise

This skill covers world-class capabilities in:

Advanced production patterns and architectures
Scalable system design and implementation
Performance optimization at scale
MLOps and DataOps best practices
Real-time processing and inference
Distributed computing frameworks
Model deployment and monitoring
Security and compliance
Cost optimization
Team leadership and mentoring

Tech Stack

Languages: Python, SQL, R, Scala, Go ML Frameworks: PyTorch, TensorFlow, Scikit-learn, XGBoost Data Tools: Spark, Airflow, dbt, Kafka, Databricks LLM Frameworks: LangChain, LlamaIndex, DSPy Deployment: Docker, Kubernetes, AWS/GCP/Azure Monitoring: MLflow, Weights & Biases, Prometheus Databases: PostgreSQL, BigQuery, Snowflake, Pinecone

Reference Documentation

1. Data Pipeline Architecture

Comprehensive guide available in references/data_pipeline_architecture.md covering:

Advanced patterns and best practices
Production implementation strategies
Performance optimization techniques
Scalability considerations
Security and compliance
Real-world case studies

2. Data Modeling Patterns

Complete workflow documentation in references/data_modeling_patterns.md including:

Step-by-step processes
Architecture design patterns
Tool integration guides
Performance tuning strategies
Troubleshooting procedures

3. Dataops Best Practices

Technical reference guide in references/dataops_best_practices.md with:

System design principles
Implementation examples
Configuration best practices
Deployment strategies
Monitoring and observability

Production Patterns

Pattern 1: Scalable Data Processing

Enterprise-scale data processing with distributed computing:

Horizontal scaling architecture
Fault-tolerant design
Real-time and batch processing
Data quality validation
Performance monitoring

Pattern 2: ML Model Deployment

Production ML system with high availability:

Model serving with low latency
A/B testing infrastructure
Feature store integration
Model monitoring and drift detection
Automated retraining pipelines

Pattern 3: Real-Time Inference

High-throughput inference system:

Batching and caching strategies
Load balancing
Auto-scaling
Latency optimization
Cost optimization

Best Practices

Development

Test-driven development
Code reviews and pair programming
Documentation as code
Version control everything
Continuous integration

Production

Monitor everything critical
Automate deployments
Feature flags for releases
Canary deployments
Comprehensive logging

Team Leadership

Mentor junior engineers
Drive technical decisions
Establish coding standards
Foster learning culture
Cross-functional collaboration

Performance Targets

Latency:

P50: < 50ms
P95: < 100ms
P99: < 200ms

Throughput:

Requests/second: > 1000
Concurrent users: > 10,000

Availability:

Uptime: 99.9%
Error rate: < 0.1%

Security & Compliance

Authentication & authorization
Data encryption (at rest & in transit)
PII handling and anonymization
GDPR/CCPA compliance
Regular security audits
Vulnerability management

Common Commands

# Development
python -m pytest tests/ -v --cov
python -m black src/
python -m pylint src/

# Training
python scripts/train.py --config prod.yaml
python scripts/evaluate.py --model best.pth

# Deployment
docker build -t service:v1 .
kubectl apply -f k8s/
helm upgrade service ./charts/

# Monitoring
kubectl logs -f deployment/service
python scripts/health_check.py

Resources

Advanced Patterns: references/data_pipeline_architecture.md
Implementation Guide: references/data_modeling_patterns.md
Technical Reference: references/dataops_best_practices.md
Automation Scripts: scripts/ directory

Senior-Level Responsibilities

As a world-class senior professional:

1. Technical Leadership

Drive architectural decisions
Mentor team members
Establish best practices
Ensure code quality

2. Strategic Thinking

Align with business goals
Evaluate trade-offs
Plan for scale
Manage technical debt

3. Collaboration

Work across teams
Communicate effectively
Build consensus
Share knowledge

4. Innovation

Stay current with research
Experiment with new approaches
Contribute to community
Drive continuous improvement

5. Production Excellence

Ensure high availability
Monitor proactively
Optimize performance
Respond to incidents

#!/usr/bin/env python3
"""
Data Quality Validator
Production-grade tool for senior data engineer
"""

import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

class DataQualityValidator:
    """Production-grade data quality validator"""
    
    def __init__(self, config: Dict):
        self.config = config
        self.results = {
            'status': 'initialized',
            'start_time': datetime.now().isoformat(),
            'processed_items': 0
        }
        logger.info(f"Initialized {self.__class__.__name__}")
    
    def validate_config(self) -> bool:
        """Validate configuration"""
        logger.info("Validating configuration...")
        # Add validation logic
        logger.info("Configuration validated")
        return True
    
    def process(self) -> Dict:
        """Main processing logic"""
        logger.info("Starting processing...")
        
        try:
            self.validate_config()
            
            # Main processing
            result = self._execute()
            
            self.results['status'] = 'completed'
            self.results['end_time'] = datetime.now().isoformat()
            
            logger.info("Processing completed successfully")
            return self.results
            
        except Exception as e:
            self.results['status'] = 'failed'
            self.results['error'] = str(e)
            logger.error(f"Processing failed: {e}")
            raise
    
    def _execute(self) -> Dict:
        """Execute main logic"""
        # Implementation here
        return {'success': True}

def main():
    """Main entry point"""
    parser = argparse.ArgumentParser(
        description="Data Quality Validator"
    )
    parser.add_argument('--input', '-i', required=True, help='Input path')
    parser.add_argument('--output', '-o', required=True, help='Output path')
    parser.add_argument('--config', '-c', help='Configuration file')
    parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
    
    args = parser.parse_args()
    
    if args.verbose:
        logging.getLogger().setLevel(logging.DEBUG)
    
    try:
        config = {
            'input': args.input,
            'output': args.output
        }
        
        processor = DataQualityValidator(config)
        results = processor.process()
        
        print(json.dumps(results, indent=2))
        sys.exit(0)
        
    except Exception as e:
        logger.error(f"Fatal error: {e}")
        sys.exit(1)

if __name__ == '__main__':
    main()

#!/usr/bin/env python3
"""
Etl Performance Optimizer
Production-grade tool for senior data engineer
"""

import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

class EtlPerformanceOptimizer:
    """Production-grade etl performance optimizer"""
    
    def __init__(self, config: Dict):
        self.config = config
        self.results = {
            'status': 'initialized',
            'start_time': datetime.now().isoformat(),
            'processed_items': 0
        }
        logger.info(f"Initialized {self.__class__.__name__}")
    
    def validate_config(self) -> bool:
        """Validate configuration"""
        logger.info("Validating configuration...")
        # Add validation logic
        logger.info("Configuration validated")
        return True
    
    def process(self) -> Dict:
        """Main processing logic"""
        logger.info("Starting processing...")
        
        try:
            self.validate_config()
            
            # Main processing
            result = self._execute()
            
            self.results['status'] = 'completed'
            self.results['end_time'] = datetime.now().isoformat()
            
            logger.info("Processing completed successfully")
            return self.results
            
        except Exception as e:
            self.results['status'] = 'failed'
            self.results['error'] = str(e)
            logger.error(f"Processing failed: {e}")
            raise
    
    def _execute(self) -> Dict:
        """Execute main logic"""
        # Implementation here
        return {'success': True}

def main():
    """Main entry point"""
    parser = argparse.ArgumentParser(
        description="Etl Performance Optimizer"
    )
    parser.add_argument('--input', '-i', required=True, help='Input path')
    parser.add_argument('--output', '-o', required=True, help='Output path')
    parser.add_argument('--config', '-c', help='Configuration file')
    parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
    
    args = parser.parse_args()
    
    if args.verbose:
        logging.getLogger().setLevel(logging.DEBUG)
    
    try:
        config = {
            'input': args.input,
            'output': args.output
        }
        
        processor = EtlPerformanceOptimizer(config)
        results = processor.process()
        
        print(json.dumps(results, indent=2))
        sys.exit(0)
        
    except Exception as e:
        logger.error(f"Fatal error: {e}")
        sys.exit(1)

if __name__ == '__main__':
    main()

#!/usr/bin/env python3
"""
Pipeline Orchestrator
Production-grade tool for senior data engineer
"""

import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

class PipelineOrchestrator:
    """Production-grade pipeline orchestrator"""
    
    def __init__(self, config: Dict):
        self.config = config
        self.results = {
            'status': 'initialized',
            'start_time': datetime.now().isoformat(),
            'processed_items': 0
        }
        logger.info(f"Initialized {self.__class__.__name__}")
    
    def validate_config(self) -> bool:
        """Validate configuration"""
        logger.info("Validating configuration...")
        # Add validation logic
        logger.info("Configuration validated")
        return True
    
    def process(self) -> Dict:
        """Main processing logic"""
        logger.info("Starting processing...")
        
        try:
            self.validate_config()
            
            # Main processing
            result = self._execute()
            
            self.results['status'] = 'completed'
            self.results['end_time'] = datetime.now().isoformat()
            
            logger.info("Processing completed successfully")
            return self.results
            
        except Exception as e:
            self.results['status'] = 'failed'
            self.results['error'] = str(e)
            logger.error(f"Processing failed: {e}")
            raise
    
    def _execute(self) -> Dict:
        """Execute main logic"""
        # Implementation here
        return {'success': True}

def main():
    """Main entry point"""
    parser = argparse.ArgumentParser(
        description="Pipeline Orchestrator"
    )
    parser.add_argument('--input', '-i', required=True, help='Input path')
    parser.add_argument('--output', '-o', required=True, help='Output path')
    parser.add_argument('--config', '-c', help='Configuration file')
    parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
    
    args = parser.parse_args()
    
    if args.verbose:
        logging.getLogger().setLevel(logging.DEBUG)
    
    try:
        config = {
            'input': args.input,
            'output': args.output
        }
        
        processor = PipelineOrchestrator(config)
        results = processor.process()
        
        print(json.dumps(results, indent=2))
        sys.exit(0)
        
    except Exception as e:
        logger.error(f"Fatal error: {e}")
        sys.exit(1)

if __name__ == '__main__':
    main()

Related skills

Microsoft FoundryDeploy, evaluate, and continuously improve Microsoft Foundry agents from a single agent interface.478k1.3k

Ai Research ReproductionOrchestrate trustworthy, auditable reproduction of deep learning repositories directly from their READMEs.164k507

Run TrainSafely execute selected deep learning training commands with standardized evidence capture.164k507

Explore RunSafely run isolated exploratory experiments with clear recording and conservative selection before committing changes.164k507

Paper Context ResolverFetch precise reproduction-critical details like dataset splits, preprocessing steps, or evaluation protocols from the original academic paper when the repo README leav141k507

Repo Intake And PlanScan unfamiliar AI research repositories and receive a minimal, trustworthy reproduction target before investing significant time.140k507

Forks & variants (2)

Senior Data Engineer has 2 known copies in the catalog totaling 93 installs. They canonicalize to this original listing.

ovachiever - 50 installs
smithery.ai - 43 installs

How it compares

Use senior-data-engineer when pipeline production constraints matter more than lightweight notebook exploration or one-off data transforms.

FAQ

What frameworks does this skill use for pipeline orchestration?

Airflow for workflow orchestration, dbt for transformation, Spark for distributed processing, and Kafka for real-time streaming.

How does it handle ML model monitoring and drift detection?

Uses MLflow and Weights & Biases for experiment tracking, implements automated retraining pipelines, and detects model drift via performance monitoring.

What security and compliance features are included?

Encryption at rest and in transit, PII anonymization, GDPR/CCPA compliance, IAM controls, and regular vulnerability audits.

Is Senior Data Engineer safe to install?

skills.sh reports 3 of 3 security scanners passed. Review the Security Audits panel on this page before installing in production.

Data Science & MLpipelinesanalytics

About

Senior Data Engineer by the numbers

senior-data-engineer capabilities & compatibility

What senior-data-engineer says it does

Add your badge

What it does

Who is it for?

When should I use this skill?

What you get

By the numbers

Files

Senior Data Engineer

Quick Start

Main Capabilities

Core Expertise

Tech Stack

Reference Documentation

1. Data Pipeline Architecture

2. Data Modeling Patterns

3. Dataops Best Practices

Production Patterns

Pattern 1: Scalable Data Processing

Pattern 2: ML Model Deployment

Pattern 3: Real-Time Inference

Best Practices

Development

Production

Team Leadership

Performance Targets

Security & Compliance

Common Commands

Resources

Senior-Level Responsibilities

Data Modeling Patterns

Overview

Core Principles

Production-First Design

Performance by Design

Security & Privacy

Advanced Patterns

Pattern 1: Distributed Processing

Pattern 2: Real-Time Systems

Pattern 3: ML at Scale

Best Practices

Code Quality

Performance

Reliability

Tools & Technologies

Further Reading

Data Pipeline Architecture

Overview

Core Principles

Production-First Design

Performance by Design

Security & Privacy

Advanced Patterns

Pattern 1: Distributed Processing

Pattern 2: Real-Time Systems

Pattern 3: ML at Scale

Best Practices

Code Quality

Performance

Reliability

Tools & Technologies

Further Reading

Dataops Best Practices

Overview

Core Principles

Production-First Design

Performance by Design

Security & Privacy

Advanced Patterns

Pattern 1: Distributed Processing

Pattern 2: Real-Time Systems

Pattern 3: ML at Scale

Best Practices

Code Quality

Performance

Reliability

Tools & Technologies