Exploratory Data Analysis

{
  "description": "\"Analyze datasets to discover patterns, anomalies, and relationships. Use when exploring data files, generating statistical summaries, checking data quality, or creating visualizations. Supports CSV, Excel, JSON, Parquet, and more.\"",
  "references": {
    "files": [
      "references/eda_best_practices.md",
      "references/statistical_tests_guide.md"
    ]
  },
  "content": "Discover patterns, anomalies, and relationships in tabular data through statistical analysis and visualization.\r\n\r\n**Supported formats**: CSV, Excel (.xlsx, .xls), JSON, Parquet, TSV, Feather, HDF5, Pickle\r\n\r\n\r\nUser request: \"Explore this sales_data.csv file\"\r\n\r\n```bash\r\npython scripts/eda_analyzer.py sales_data.csv -o ./output\r\n\r\npython scripts/visualizer.py sales_data.csv -o ./output\r\n```\r\n\r\n```python\r\nimport json\r\nwith open('./output/eda_analysis.json') as f:\r\n    results = json.load(f)",
  "name": "exploratory-data-analysis",
  "id": "scientific-thinking-exploratory-data-analysis",
  "sections": {
    "Analysis Capabilities": "### Statistical Analysis\r\n\r\nRun `scripts/eda_analyzer.py` to generate comprehensive analysis:\r\n\r\n```bash\r\npython scripts/eda_analyzer.py sales_data.csv -o ./output\r\n```\r\n\r\nProduces `output/eda_analysis.json` containing:\r\n- Dataset shape, types, memory usage\r\n- Missing data patterns and percentages\r\n- Summary statistics (numeric and categorical)\r\n- Outlier detection (IQR and Z-score methods)\r\n- Distribution analysis with normality tests\r\n- Correlation matrices (Pearson and Spearman)\r\n- Data quality metrics (completeness, duplicates)\r\n- Automated insights\r\n\r\n### Visualizations\r\n\r\nRun `scripts/visualizer.py` to generate plots:\r\n\r\n```bash\r\npython scripts/visualizer.py sales_data.csv -o ./output\r\n```\r\n\r\nCreates high-resolution (300 DPI) PNG files in `output/eda_visualizations/`:\r\n- Missing data heatmaps and bar charts\r\n- Distribution plots (histograms with KDE)\r\n- Box plots and violin plots for outliers\r\n- Correlation heatmaps\r\n- Scatter matrices for numeric relationships\r\n- Categorical bar charts\r\n- Time series plots (if datetime columns detected)\r\n\r\n### Automated Insights\r\n\r\nAccess generated insights from the `\"insights\"` key in the analysis JSON:\r\n- Dataset size considerations\r\n- Missing data warnings (when exceeding thresholds)\r\n- Strong correlations for feature engineering\r\n- High outlier rate flags\r\n- Skewness requiring transformations\r\n- Duplicate detection\r\n- Categorical imbalance warnings",
    "Standard Workflow": "1. Run statistical analysis:\r\n```bash\r\npython scripts/eda_analyzer.py <data_file> -o <output_dir>\r\n```\r\n\r\n2. Generate visualizations:\r\n```bash\r\npython scripts/visualizer.py <data_file> -o <output_dir>\r\n```\r\n\r\n3. Read analysis results from `<output_dir>/eda_analysis.json`\r\n\r\n4. Create report using `assets/report_template.md` structure\r\n\r\n5. Present findings with key insights and visualizations",
    "Error Handling": "**Unsupported formats**: Request conversion to supported format (CSV, Excel, JSON, Parquet)\r\n\r\n**Files too large**: Recommend sampling or chunked processing\r\n\r\n**Corrupted data**: Report specific errors, suggest cleaning steps, attempt partial analysis\r\n\r\n**Empty columns**: Flag in data quality section, recommend removal or investigation",
    "Special Cases": "### Dataset Size Strategy\r\n\r\n**If < 100 rows**: Note sample size limitations, use non-parametric methods\r\n\r\n**If 100-1M rows**: Standard workflow applies\r\n\r\n**If > 1M rows**: Sample first for quick exploration, note sample size in report, recommend distributed computing for full analysis\r\n\r\n### Data Characteristics\r\n\r\n**High-dimensional (>50 columns)**: Focus on key variables first, use correlation analysis to identify groups, consider PCA or feature selection. See `references/eda_best_practices.md` for guidance.\r\n\r\n**Time series**: Datetime columns auto-detected, temporal visualizations generated automatically. Consider trends, seasonality, patterns.\r\n\r\n**Imbalanced**: Categorical analysis flags imbalances automatically. Report distributions prominently, recommend stratified sampling if needed.",
    "Key Points": "- Run both scripts for complete analysis\r\n- Structure reports using the template\r\n- Provide actionable insights, not just statistics\r\n- Use reference guides for detailed interpretations\r\n- Document data quality issues and limitations\r\n- Make clear recommendations for next steps",
    "Report Template": "Use `assets/report_template.md` to structure findings. Template includes:\r\n- Executive summary\r\n- Dataset overview\r\n- Data quality assessment\r\n- Univariate, bivariate, and multivariate analysis\r\n- Outlier analysis\r\n- Key insights and recommendations\r\n- Limitations and appendices\r\n\r\nFill sections with analysis JSON results and embed visualizations using markdown image syntax.",
    "Output Guidelines": "**Format findings as markdown**:\r\n- Use headers, tables, and lists for structure\r\n- Embed visualizations: `![Description](path/to/image.png)`\r\n- Include code blocks for suggested transformations\r\n- Highlight key insights\r\n\r\n**Make reports actionable**:\r\n- Provide clear recommendations\r\n- Flag data quality issues requiring attention\r\n- Suggest next steps (modeling, feature engineering, further analysis)\r\n- Tailor communication to user's technical level",
    "Example: Complete Analysis": "```",
    "Reference Materials": "### Statistical Interpretation\r\n\r\nSee `references/statistical_tests_guide.md` for detailed guidance on:\r\n- Normality tests (Shapiro-Wilk, Anderson-Darling, Kolmogorov-Smirnov)\r\n- Distribution characteristics (skewness, kurtosis)\r\n- Correlation methods (Pearson, Spearman)\r\n- Outlier detection (IQR, Z-score)\r\n- Hypothesis testing and data transformations\r\n\r\nUse when interpreting statistical results or explaining findings.\r\n\r\n### Methodology\r\n\r\nSee `references/eda_best_practices.md` for comprehensive guidance on:\r\n- 6-step EDA process framework\r\n- Univariate, bivariate, multivariate analysis approaches\r\n- Visualization and statistical analysis guidelines\r\n- Common pitfalls and domain-specific considerations\r\n- Communication strategies for different audiences\r\n\r\nUse when planning analysis or handling specific scenarios.",
    "Resources": "**Scripts** (handle all formats automatically):\r\n- `scripts/eda_analyzer.py` - Statistical analysis engine\r\n- `scripts/visualizer.py` - Visualization generator\r\n\r\n**References** (load as needed):\r\n- `references/statistical_tests_guide.md` - Test interpretation and methodology\r\n- `references/eda_best_practices.md` - EDA process and best practices\r\n\r\n**Template**:\r\n- `assets/report_template.md` - Professional report structure"
  }
}

---
name: exploratory-data-analysis
description: "Analyze datasets to discover patterns, anomalies, and relationships. Use when exploring data files, generating statistical summaries, checking data quality, or creating visualizations. Supports CSV, Excel, JSON, Parquet, and more."
---

# Exploratory Data Analysis

Discover patterns, anomalies, and relationships in tabular data through statistical analysis and visualization.

**Supported formats**: CSV, Excel (.xlsx, .xls), JSON, Parquet, TSV, Feather, HDF5, Pickle

## Standard Workflow

1. Run statistical analysis:
```bash
python scripts/eda_analyzer.py <data_file> -o <output_dir>
```

2. Generate visualizations:
```bash
python scripts/visualizer.py <data_file> -o <output_dir>
```

3. Read analysis results from `<output_dir>/eda_analysis.json`

4. Create report using `assets/report_template.md` structure

5. Present findings with key insights and visualizations

## Analysis Capabilities

### Statistical Analysis

Run `scripts/eda_analyzer.py` to generate comprehensive analysis:

```bash
python scripts/eda_analyzer.py sales_data.csv -o ./output
```

Produces `output/eda_analysis.json` containing:
- Dataset shape, types, memory usage
- Missing data patterns and percentages
- Summary statistics (numeric and categorical)
- Outlier detection (IQR and Z-score methods)
- Distribution analysis with normality tests
- Correlation matrices (Pearson and Spearman)
- Data quality metrics (completeness, duplicates)
- Automated insights

### Visualizations

Run `scripts/visualizer.py` to generate plots:

```bash
python scripts/visualizer.py sales_data.csv -o ./output
```

Creates high-resolution (300 DPI) PNG files in `output/eda_visualizations/`:
- Missing data heatmaps and bar charts
- Distribution plots (histograms with KDE)
- Box plots and violin plots for outliers
- Correlation heatmaps
- Scatter matrices for numeric relationships
- Categorical bar charts
- Time series plots (if datetime columns detected)

### Automated Insights

Access generated insights from the `"insights"` key in the analysis JSON:
- Dataset size considerations
- Missing data warnings (when exceeding thresholds)
- Strong correlations for feature engineering
- High outlier rate flags
- Skewness requiring transformations
- Duplicate detection
- Categorical imbalance warnings

## Reference Materials

### Statistical Interpretation

See `references/statistical_tests_guide.md` for detailed guidance on:
- Normality tests (Shapiro-Wilk, Anderson-Darling, Kolmogorov-Smirnov)
- Distribution characteristics (skewness, kurtosis)
- Correlation methods (Pearson, Spearman)
- Outlier detection (IQR, Z-score)
- Hypothesis testing and data transformations

Use when interpreting statistical results or explaining findings.

### Methodology

See `references/eda_best_practices.md` for comprehensive guidance on:
- 6-step EDA process framework
- Univariate, bivariate, multivariate analysis approaches
- Visualization and statistical analysis guidelines
- Common pitfalls and domain-specific considerations
- Communication strategies for different audiences

Use when planning analysis or handling specific scenarios.

## Report Template

Use `assets/report_template.md` to structure findings. Template includes:
- Executive summary
- Dataset overview
- Data quality assessment
- Univariate, bivariate, and multivariate analysis
- Outlier analysis
- Key insights and recommendations
- Limitations and appendices

Fill sections with analysis JSON results and embed visualizations using markdown image syntax.

## Example: Complete Analysis

User request: "Explore this sales_data.csv file"

```bash
# 1. Run analysis
python scripts/eda_analyzer.py sales_data.csv -o ./output

# 2. Generate visualizations
python scripts/visualizer.py sales_data.csv -o ./output
```

```python
# 3. Read results
import json
with open('./output/eda_analysis.json') as f:
    results = json.load(f)

# 4. Build report from assets/report_template.md
# - Fill sections with results
# - Embed images: ![Missing Data](./output/eda_visualizations/missing_data.png)
# - Include insights from results['insights']
# - Add recommendations
```

## Special Cases

### Dataset Size Strategy

**If < 100 rows**: Note sample size limitations, use non-parametric methods

**If 100-1M rows**: Standard workflow applies

**If > 1M rows**: Sample first for quick exploration, note sample size in report, recommend distributed computing for full analysis

### Data Characteristics

**High-dimensional (>50 columns)**: Focus on key variables first, use correlation analysis to identify groups, consider PCA or feature selection. See `references/eda_best_practices.md` for guidance.

**Time series**: Datetime columns auto-detected, temporal visualizations generated automatically. Consider trends, seasonality, patterns.

**Imbalanced**: Categorical analysis flags imbalances automatically. Report distributions prominently, recommend stratified sampling if needed.

## Output Guidelines

**Format findings as markdown**:
- Use headers, tables, and lists for structure
- Embed visualizations: `![Description](path/to/image.png)`
- Include code blocks for suggested transformations
- Highlight key insights

**Make reports actionable**:
- Provide clear recommendations
- Flag data quality issues requiring attention
- Suggest next steps (modeling, feature engineering, further analysis)
- Tailor communication to user's technical level

## Error Handling

**Unsupported formats**: Request conversion to supported format (CSV, Excel, JSON, Parquet)

**Files too large**: Recommend sampling or chunked processing

**Corrupted data**: Report specific errors, suggest cleaning steps, attempt partial analysis

**Empty columns**: Flag in data quality section, recommend removal or investigation

## Resources

**Scripts** (handle all formats automatically):
- `scripts/eda_analyzer.py` - Statistical analysis engine
- `scripts/visualizer.py` - Visualization generator

**References** (load as needed):
- `references/statistical_tests_guide.md` - Test interpretation and methodology
- `references/eda_best_practices.md` - EDA process and best practices

**Template**:
- `assets/report_template.md` - Professional report structure

## Key Points

- Run both scripts for complete analysis
- Structure reports using the template
- Provide actionable insights, not just statistics
- Use reference guides for detailed interpretations
- Document data quality issues and limitations
- Make clear recommendations for next steps