Clip Aware Embeddings
- 118 installs
- 178 repo stars
- Updated July 14, 2026
- erichowens/some_claude_skills
Generate and query CLIP-aligned embeddings for images and text, enabling semantic search, similarity ranking, and multimodal retrieval in apps that mix visual and language assets.
About
Covers CLIP-aware embedding design for erichowens/some_claude_skills: preprocessing images and captions, producing consistent vectors, indexing for similarity search, and integrating retrieval into agents or APIs that need multimodal semantic matching at scale.
- Normalizes image and text inputs for CLIP models
- Defines batching, caching, and dimension consistency
- Pairs embeddings with vector index strategies
- Handles multimodal query and re-ranking patterns
- Notes GPU, latency, and cost tradeoffs
Clip Aware Embeddings by the numbers
- 118 all-time installs (skills.sh)
- Ranked #774 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Aug 4, 2026 (Skillselion catalog sync)
npx skills add https://github.com/erichowens/some_claude_skills --skill clip-aware-embeddingsAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 118 |
|---|---|
| repo stars | ★ 178 |
| Last updated | July 14, 2026 |
| Repository | erichowens/some_claude_skills ↗ |
What it does
Generate and query CLIP-aligned embeddings for images and text, enabling semantic search, similarity ranking, and multimodal retrieval in apps that mix visual and language assets.
Files
CLIP-Aware Image Embeddings
Smart image-text matching that knows when CLIP works and when to use alternatives.
MCP Integrations
| MCP | Purpose |
|---|---|
| Firecrawl | Research latest CLIP alternatives and benchmarks |
| Hugging Face (if configured) | Access model cards and documentation |
Quick Decision Tree
Your task:
├─ Semantic search ("find beach images") → CLIP ✓
├─ Zero-shot classification (broad categories) → CLIP ✓
├─ Counting objects → DETR, Faster R-CNN ✗
├─ Fine-grained ID (celebrities, car models) → Specialized model ✗
├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗
└─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗When to Use This Skill
✅ Use for:
- Semantic image search
- Broad category classification
- Image similarity matching
- Zero-shot tasks on new categories
❌ Do NOT use for:
- Counting objects in images
- Fine-grained classification
- Spatial understanding
- Attribute binding
- Negation handling
Installation
pip install transformers pillow torch sentence-transformers --break-system-packagesValidation: Run python scripts/validate_setup.py
Basic Usage
Image Search
from transformers import CLIPProcessor, CLIPModel
from PIL import Image
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
# Embed images
images = [Image.open(f"img{i}.jpg") for i in range(10)]
inputs = processor(images=images, return_tensors="pt")
image_features = model.get_image_features(**inputs)
# Search with text
text_inputs = processor(text=["a beach at sunset"], return_tensors="pt")
text_features = model.get_text_features(**text_inputs)
# Compute similarity
similarity = (image_features @ text_features.T).softmax(dim=0)Common Anti-Patterns
Anti-Pattern 1: "CLIP for Everything"
❌ Wrong:
# Using CLIP to count cars in an image
prompt = "How many cars are in this image?"
# CLIP cannot count - it will give nonsense resultsWhy wrong: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.
✓ Right:
from transformers import DetrImageProcessor, DetrForObjectDetection
processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
# Detect objects
results = model(**processor(images=image, return_tensors="pt"))
# Filter for cars and count
car_detections = [d for d in results if d['label'] == 'car']
count = len(car_detections)How to detect: If query contains "how many", "count", or numeric questions → Use object detection
---
Anti-Pattern 2: Fine-Grained Classification
❌ Wrong:
# Trying to identify specific celebrities with CLIP
prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]
# CLIP will perform poorly - not trained for fine-grained face IDWhy wrong: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.
✓ Right:
# Use a fine-tuned face recognition model
from transformers import AutoFeatureExtractor, AutoModelForImageClassification
model = AutoModelForImageClassification.from_pretrained(
"microsoft/resnet-50" # Then fine-tune on celebrity dataset
)
# Or use dedicated face recognition: ArcFace, CosFaceHow to detect: If query asks to distinguish between similar items in same category → Use specialized model
---
Anti-Pattern 3: Spatial Understanding
❌ Wrong:
# CLIP cannot understand spatial relationships
prompts = [
"cat to the left of dog",
"cat to the right of dog"
]
# Will give nearly identical scoresWhy wrong: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.
✓ Right:
# Use a spatial reasoning model
# Examples: GQA models, Visual Genome models, SWIG
from swig_model import SpatialRelationModel
model = SpatialRelationModel()
result = model.predict_relation(image, "cat", "dog")
# Returns: "left", "right", "above", "below", etc.How to detect: If query contains directional words (left, right, above, under, next to) → Use spatial model
---
Anti-Pattern 4: Attribute Binding
❌ Wrong:
prompts = [
"red car and blue truck",
"blue car and red truck"
]
# CLIP often gives similar scores for bothWhy wrong: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.
✓ Right - Use PC-CLIP or DCSMs:
# PC-CLIP: Fine-tuned for pairwise comparisons
from pc_clip import PCCLIPModel
model = PCCLIPModel.from_pretrained("pc-clip-vit-l")
# Or use DCSMs (Dense Cosine Similarity Maps)How to detect: If query has multiple objects with different attributes → Use compositional model
---
Evolution Timeline
2021: CLIP Released
- Revolutionary: zero-shot, 400M image-text pairs
- Widely adopted for everything
- Limitations not yet understood
2022-2023: Limitations Discovered
- Cannot count objects
- Poor at fine-grained classification
- Fails spatial reasoning
- Can't bind attributes
2024: Alternatives Emerge
- DCSMs: Preserve patch/token topology
- PC-CLIP: Trained on pairwise comparisons
- SpLiCE: Sparse interpretable embeddings
2025: Current Best Practices
- Use CLIP for what it's good at
- Task-specific models for limitations
- Compositional models for complex queries
LLM Mistake: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.
---
Validation Script
Before using CLIP, check if it's appropriate:
python scripts/validate_clip_usage.py \
--query "your query here" \
--check-allReturns:
- ✅ CLIP is appropriate
- ❌ Use alternative (with suggestion)
Task-Specific Guidance
Image Search (CLIP ✓)
# Good use of CLIP
queries = ["beach", "mountain", "city skyline"]
# Works well for broad semantic conceptsZero-Shot Classification (CLIP ✓)
# Good: Broad categories
categories = ["indoor", "outdoor", "nature", "urban"]
# CLIP excels at thisObject Counting (CLIP ✗)
# Use object detection instead
from transformers import DetrImageProcessor, DetrForObjectDetection
# See /references/object_detection.mdFine-Grained Classification (CLIP ✗)
# Use specialized models
# See /references/fine_grained_models.mdSpatial Reasoning (CLIP ✗)
# Use spatial relation models
# See /references/spatial_models.md---
Troubleshooting
Issue: CLIP gives unexpected results
Check: 1. Is this a counting task? → Use object detection 2. Fine-grained classification? → Use specialized model 3. Spatial query? → Use spatial model 4. Multiple objects with attributes? → Use compositional model
Validation:
python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"Issue: Low similarity scores
Possible causes: 1. Query too specific (CLIP works better with broad concepts) 2. Fine-grained task (not CLIP's strength) 3. Need to adjust threshold
Solution: Try broader query or use alternative model
---
Model Selection Guide
| Model | Best For | Avoid For |
|---|---|---|
| CLIP ViT-L/14 | Semantic search, broad categories | Counting, fine-grained, spatial |
| DETR | Object detection, counting | Semantic similarity |
| DINOv2 | Fine-grained features | Text-image matching |
| PC-CLIP | Attribute binding, comparisons | General embedding |
| DCSMs | Compositional reasoning | Simple similarity |
Performance Notes
CLIP models:
- ViT-B/32: Fast, lower quality
- ViT-L/14: Balanced (recommended)
- ViT-g-14: Highest quality, slower
Inference time (single image, CPU):
- ViT-B/32: ~100ms
- ViT-L/14: ~300ms
- ViT-g-14: ~1000ms
Further Reading
/references/clip_limitations.md- Detailed analysis of CLIP's failures/references/alternatives.md- When to use what model/references/compositional_reasoning.md- DCSMs and PC-CLIP deep dive/scripts/validate_clip_usage.py- Pre-flight validation tool/scripts/diagnose_clip_issue.py- Debug unexpected results
---
See CHANGELOG.md for version history.
Changelog
All notable changes to the clip-aware-embeddings skill will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[1.3.0] - 2025-11-26
Changed
- Updated frontmatter to standard
allowed-toolsformat - Added activation keywords to description
Added
- MCP Integrations section (Firecrawl, Hugging Face)
- Moved inline changelog to separate CHANGELOG.md file
[1.2.0] - 2025-03-15
Added
- DCSMs and PC-CLIP alternatives
- Updated for 2025 best practices
- Improved validation scripts
[1.1.0] - 2024-06-10
Added
- Anti-pattern detection
- Expanded troubleshooting
[1.0.0] - 2024-01-15
Added
- Initial release
- CLIP usage decision tree
- Anti-patterns for counting, fine-grained, spatial, attribute binding
- Model selection guide
- Performance notes
- Reference documentation structure
#!/usr/bin/env python3
"""
CLIP Usage Validator - Checks if CLIP is appropriate for a given query
This demonstrates domain-specific validation that encodes expert knowledge.
"""
import sys
import re
from enum import Enum
from dataclasses import dataclass
from typing import Optional
class TaskType(Enum):
SEMANTIC_SEARCH = "semantic_search"
COUNTING = "counting"
FINE_GRAINED = "fine_grained"
SPATIAL = "spatial"
COMPOSITIONAL = "compositional"
ZERO_SHOT = "zero_shot"
@dataclass
class ValidationResult:
is_appropriate: bool
task_type: TaskType
confidence: float
reason: str
alternative: Optional[str] = None
class CLIPValidator:
"""Validates whether CLIP is appropriate for a given task."""
# Keywords that indicate specific task types
COUNTING_KEYWORDS = [
'how many', 'count', 'number of', 'total', 'quantity',
'several', 'few', 'multiple'
]
SPATIAL_KEYWORDS = [
'left', 'right', 'above', 'below', 'next to', 'beside',
'between', 'in front', 'behind', 'under', 'over', 'near'
]
FINE_GRAINED_DOMAINS = [
'celebrity', 'celebrities', 'actor', 'actress',
'car model', 'vehicle model', 'car make',
'flower species', 'bird species', 'dog breed',
'person', 'face', 'people'
]
COMPOSITIONAL_PATTERNS = [
r'(\w+)\s+(\w+)\s+and\s+(\w+)\s+(\w+)', # "red car and blue truck"
r'both\s+',
r'neither\s+',
r'either\s+',
]
GOOD_USE_CASES = [
'find images', 'search for', 'similar to', 'looks like',
'classify', 'categorize', 'what is this', 'identify',
'semantic', 'concept', 'theme'
]
def validate(self, query: str) -> ValidationResult:
"""
Validate if CLIP is appropriate for the query.
Args:
query: Natural language query
Returns:
ValidationResult with recommendation
"""
query_lower = query.lower()
# Check for counting tasks
if any(kw in query_lower for kw in self.COUNTING_KEYWORDS):
return ValidationResult(
is_appropriate=False,
task_type=TaskType.COUNTING,
confidence=0.95,
reason="Query requires counting objects. CLIP cannot preserve spatial information needed for counting.",
alternative="Use object detection models: DETR, Faster R-CNN, YOLO"
)
# Check for spatial reasoning
if any(kw in query_lower for kw in self.SPATIAL_KEYWORDS):
return ValidationResult(
is_appropriate=False,
task_type=TaskType.SPATIAL,
confidence=0.90,
reason="Query requires spatial understanding. CLIP's embeddings lose spatial topology.",
alternative="Use spatial reasoning models: GQA, SWIG, Visual Genome models"
)
# Check for fine-grained classification
if any(domain in query_lower for domain in self.FINE_GRAINED_DOMAINS):
return ValidationResult(
is_appropriate=False,
task_type=TaskType.FINE_GRAINED,
confidence=0.85,
reason="Query requires fine-grained classification. CLIP trained on coarse categories.",
alternative="Use specialized models: Fine-tuned ResNet/EfficientNet for the specific domain"
)
# Check for compositional reasoning
if any(re.search(pattern, query_lower) for pattern in self.COMPOSITIONAL_PATTERNS):
return ValidationResult(
is_appropriate=False,
task_type=TaskType.COMPOSITIONAL,
confidence=0.80,
reason="Query requires attribute binding. CLIP cannot bind attributes to specific objects.",
alternative="Use compositional models: DCSMs (Dense Cosine Similarity Maps), PC-CLIP"
)
# Check if it's a good CLIP use case
if any(use_case in query_lower for use_case in self.GOOD_USE_CASES):
return ValidationResult(
is_appropriate=True,
task_type=TaskType.SEMANTIC_SEARCH,
confidence=0.90,
reason="Query is appropriate for CLIP: semantic search or broad categorization.",
alternative=None
)
# Default: probably okay but lower confidence
return ValidationResult(
is_appropriate=True,
task_type=TaskType.ZERO_SHOT,
confidence=0.60,
reason="Query appears suitable for CLIP, but verify results carefully.",
alternative="If results are poor, consider task-specific models"
)
def print_result(query: str, result: ValidationResult):
"""Pretty-print validation results."""
print("\n" + "="*70)
print(f"CLIP USAGE VALIDATION")
print("="*70)
print(f"\nQuery: {query}")
print(f"Task Type: {result.task_type.value}")
print(f"Confidence: {result.confidence:.0%}")
print()
if result.is_appropriate:
print("✅ CLIP IS APPROPRIATE")
print(f"\nReason: {result.reason}")
if result.alternative:
print(f"\n💡 Note: {result.alternative}")
else:
print("❌ CLIP IS NOT APPROPRIATE")
print(f"\nReason: {result.reason}")
print(f"\n💡 Use Instead: {result.alternative}")
print("\n" + "="*70 + "\n")
def run_examples():
"""Run validation on example queries."""
examples = [
"Find images of beaches at sunset",
"How many cars are in this image?",
"Identify which celebrity this is",
"Is the cat to the left or right of the dog?",
"Find images with a red car and a blue truck",
"Classify this image as indoor or outdoor",
]
validator = CLIPValidator()
print("\n" + "="*70)
print("EXAMPLE VALIDATIONS")
print("="*70)
for query in examples:
result = validator.validate(query)
print(f"\n{query}")
print(f" → {'✅ CLIP' if result.is_appropriate else '❌ Alternative'}: {result.task_type.value}")
if not result.is_appropriate:
print(f" → {result.alternative}")
print("\n" + "="*70 + "\n")
def main():
if len(sys.argv) < 2:
print("Usage:")
print(" python validate_clip_usage.py 'your query here'")
print(" python validate_clip_usage.py --examples")
print("\nExample:")
print(" python validate_clip_usage.py 'Find images of mountains'")
sys.exit(1)
if sys.argv[1] == '--examples':
run_examples()
return
query = ' '.join(sys.argv[1:])
validator = CLIPValidator()
result = validator.validate(query)
print_result(query, result)
# Exit code: 0 if appropriate, 1 if not
sys.exit(0 if result.is_appropriate else 1)
if __name__ == '__main__':
main()
Related skills
Microsoft FoundryDeploy, evaluate, and continuously improve Microsoft Foundry agents from a single agent interface.490k1.3k
Ai Research ReproductionOrchestrate trustworthy, auditable reproduction of deep learning repositories directly from their READMEs.176k511Run TrainSafely execute selected deep learning training commands with standardized evidence capture.176k511Explore RunSafely run isolated exploratory experiments with clear recording and conservative selection before committing changes.176k511Paper Context ResolverFetch precise reproduction-critical details like dataset splits, preprocessing steps, or evaluation protocols from the original academic paper when the repo README leav141k511Repo Intake And PlanScan unfamiliar AI research repositories and receive a minimal, trustworthy reproduction target before investing significant time.140k511