Multimodal Models

Name: Multimodal Models
Author: eyadsibai

eyadsibai/ltk

56 installs
7 repo stars
Updated January 15, 2026
eyadsibai/ltk

Helps with ai & agent building tasks.

About

multimodal-models is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.

multimodal-models
AI & Agent Building
AI-coding skill

Multimodal Models by the numbers

56 all-time installs (skills.sh)
Ranked #6,668 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
Data as of Jul 30, 2026 (Skillselion catalog sync)

npx skills add https://github.com/eyadsibai/ltk --skill multimodal-models

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

[![Listed on Skillselion](https://skillselion.com/badge/skills/eyadsibai/ltk/multimodal-models.svg)](https://skillselion.com/skills/eyadsibai/ltk/multimodal-models)

Installs	56
repo stars	★ 7
Last updated	January 15, 2026
Repository	eyadsibai/ltk ↗

What it does

Helps with ai & agent building tasks.

Files

SKILL.mdMarkdownGitHub ↗

Multimodal Models

Pre-trained models for vision, audio, and cross-modal tasks.

---

Model Overview

Model	Modality	Task
CLIP	Image + Text	Zero-shot classification, similarity
Whisper	Audio → Text	Transcription, translation
Stable Diffusion	Text → Image	Image generation, editing

---

CLIP (Vision-Language)

Zero-shot image classification without training on specific labels.

CLIP Use Cases

Task	How
Zero-shot classification	Compare image to text label embeddings
Image search	Find images matching text query
Content moderation	Classify against safety categories
Image similarity	Compare image embeddings

CLIP Models

Model	Parameters	Trade-off
ViT-B/32	151M	Recommended balance
ViT-L/14	428M	Best quality, slower
RN50	102M	Fastest, lower quality

CLIP Concepts

Concept	Description
Dual encoder	Separate encoders for image and text
Contrastive learning	Trained to match image-text pairs
Normalization	Always normalize embeddings before similarity
Descriptive labels	Better labels = better zero-shot accuracy

Key concept: CLIP embeds images and text in same space. Classification = find nearest text embedding.

CLIP Limitations

Not for fine-grained classification
No spatial understanding (whole image only)
May reflect training data biases

---

Whisper (Speech Recognition)

Robust multilingual transcription supporting 99 languages.

Whisper Use Cases

Task	Configuration
Transcription	Default `transcribe` task
Translation to English	`task="translate"`
Subtitles	Output format SRT/VTT
Word timestamps	`word_timestamps=True`

Whisper Models

Model	Size	Speed	Recommendation
turbo	809M	Fast	Recommended
large	1550M	Slow	Maximum quality
small	244M	Medium	Good balance
base	74M	Fast	Quick tests
tiny	39M	Fastest	Prototyping only

Whisper Concepts

Concept	Description
Language detection	Auto-detects, or specify for speed
Initial prompt	Improves technical terms accuracy
Timestamps	Segment-level or word-level
faster-whisper	4× faster alternative implementation

Key concept: Specify language when known—auto-detection adds latency.

Whisper Limitations

May hallucinate on silence/noise
No speaker diarization (who said what)
Accuracy degrades on >30 min audio
Not suitable for real-time captioning

---

Stable Diffusion (Image Generation)

Text-to-image generation with various control methods.

SD Use Cases

Task	Pipeline
Text-to-image	`DiffusionPipeline`
Style transfer	`Image2Image`
Fill regions	`Inpainting`
Guided generation	`ControlNet`
Custom styles	LoRA adapters

SD Models

Model	Resolution	Quality
SDXL	1024×1024	Best
SD 1.5	512×512	Good, faster
SD 2.1	768×768	Middle ground

Key Parameters

Parameter	Effect	Typical Value
num_inference_steps	Quality vs speed	20-50
guidance_scale	Prompt adherence	7-12
negative_prompt	Avoid artifacts	"blurry, low quality"
strength (img2img)	How much to change	0.5-0.8
seed	Reproducibility	Fixed number

Control Methods

Method	Input	Use Case
ControlNet	Edge/depth/pose	Structural guidance
LoRA	Trained weights	Custom styles
Img2Img	Source image	Style transfer
Inpainting	Image + mask	Fill regions

Memory Optimization

Technique	Effect
CPU offload	Reduces VRAM usage
Attention slicing	Trades speed for memory
VAE tiling	Large image support
xFormers	Faster attention
DPM scheduler	Fewer steps needed

Key concept: Use SDXL for quality, SD 1.5 for speed. Always use negative prompts.

SD Limitations

GPU strongly recommended (CPU very slow)
Large VRAM requirements for SDXL
May generate anatomical errors
Prompt engineering matters

---

Common Patterns

Embedding and Similarity

All three models use embeddings:

CLIP: Image/text embeddings for similarity
Whisper: Audio embeddings for transcription
SD: Text embeddings for image conditioning

GPU Acceleration

Model	VRAM Needed
CLIP ViT-B/32	~2 GB
Whisper turbo	~6 GB
SD 1.5	~6 GB
SDXL	~10 GB

Best Practices

Practice	Why
Use recommended model sizes	Best quality/speed balance
Cache embeddings (CLIP)	Expensive to recompute
Specify language (Whisper)	Faster than auto-detect
Use negative prompts (SD)	Avoid common artifacts
Set seeds for reproducibility	Consistent results