
System Design Interrogation
- 13 installs
- 213 repo stars
- Updated August 4, 2026
- yonatangross/orchestkit
Helps with design & ui/ux tasks.
About
system-design-interrogation is a Claude Code skill for design & ui/ux. It helps solo builders move faster with AI-assisted coding.
- system-design-interrogation
- Design & UI/UX
- AI-coding skill
System Design Interrogation by the numbers
- 13 all-time installs (skills.sh)
- Ranked #1,418 of 1,880 Design & UI/UX skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/yonatangross/orchestkit --skill system-design-interrogationAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 13 |
|---|---|
| repo stars | ★ 213 |
| Last updated | August 4, 2026 |
| Repository | yonatangross/orchestkit ↗ |
What it does
Helps with design & ui/ux tasks.
Files
System Design Interrogation
The Problem
Rushing to implementation without systematic design thinking leads to:
- Scalability issues discovered too late
- Security holes from missing tenant isolation
- Data model mismatches
- Frontend/backend contract conflicts
- Poor user experience
The Solution: Question Before Implementing
┌────────────────────────────────────────────────────────────────────────────┐
│ SYSTEM DESIGN INTERROGATION │
├────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ │
│ │ FEATURE │ │
│ │ REQUEST │ │
│ └──────┬──────┘ │
│ │ │
│ ┌──────────────────────────┼──────────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ SCALE │ │ DATA │ │SECURITY│ │
│ └───┬────┘ └───┬────┘ └───┬────┘ │
│ │ │ │ │
│ • Users? • Where? • Who access? │
│ • Volume? • Pattern? • Isolation? │
│ • Growth? • Search? • Attacks? │
│ │ │ │ │
│ └──────────────────────┼───────────────────────┘ │
│ │ │
│ ┌────────────────────────┼────────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌────────┐ ┌──────────┐ ┌────────┐ │
│ │ UX │ │COHERENCE │ │ TRADE- │ │
│ └───┬────┘ └────┬─────┘ │ OFFS │ │
│ │ │ └───┬────┘ │
│ • Latency? • Contracts? • Speed? │
│ • Feedback? • Types? • Quality? │
│ • Errors? • API? • Cost? │
│ │ │ │ │
│ └─────────────────────┴──────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────┐ │
│ │ IMPLEMENTATION│ │
│ │ READY │ │
│ └───────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────────┘The Five Dimensions
1. Scale
Key Questions:
- How many users/tenants will use this?
- What's the expected data volume (now and in 1 year)?
- What's the request rate? Read-heavy or write-heavy?
- Does complexity grow linearly or exponentially with data?
- What happens at 10x current load? 100x?
OrchestKit Example:
Feature: "Add document tagging"
- Users: 1000 active users
- Documents per user: ~50 average
- Tags per document: 3-5
- Total tags: 50,000 → 500,000
- Access: Read-heavy (10:1 read:write)
- Search: Need tag autocomplete (prefix search)2. Data
Key Questions:
- Where does this data naturally belong?
- What's the primary access pattern?
- Is it master data or transactional?
- What's the retention policy?
- Does it need to be searchable? How?
OrchestKit Example:
Feature: "Add document tagging"
- Data: Tags belong WITH documents (denormalized) or separate table?
- Pattern: Get tags for document (by doc_id), get documents by tag
- Storage: PostgreSQL (relational) or add to document JSON?
- Search: Full-text for tag names, filter by tag for documents
- Decision: Separate `tags` table with many-to-many join3. Security
Key Questions:
- Who can access this data/feature?
- How is tenant isolation enforced?
- What happens if authorization fails?
- What attack vectors does this introduce?
- Is there PII involved?
OrchestKit Example:
Feature: "Add document tagging"
- Access: User can only see/manage their own tags
- Isolation: All tag queries MUST include tenant_id filter
- AuthZ: Check user owns document before tagging
- Attacks: Tag injection? Limit tag length, sanitize input
- PII: Tags might contain PII → treat as sensitive4. UX Impact
Key Questions:
- What's the expected latency for this operation?
- What feedback does the user get during the operation?
- What happens on failure? Can they retry?
- Is there optimistic UI possible?
- How does this affect the overall workflow?
OrchestKit Example:
Feature: "Add document tagging"
- Latency: < 100ms for add/remove tag
- Feedback: Optimistic update, show tag immediately
- Failure: Rollback tag, show error toast
- Optimistic: Yes - add tag to UI before server confirms
- Workflow: Tags should be inline editable, no modal5. Coherence
Key Questions:
- Which layers does this touch?
- What contracts/interfaces change?
- Are types consistent frontend ↔ backend?
- Does this break existing clients?
- How does this affect the API?
OrchestKit Example:
Feature: "Add document tagging"
- Layers: DB → Backend API → Frontend UI → State
- Contracts: Document type needs `tags: Tag[]` field
- Types: Tag = { id: UUID, name: string, color?: string }
- Breaking: No - additive change to Document response
- API: POST /documents/{id}/tags, DELETE /documents/{id}/tags/{tag_id}The Process
Before Writing Any Code
1. State the Feature - One sentence description 2. Run Through 5 Dimensions - Answer key questions for each 3. Identify Trade-offs - Speed vs quality, complexity vs flexibility 4. Document Decisions - Record answers in design doc or issue 5. Review with Team - Get alignment before implementing
Quick Assessment Template
## Feature: [Name]
### Scale
- Users:
- Data volume:
- Access pattern:
- Growth projection:
### Data
- Storage location:
- Schema changes:
- Search requirements:
- Retention:
### Security
- Authorization:
- Tenant isolation:
- Attack surface:
- PII handling:
### UX
- Target latency:
- Feedback mechanism:
- Error handling:
- Optimistic updates:
### Coherence
- Affected layers:
- Type changes:
- API changes:
- Breaking changes:
### Decision
[Final approach with rationale]Integration with OrchestKit Workflow
In Brainstorming Phase
Before implementation, run system design interrogation:
/brainstorm → System Design Questions → Implementation PlanIn Code Review
Reviewer should verify:
- Scale considerations documented
- Security layer covered
- Types consistent across stack
- UX states handled
In Testing
Tests should cover:
- Scale: Load tests for expected volume
- Security: Tenant isolation tests
- Coherence: Integration tests across layers
- UX: Error state tests
Anti-Patterns
❌ "I'll add an index later if it's slow"
→ Ask: What's the expected query pattern NOW?
❌ "We can add tenant filtering in a future PR"
→ Ask: How is isolation enforced from DAY ONE?
❌ "The frontend can handle any response shape"
→ Ask: What's the TypeScript type for this?
❌ "Users won't do that"
→ Ask: What's the attack vector? What if they DO?
❌ "It's just a small feature"
→ Ask: How does this grow with 100x users?Quick Reference Card
| Dimension | Key Question | Red Flag |
|---|---|---|
| Scale | How many? | "All users" |
| Data | Where stored? | "I'll figure it out" |
| Security | Who can access? | "Everyone" |
| UX | What's the latency? | "It'll be fast" |
| Coherence | What types change? | "No changes needed" |
---
Version: 1.0.0 (December 2025)
Related Skills
brainstorming- Transform rough ideas into designs before applying system design interrogationarchitecture-decision-record- Document key decisions discovered during interrogationexplore- Deep codebase exploration to understand existing architecture before planningverify- Comprehensive feature verification after implementation
Key Decisions
| Decision | Choice | Rationale |
|---|---|---|
| Dimensions count | Five (Scale, Data, Security, UX, Coherence) | Covers all critical architectural concerns without overlap |
| Process timing | Before any code | Prevents costly rework from missed requirements |
| Question format | Structured templates | Ensures consistent coverage, prevents omissions |
| Documentation | Markdown template | Portable, version-controlled, reviewable |
| Integration | Pairs with brainstorming | Brainstorming explores options, interrogation validates choice |
Capability Details
scale-assessment
Keywords: scale, load, traffic, users, concurrent, throughput Solves:
- How many users will this feature serve?
- What's the expected request rate?
- How does this scale with data growth?
data-architecture
Keywords: data, storage, database, schema, migration, structure Solves:
- Where should this data live?
- What's the access pattern?
- How does this affect existing schemas?
security-considerations
Keywords: security, auth, permission, tenant, isolation, attack Solves:
- What are the security implications?
- How is tenant isolation maintained?
- What attack vectors exist?
coherence-validation
Keywords: coherence, consistency, contract, interface, integration Solves:
- How does this fit the existing architecture?
- What contracts need updating?
- Are frontend/backend aligned?
ux-impact
Keywords: ux, user experience, latency, feedback, error Solves:
- What's the user experience impact?
- How long will users wait?
- What feedback do they get?
Before Implementation Checklist
Quick Assessment (5 min)
Before writing any code, answer these questions:
1. Scale (1 min)
- [ ] How many users? ___
- [ ] How much data? ___
- [ ] Read or write heavy? ___
- [ ] Will it scale 10x? ___
2. Data (1 min)
- [ ] Where does data live? ___
- [ ] What's the access pattern? ___
- [ ] Need search capability? ___
- [ ] Schema changes needed? ___
3. Security (1 min)
- [ ] Who can access? ___
- [ ] Tenant isolation how? ___
- [ ] Attack vectors? ___
- [ ] PII involved? ___
4. UX (1 min)
- [ ] Expected latency? ___
- [ ] Loading state? ___
- [ ] Error handling? ___
- [ ] Optimistic updates? ___
5. Coherence (1 min)
- [ ] Types defined all layers? ___
- [ ] API contract clear? ___
- [ ] Breaking changes? ___
- [ ] Migration needed? ___
---
Detailed Assessment (15 min)
Scale Deep Dive
Current baseline:
- Users: ___
- Data per user: ___
- Total data: ___
- Requests/day: ___
At 10x:
- Can the query handle it? [ ]
- Is there an index for this? [ ]
- What's the memory footprint? [ ]
- What breaks first? ___Data Design
Data model:
- Entity name: ___
- Belongs to: ___
- Related to: ___
- Access pattern: ___
Storage decision:
[ ] Same table (denormalize)
[ ] New table (normalize)
[ ] JSON field (flexible)
[ ] Vector column (embeddings)
Search requirements:
[ ] None
[ ] Full-text
[ ] Vector similarity
[ ] Filter/sortSecurity Design
Authorization:
- Who: ___
- What action: ___
- On what resource: ___
- Enforced where: ___
Tenant isolation:
[ ] Query has tenant_id filter
[ ] tenant_id from JWT, not request
[ ] Test exists for cross-tenant
Attack surface:
[ ] Input validated
[ ] Output sanitized
[ ] Error messages safe
[ ] Rate limitedUX Design
User flow:
1. User clicks ___
2. UI shows ___
3. Request takes ___ms
4. Response shows ___
States:
[ ] Loading
[ ] Success
[ ] Error
[ ] Empty
[ ] Partial (paginated)
Optimistic updates:
[ ] Not applicable
[ ] Update immediately, rollback on error
[ ] Show pending stateCoherence Check
Layers affected:
[ ] Database (migration)
[ ] Backend model (SQLAlchemy)
[ ] API schema (Pydantic)
[ ] Frontend types (TypeScript)
[ ] UI components (React)
[ ] Tests (all layers)
Contract:
- Endpoint: ___
- Method: ___
- Request body: ___
- Response: ___
- Error codes: ___---
Decision Documentation
## Feature: [Name]
### Summary
[One sentence description]
### Decisions
**Scale:** [How it handles growth]
**Data:** [Where and how stored]
**Security:** [Who can access, how enforced]
**UX:** [User experience approach]
**Coherence:** [Cross-layer consistency plan]
### Trade-offs
[What we're optimizing for, what we're accepting]
### Implementation Order
1. ___
2. ___
3. ___
### Sign-off
- [ ] Developer reviewed
- [ ] Answers satisfy requirements
- [ ] Ready to implement---
Stop Signs
STOP and get help if:
- [ ] "I don't know how many users"
- [ ] "I'm not sure where data goes"
- [ ] "Authorization is complicated"
- [ ] "The types don't match"
- [ ] "This might break existing clients"
Proceed with caution if:
- [ ] No tests exist for this area
- [ ] Schema migration required
- [ ] Affects hot path
- [ ] Involves PII
---
Quick Templates
Simple Feature
Scale: Single user, small data
Data: Existing table, add column
Security: User owns resource
UX: < 100ms, inline update
Coherence: Add field to existing typesComplex Feature
Scale: All users, growing data
Data: New table with relationships
Security: Role-based access
UX: Loading state, pagination
Coherence: New types, new endpointsLLM Feature
Scale: Token costs at 10x
Data: Context separation (no IDs in prompt)
Security: Tenant isolation in retrieval
UX: Streaming response
Coherence: Langfuse tracingCoherence Questions
Purpose
Coherence questions ensure consistency across the entire stack - from database to UI.
The Coherence Matrix
┌────────────────────────────────────────────────────────────────────────────┐
│ COHERENCE MATRIX │
├────────────────────────────────────────────────────────────────────────────┤
│ │
│ Frontend API Backend Database │
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ Types │ TS Types │ OpenAPI │ Pydantic │ SQLAlchemy│ ← Must match │
│ ├──────────┼──────────┼──────────┼──────────┤ │
│ State │ Zustand │ Request │ Workflow │ Tables │ ← Consistent │
│ ├──────────┼──────────┼──────────┼──────────┤ │
│ Errors │ UI Toast │ HTTP 4xx │ Exception│ Constraint│ ← Mapped │
│ ├──────────┼──────────┼──────────┼──────────┤ │
│ IDs │ Display │ Path/Body│ Context │ PK/FK │ ← Same format │
│ └──────────┴──────────┴──────────┴──────────┘ │
│ │
│ ▲ ▲ ▲ ▲ │
│ │ │ │ │ │
│ └───────────┴───────────┴───────────┘ │
│ CONTRACT │
│ │
└────────────────────────────────────────────────────────────────────────────┘Core Questions
Type Consistency
| Question | Why Ask | Check |
|---|---|---|
| What's the TypeScript type? | Frontend consistency | Interface defined |
| What's the Pydantic model? | API validation | Request/Response models |
| What's the SQLAlchemy model? | ORM mapping | Table model exists |
| Do all three match? | Contract alignment | Fields sync'd |
| Are there any optional/required mismatches? | Runtime errors | Same nullability |
Contract Questions
| Question | Why Ask | OrchestKit Example |
|---|---|---|
| What API endpoints change? | Frontend updates | GET /analyses returns new field |
| Is this a breaking change? | Client compatibility | Adding field = non-breaking |
| What's the migration path? | Deployment order | DB first, then API, then UI |
| Do existing clients handle this? | Backwards compat | Mobile app? Browser cache? |
| Is versioning needed? | API stability | v1 → v2 for breaking changes |
State Consistency
| Question | Why Ask | Check |
|---|---|---|
| Where is state managed? | Single source of truth | DB vs cache vs local |
| What triggers state updates? | Reactivity | Events, polling, SSE |
| How is stale state handled? | Data freshness | Cache invalidation |
| What's the optimistic update strategy? | UX responsiveness | Rollback on failure |
| Are there race conditions? | Concurrent updates | Locking strategy |
OrchestKit Layer Alignment
Adding a New Field
Feature: Add "status" field to Analysis
STEP 1: Database
- Add column: ALTER TABLE analyses ADD status VARCHAR(20)
- Default value: 'pending'
- Migration: Alembic script
STEP 2: Backend Model
class Analysis(Base):
status: Mapped[str] = mapped_column(default="pending")
STEP 3: Pydantic Schema
class AnalysisResponse(BaseModel):
status: str = "pending"
STEP 4: Frontend Type
interface Analysis {
status: 'pending' | 'processing' | 'complete' | 'failed';
}
STEP 5: UI Component
<StatusBadge status={analysis.status} />
CHECK:
□ All types use same values
□ Default value consistent
□ Enum values documented
□ Old records handled (migration fills default)Adding a New Endpoint
Feature: POST /analyses/{id}/retry
STEP 1: API Route
@router.post("/analyses/{id}/retry")
async def retry_analysis(id: UUID, ctx: RequestContext):
...
STEP 2: OpenAPI Docs
- Document request/response
- Add to API reference
STEP 3: Frontend API Client
api.analyses.retry(id: string): Promise<Analysis>
STEP 4: UI Action
<Button onClick={() => retryAnalysis(id)}>Retry</Button>
STEP 5: State Update
- Invalidate analysis query
- Show optimistic state
CHECK:
□ Error responses documented
□ Loading state in UI
□ Error handling in UI
□ Audit logging on backendCross-Stack Type Generation
┌─────────────────────────────────────────────────────────────┐
│ IDEAL: Single Source of Truth for Types │
├─────────────────────────────────────────────────────────────┤
│ │
│ Pydantic Model ──► OpenAPI Schema ──► TypeScript │
│ │ │ │
│ │ │ │
│ ▼ ▼ │
│ Backend uses Frontend uses │
│ same model generated types │
│ │
│ Tools: │
│ - openapi-typescript-codegen │
│ - FastAPI's automatic OpenAPI generation │
│ - Zod from OpenAPI (for runtime validation) │
│ │
└─────────────────────────────────────────────────────────────┘Red Flags
⚠️ "The frontend will just use 'any'"
→ Type safety exists for a reason. Define the type.
⚠️ "We'll update the other layer later"
→ Layers must change together. Same PR or linked PRs.
⚠️ "It's close enough"
→ 'createdAt' vs 'created_at' causes runtime errors.
⚠️ "The backend handles nulls"
→ If frontend sends null and backend expects string, it breaks.
⚠️ "We don't have time for migration"
→ Migration now is cheaper than data cleanup later.Checklist by Change Type
Adding a Field
- [ ] Database migration written
- [ ] SQLAlchemy model updated
- [ ] Pydantic schema updated
- [ ] TypeScript interface updated
- [ ] UI component handles new field
- [ ] Tests updated all layers
- [ ] Old data handled (default or migration)
Removing a Field
- [ ] No consumers depend on field
- [ ] Deprecation warning added (if public API)
- [ ] Frontend stops sending field
- [ ] Backend stops requiring field
- [ ] Database column nullable/dropped
- [ ] Documentation updated
Changing a Field Type
- [ ] Migration plan for existing data
- [ ] Backwards compatibility period
- [ ] All layers updated simultaneously
- [ ] Tests cover both old and new format
- [ ] Rollback plan documented
Example Assessment
## Feature: Change analysis.difficulty from string to enum
### Current State
- DB: VARCHAR(255) with free-form text
- Backend: str field
- Frontend: string type
### Target State
- DB: VARCHAR(20) with check constraint
- Backend: Enum field
- Frontend: union type
### Migration Plan
1. Add new enum values to backend (week 1)
- Accept both string and enum
- Normalize on save
2. Migrate existing data (week 1)
- Map "easy" → "beginner"
- Map "hard" → "advanced"
- Map unknown → "intermediate"
3. Update frontend (week 2)
- Use new enum values
- Update difficulty selector
4. Remove string support (week 3)
- Backend rejects non-enum
- Add DB constraint
### Coherence Check
□ All layers use same enum values
□ Migration handles edge cases
□ Tests cover all values
□ Old cached responses handled (TTL or clear)Scale Questions
Purpose
Scale questions prevent building features that work for 10 users but break at 10,000.
Question Framework
Volume Questions
┌─────────────────────────────────────────────────────────────┐
│ SCALE ASSESSMENT │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ USERS │ │ DATA │ │ REQUESTS │ │
│ └─────┬───────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ How many? How much? How often? │
│ Concurrent? Per user? Read vs write? │
│ Growth rate? Total? Peak times? │
│ │ │ │ │
│ └──────────────────┼──────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────┐ │
│ │ CAPACITY │ │
│ │ PLANNING │ │
│ └───────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘Core Questions
| Question | Why Ask | Example Answer |
|---|---|---|
| How many users will use this? | DB index strategy | 1,000 active users |
| What's the data volume per user? | Storage planning | 50 docs/user |
| What's the total expected data? | Shard planning | 50K → 500K docs |
| Is it read-heavy or write-heavy? | Cache strategy | 10:1 read:write |
| What's the request rate? | Rate limiting | 100 req/sec peak |
| What's the growth projection? | Future-proofing | 3x/year |
Growth Projections
QUESTION: At 10x scale, does this still work?
Current: 1,000 users, 50K documents
10x: 10,000 users, 500K documents
100x: 100,000 users, 5M documents
Check:
□ Can the DB handle this query pattern at 100x?
□ Does the algorithm scale linearly or exponentially?
□ What's the memory footprint at 100x?
□ Does the response time stay acceptable?OrchestKit Scale Considerations
Current Baseline
| Resource | Current | 10x Target |
|---|---|---|
| Active Users | 1,000 | 10,000 |
| Documents | 50,000 | 500,000 |
| Analyses/month | 5,000 | 50,000 |
| Vector searches/day | 10,000 | 100,000 |
| LLM calls/day | 2,000 | 20,000 |
Feature-Specific Questions
For Search Features:
- How many documents will be searched?
- Is real-time indexing required?
- What's acceptable search latency? (<100ms? <500ms?)
- Can results be cached? For how long?
For LLM Features:
- What's the token budget per request?
- What's the acceptable latency? (streaming vs batch)
- Can responses be cached?
- What's the cost at 10x scale?
For Data Processing:
- Batch or real-time?
- What's the processing time per item?
- Can it be parallelized?
- What's the failure/retry strategy?
Red Flags
⚠️ "It works fine in development"
→ Development has 100 records. Production has 100,000.
⚠️ "We'll optimize later"
→ Fundamental design issues can't be optimized away.
⚠️ "Users won't create that much data"
→ Power users ALWAYS create more than expected.
⚠️ "The database can handle it"
→ Without indexes and proper queries, it can't.
⚠️ "We can add caching"
→ Caching doesn't fix O(n²) algorithms.Decision Framework
When to Optimize Now
- [ ] Feature touches hot path (every request)
- [ ] Data grows with user activity (not just user count)
- [ ] Algorithm complexity is O(n²) or worse
- [ ] External API call per item (LLM, third-party)
When to Defer
- [ ] Feature is rarely used (<1% of requests)
- [ ] Data is bounded (e.g., settings, not user content)
- [ ] Simple optimization is obvious and easy
- [ ] No external dependencies in loop
Example Assessment
## Feature: Full-text search on analyses
### Scale Assessment
**Current state:**
- 5,000 analyses total
- 50 searches/day
- Simple LIKE query on title
**At 10x:**
- 50,000 analyses
- 500 searches/day
- LIKE query becomes slow (no index used)
**Decision:**
- Add PostgreSQL full-text search with GIN index
- ts_vector column on analyses table
- Index on (tenant_id, search_vector)
- Cache frequent searches (5 min TTL)
**Why now:**
- Full-text search is core feature
- LIKE won't scale past 10K records
- Retrofitting search is expensiveSecurity Questions
Purpose
Security questions prevent authorization bypasses, data leaks, and tenant isolation failures.
Question Framework
┌─────────────────────────────────────────────────────────────┐
│ SECURITY ASSESSMENT │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ WHO │ │ WHAT │ │ HOW │ │
│ └─────┬───────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ Who can access? What data? How enforced? │
│ Roles/perms? Sensitivity? At what layer? │
│ Tenant scope? PII involved? Audit trail? │
│ │ │ │ │
│ └──────────────────┼──────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ ATTACKS │ │ FALLBACK │ │ AUDIT │ │
│ └─────┬───────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ What vectors? What if fails? What to log? │
│ Injection? Deny by default? Who reviews? │
│ IDOR? Error messages? Retention? │
│ │ │ │ │
│ └──────────────────┴──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘Core Questions
Authorization
| Question | Why Ask | OrchestKit Example |
|---|---|---|
| Who can access this? | Define authZ rules | Owner only, or shared? |
| What roles have access? | RBAC implementation | admin, user, viewer |
| Is tenant isolation enforced? | Multi-tenant security | Every query has tenant_id |
| What if auth fails? | Error handling | 403 Forbidden, not 500 |
| Can users access others' data? | IDOR prevention | Check owner before access |
Data Sensitivity
| Question | Why Ask | OrchestKit Example |
|---|---|---|
| Does this involve PII? | Compliance (GDPR) | User email in analysis? |
| What's the sensitivity level? | Encryption needs | API keys = highest |
| Should it be logged? | Audit vs privacy | Log access, not content |
| What's the retention policy? | Data lifecycle | Delete after 90 days? |
| Who can export this? | Data exfiltration | Only admins can bulk export |
Attack Vectors
| Vector | Question | Mitigation |
|---|---|---|
| Injection | User input in query? | Parameterized queries |
| IDOR | ID in URL/body? | Check ownership |
| XSS | User content displayed? | Sanitize output |
| CSRF | State-changing action? | CSRF tokens |
| Privilege escalation | Role in request? | Server-side role check |
OrchestKit-Specific Checks
Multi-Tenant Isolation
EVERY database query MUST answer:
□ Does the query include tenant_id filter?
□ Is tenant_id from JWT (not user input)?
□ Is there a test for cross-tenant access?
□ Is RLS enabled on the table?LLM Security
For any LLM feature, verify:
□ No user_id/tenant_id in prompt
□ No document_id/analysis_id in prompt
□ Output validated for hallucinated IDs
□ User content sanitized before prompt
□ PII detection on input/outputAPI Endpoint Security
# Every endpoint should have:
@router.get("/analyses/{id}")
async def get_analysis(
id: UUID,
ctx: RequestContext = Depends(get_request_context), # ✓ Auth
db: AsyncSession = Depends(get_db),
):
# ✓ Tenant isolation in query
analysis = await db.execute(
"""
SELECT * FROM analyses
WHERE id = :id
AND tenant_id = :tenant_id -- REQUIRED
AND user_id = :user_id -- For user-owned resources
""",
{"id": id, "tenant_id": ctx.tenant_id, "user_id": ctx.user_id}
)
if not analysis:
raise HTTPException(404) # ✓ Don't leak existence
# ✓ Audit log
logger.audit("analysis.accessed", analysis_id=id, user_id=ctx.user_id)
return analysisSecurity Checklist by Layer
Layer 1: Input Validation
- [ ] All inputs validated with Pydantic
- [ ] Size limits on text fields
- [ ] File upload validation (type, size)
- [ ] Rate limiting configured
Layer 2: Authorization
- [ ] Every endpoint has auth check
- [ ] Permission check before action
- [ ] Resource ownership verified
- [ ] Admin actions require admin role
Layer 3: Data Access
- [ ] All queries parameterized
- [ ] Tenant filter on every query
- [ ] No raw SQL with user input
- [ ] Sensitive data encrypted
Layer 4: Output
- [ ] Error messages don't leak info
- [ ] PII redacted from logs
- [ ] Sensitive fields not in response
- [ ] Headers configured (CORS, CSP)
Red Flags
⚠️ "It's an internal API"
→ Internal APIs get exposed. Secure by default.
⚠️ "Only admins will use this"
→ Admin credentials get compromised. Least privilege.
⚠️ "We trust the frontend"
→ Never trust client input. Server validates everything.
⚠️ "It's behind authentication"
→ AuthN ≠ AuthZ. Just because they're logged in...
⚠️ "That would never happen"
→ Assume attackers WILL try everything.Example Assessment
## Feature: Share analysis with team members
### Security Assessment
**Who can access:**
- Owner can share
- Recipients can view (not edit)
- Only within same tenant
**Attack vectors:**
- IDOR: User tries to share to user in different tenant
→ Check recipient is in same tenant
- Privilege escalation: Viewer tries to edit
→ Check permission on every action
- Information disclosure: Shared analysis leaks via URL
→ Use non-guessable share tokens
**Implementation:**
- Share creates ShareToken with expiry
- ShareToken scoped to tenant
- Recipient verified in same tenant
- Read-only access flag on share
- Audit log on share creation and access
**Tests required:**
- Cross-tenant share blocked
- Expired share rejected
- View-only can't edit
- Share token not guessable