
Entity Reader
- 1 installs
- 1 repo stars
- Updated July 10, 2026
- daoductam/advanced-backend-docs
Parse and extract structured data from backend documentation.
About
Entity-reader extracts and structures entity definitions from backend docs. Developers use it to automatically generate models and schemas from documentation.
- Documentation entity extraction
- Structured data parsing
Entity Reader by the numbers
- 1 all-time installs (skills.sh)
- Ranked #3,836 of 4,347 Backend & APIs skills by installs in the Skillselion catalog
- Data as of Jul 11, 2026 (Skillselion catalog sync)
npx skills add https://github.com/daoductam/advanced-backend-docs --skill entity-readerAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| repo stars | ★ 1 |
| Last updated | July 10, 2026 |
| Repository | daoductam/advanced-backend-docs ↗ |
What it does
Parse and extract structured data from backend documentation.
Files
Entity / Database Reader Skill
Primary focus: Spring Boot JPA / Hibernate. Also supports: SQLAlchemy, Django, TypeORM, Prisma, Sequelize, raw SQL DDL.
---
Step 0 — Brainstorming & Requirements Clarification (MANDATORY)
⛔ HARD GATE — Do not read any files, explore the codebase, write SQL, or connect to any database until this step is complete.
The purpose of this step is to avoid wasted effort. Even a request that seems fully self-contained often has hidden ambiguity (wrong mode assumed, missing filter conditions, wrong output format). Always confirm before acting.
Waiver Condition (read carefully before skipping)
You may skip Step 0 only if the user's message explicitly contains ALL of the following — not implied, not inferred, but literally stated:
| Required item | Explicitly provided means… |
|---|---|
| Mode | User said "generate SQL", "document schema", "export data", etc. |
| Table/entity scope | Named specific tables, entities, or said "all" |
| Output format | Said "SQL query", "Excel", "CSV", "Markdown", etc. |
| Filter/conditions (Mode B only) | WHERE conditions, status values, ID lists, or "no filter" |
| Output columns (Mode B only) | Listed columns explicitly or said "all fields" |
If any single item is missing or ambiguous → ask. Do not assume. A detailed-sounding request is not the same as an explicitly complete one.
---
Step 0A — Determine the Mode
If mode is not explicit, ask:
"To get started, which of these fits your goal?
- A — Schema docs: I'll read your entity/model files and document the table structure (output: Excel, Markdown, Word, etc.)
- B — SQL / data export: I'll write a SQL query or export actual data from a live database
- C — Both: Schema documentation + data export"
- Mode A: Schema Documentation → parse entity classes/DDL → produce docs
- Mode B: SQL / Data Export → write queries or connect to live DB and pull records
- Mode C: Both
---
Step 0B — Targeted Questions by Mode
Mode A — Schema Documentation:
1. Purpose: What's this for? (team docs, migration planning, audit, filling a template?) 2. Scope: All entities, or specific ones? List them if specific. 3. Output format: Excel / Markdown / Word / JSON / SQL DDL? 4. Template: Do you have an existing template to fill? (e.g. MO Vietnamese DB format)
→ ⛔ STOP. Wait for the user to answer before reading any files.
---
Mode B — SQL / Data Export:
First clarify the sub-mode:
- B1 — SQL query only (no live DB needed): "I'll explore the codebase schema and write you a SQL query to run yourself."
- B2 — Live DB export: "I'll connect to a live database and pull the actual data."
Then ask:
1. Scope: Which tables or entities? Be specific. 2. Filters: Status conditions, date ranges, specific IDs, or "all records"? 3. Output columns: Which columns to include, or "all"? 4. Output format: SQL query / Excel / CSV / JSON? 5. (B2 only) Connection: DB type + host + port + database + credentials
→ ⛔ STOP. Wait for the user to answer before writing any SQL or connecting to any database.
---
Step 0C — Confirm Before Proceeding
Once the user has answered, echo back a short plan and ask for green light:
"Got it. Here's my plan:
- Mode: [A / B1 / B2 / C]
- Scope: [tables/entities]
- Filters: [conditions or 'none']
- Output: [format]
>
Ready to proceed — shall I start?"
→ ⛔ STOP. Only move to Step 1 after the user says yes (or equivalent like "go", "proceed", "yes").
---
Step 1 — Ingest Source Files (Mode A: Schema Documentation)
If user selected Mode B (Data Export) in Step 0, skip to Step 1-B below.
If user selected Mode A or Mode C: Based on the scope defined in Step 0, locate and scan the relevant entity files.
If the user specified specific tables in Step 0: Only scan files matching those table/entity names. If the user wants all entities: Scan all files in the provided directory.
Check uploaded files or the specified directory. List what was found and confirm with the user before proceeding.
Detect framework by file content:
| Framework | Key signals |
|---|---|
| Spring JPA (primary) | .java + @Entity, @Table, @Column, @Id |
| Spring XML ORM | .hbm.xml / orm.xml with <class> or <entity> |
| Python SQLAlchemy | .py + Column(, declarative_base, mapped_column |
| Python Django | .py + models.Model, models.CharField |
| TypeORM | .ts + @Entity(), @Column() |
| Prisma | schema.prisma + model blocks |
| Sequelize | .js/.ts + DataTypes., sequelize.define |
| SQL DDL | .sql + CREATE TABLE |
For Spring JPA, read references/java-jpa.md for detailed parsing patterns. For other frameworks, read the relevant file in references/.
---
Step 1-B — Connect to Database (Mode B: Data Export)
If user selected Mode A (Schema Documentation), skip this step.
If user selected Mode B or Mode C: Connect to the live database using the credentials provided in Step 0.
1. Establish connection:
import psycopg2 # for PostgreSQL
# or: import mysql.connector # for MySQL
# or: import cx_Oracle # for Oracle
# etc.
conn = psycopg2.connect(
host="hostname",
port=5432,
database="dbname",
user="username",
password="password"
)2. Verify connection: Test with a simple query (e.g., SELECT 1 or SELECT version())
3. List available tables: Query system catalogs to show user what tables exist
- PostgreSQL:
SELECT tablename FROM pg_tables WHERE schemaname = 'public' - MySQL:
SHOW TABLES - Confirm with user if the tables from Step 0 exist
4. Build filtered queries: Based on the filtering criteria from Step 0, construct WHERE clauses
Example query construction:
# Base query
query = f"SELECT * FROM {table_name}"
# Add WHERE clauses from Step 0 filtering criteria
where_clauses = []
# Specific IDs (e.g., MIDs)
if specific_ids:
where_clauses.append(f"id IN ({','.join(map(str, specific_ids))})")
# Time range
if time_range:
where_clauses.append(f"created_at >= '{start_date}' AND created_at < '{end_date}'")
# Status conditions
if status_filter:
where_clauses.append(f"status = {status_value}")
# Image existence check
if must_have_image:
where_clauses.append("image_url IS NOT NULL")
# Custom WHERE clause from user
if custom_where:
where_clauses.append(custom_where)
# Combine all conditions
if where_clauses:
query += " WHERE " + " AND ".join(where_clauses)
# Add LIMIT if sample requested
if limit:
query += f" LIMIT {limit}"5. Preview query: Show the constructed query to the user for approval before executing
6. Execute and fetch data: Run the query and retrieve results
Security note: Always use parameterized queries to prevent SQL injection. Never concatenate user input directly into SQL.
---
Step 1.5 — Review and Confirm Entities (MANDATORY GATE)
⛔ HARD STOP — After scanning entities but before parsing, you MUST confirm with the user.
Why this step exists: Prevents accidentally documenting wrong tables, test entities, or deprecated tables. Requires user to explicitly approve what will be documented.
What to show the user:
List all entities found with basic info:
"I scanned the directory/files and found {N} entities:
>
| # | Entity Name | Table Name | File Path | Fields Count (approx) |
|---|-------------|------------|-----------|----------------------|
| 1 | User | users | src/entities/User.java | ~8 |
| 2 | Order | orders | src/entities/Order.java | ~12 |
| 3 | Product | products | src/entities/Product.java | ~10 |
| 4 | UserBackup | users_backup | src/entities/UserBackup.java | ~9 |
| 5 | TestEntity | test_data | src/test/TestEntity.java | ~3 |
| ... | ... | ... | ... | ... |
>
Before I proceed to document these tables, please confirm:
>
Question 1: Should I document ALL {N} entities, or do you want to exclude any?
- If excluding: Which ones should I skip? (provide numbers or names)
>
Question 2: For each table being added to documentation, why is it needed?
- This helps ensure we're not documenting test data, backups, or deprecated tables by mistake.
>
Please review the list and tell me:
- Which entities to include (e.g., "all", "1-3, 5", "exclude 4 and 5")
- Reason for documenting them (e.g., "production tables", "active development", "migration requirement")
User must provide:
1. Explicit confirmation of which entities to include 2. Justification/reason for why they're being documented (even brief like "production tables" is fine)
Handle user response:
# Example: User says "Include 1-3, exclude 4-5 because they are backup and test"
entities_to_document = [entity_list[0], entity_list[1], entity_list[2]]
excluded_entities = [entity_list[3], entity_list[4]]
reason = "Production tables for main application"
# Log the decision
print(f"✅ Confirmed: Documenting {len(entities_to_document)} entities")
print(f" Reason: {reason}")
print(f"❌ Excluded: {len(excluded_entities)} entities")
for e in excluded_entities:
print(f" - {e['name']} ({e['table_name']})")→ Only proceed to Step 2 after receiving explicit confirmation.
Exception: If the user specified exact table names in Step 0 (e.g., "document User and Order tables only"), you can skip asking again since they already specified.
---
Step 2 — Parse Entity Metadata
Parse only the entities approved in Step 1.5 (or Step 0 if already specified).
Step 2A — Extract Metadata
For each entity extract the following per field:
| Column | Source |
|---|---|
| Entity Name | Class name |
| Table Name | @Table(name=...) or snake_case of class name |
| Field Name | Java field name |
| Column Name | @Column(name=...) or snake_case of field name |
| Java Type | Declared type (Long, String, LocalDateTime, …) |
| SQL Type | Mapped DB type — see references/java-jpa.md type map |
| PK | @Id present |
| Nullable | @Column(nullable=false) → No; default → Yes |
| Unique | @Column(unique=true) or @UniqueConstraint |
| FK / Relation | @ManyToOne, @OneToMany, @JoinColumn(name=...) |
| Length | @Column(length=...) |
| Precision / Scale | @Column(precision=..., scale=...) |
| Default Value | @Column(columnDefinition=...) default clause |
| Validation | Bean Validation annotations (@NotNull, @Size, @Email, …) |
| Description | Javadoc /** ... */ on the field |
Spring-specific extras to capture:
@GeneratedValue(strategy=...)→ note in PK column@Temporal(TemporalType.DATE/TIME/TIMESTAMP)→ refine SQL Type@Enumerated(EnumType.STRING/ORDINAL)→ note in SQL Type@Lob→ SQL Type =TEXTorBLOB@CreationTimestamp/@UpdateTimestamp→ note in Description@Version→ note "optimistic lock" in Description- Lombok annotations (
@Data,@Builder, etc.) → note in entity header, parse fields normally
---
Step 2B — Detect Duplicate Table Structures
After parsing all entities, check for duplicate or nearly identical table structures. This prevents cluttering documentation with redundant tables and helps the user identify potential data model issues.
Use the detection script: Import and run scripts/detect_duplicates.py:
from scripts.detect_duplicates import detect_duplicate_tables, format_duplicate_report
# After parsing all entities into a list
duplicates = detect_duplicate_tables(entities, threshold=90.0)
if duplicates:
# Found duplicates - need to ask user
for dup in duplicates:
print(format_duplicate_report(dup))What qualifies as a duplicate:
- Identical column set: Same columns (by name) in both tables, even if field names differ
- High similarity (90%+ match): Tables share 90% or more of the same column names
- Same column types and constraints: Same columns with matching types, nullability, and key constraints
When duplicates are detected:
⛔ STOP and ask the user for EACH duplicate pair:
"I found potential duplicate table structures:
>
- `{table1_name}` ({N} columns) and `{table2_name}` ({M} columns) share {X}% of their columns
- Shared columns: {list first 5-10 shared column names}{if only-in-table1 exists:} - Only in {table1_name}: {list them}{if only-in-table2 exists:} - Only in {table2_name}: {list them}>
This might indicate:
1. One is a copy/backup of the other
2. They serve similar purposes and could be consolidated
3. They're intentionally similar but serve different contexts
>
How would you like me to handle this?
- A — Include both tables in the documentation
- B — Skip{table2_name}(keep{table1_name}only)
- C — Skip{table1_name}(keep{table2_name}only)
- D — Document both but add a note about the duplication
- E — Let me review the entity files first and then decide"
Apply user's choice using the script:
from scripts.detect_duplicates import filter_entities_by_user_choice
# Build user_choices dict based on their responses
# Example: {'dup_0': 'keep_second', 'dup_1': 'keep_both_annotated'}
user_choices = {}
for i, dup in enumerate(duplicates):
choice_key = f"dup_{i}"
# Map user response A/B/C/D to internal keys
# A -> 'keep_both'
# B -> 'keep_first'
# C -> 'keep_second'
# D -> 'keep_both_annotated'
user_choices[choice_key] = mapped_choice
# Filter entities based on choices
entities = filter_entities_by_user_choice(entities, duplicates, user_choices)If user chooses option D (keep both with annotation), the script adds: [Note: Similar structure to {other_table_name}] to the first field's description.
Example scenario:
If you detect that development_unit_backup has 95% similarity to development_unit (sharing columns like id, name, code, manager_id), you should: 1. Show the similarity report with shared and unique columns 2. Ask which table to include or whether to annotate both 3. Explain that including both without annotation might confuse readers 4. Apply the user's choice before generating output
Important: Don't auto-skip duplicates without asking. The user knows the domain model and may have valid reasons for both tables existing.
---
Step 3 — Confirm Output Mode (if not already decided in Step 0)
If the user already specified the output format in Step 0, skip this step and proceed directly to Step 4.
Otherwise, after parsing, confirm with the user:
"I found N entities with M fields total.
>
Based on your goal from our earlier discussion, would you like me to proceed with [format from Step 0], or would you prefer a different format?"
If they want to change the format, present the three modes:
Mode A — Generate a new output file
Create a fresh file from the extracted data. Ask which format:
- Excel (.xlsx) ← default recommendation
- CSV
- Word document (.docx)
- Markdown
- JSON
→ Go to Step 4A.
Mode B — Fill an existing template
The user has an Excel or Word template with placeholders or a pre-defined table structure that should be populated with the entity data.
The skill supports ANY Excel (.xlsx) or Word (.docx) template format — not tied to any specific organization's format.
→ Go to Step 4B.
Mode C — Generate SQL (DDL)
Produce CREATE TABLE SQL statements from the entity metadata.
Ask:
- Target dialect: MySQL / MariaDB, PostgreSQL, SQL Server, Oracle, SQLite, H2 (default: PostgreSQL)
- Include
DROP TABLE IF EXISTSbefore each table? (Yes / No) - Include FK
CONSTRAINT/REFERENCESclauses? (Yes / No) - Include index definitions from
@Indexannotations? (Yes / No)
→ Go to Step 4C.
---
Step 4A — Generate New Output File
Excel (.xlsx)
Read /mnt/skills/public/xlsx/SKILL.md before writing code.
Use scripts/entity_to_excel.py — import write_excel(entities, output_path).
Structure:
- "Summary" sheet — one row per entity: name, table, field count, PK field(s)
- Per-entity sheets if > 5 entities; single "All Fields" sheet if ≤ 5
Columns (in order):
Field Name | Column Name | Java Type | SQL Type | PK | Nullable | Unique | FK/Relation | Length | Precision | Scale | Default | Validation | DescriptionFormatting rules:
- Header: bold,
#4472C4background, white text, frozen row - PK rows: light green
#E2EFDA - FK rows: light yellow
#FFF2CC - Alternate row shading
#EBF3FB - Auto-fit column widths
Word (.docx)
Read /mnt/skills/public/docx/SKILL.md before writing code.
Structure: Title → Table of Contents → per-entity section (Heading 2 + field table + relation notes).
CSV / Markdown / JSON
- CSV:
pandas.to_csv(), one file (with Entity column) or per-entity files - Markdown:
## EntityName+| col | col |table - JSON:
{ "entities": [ { "name": "...", "tableName": "...", "fields": [...] } ] }
---
Step 4B — Fill Existing Template
The skill intelligently works with any Excel or Word template, auto-detecting structure and mapping entity fields automatically.
Step 4B-1: Inspect the Template
Ask the user to upload or specify the path to their template file (.xlsx or .docx).
Use the print_detected_structure() helper to analyze what the skill detected:
from scripts.fill_template import print_detected_structure
print_detected_structure(template_path)This outputs:
- For Excel: header row location, column mapping, data start row
- For Word: format type (table/sections), column mapping if applicable
Step 4B-2: Confirm or Adjust Mapping
Show the user what was detected and ask if it looks correct:
"I detected your template structure:
- Header row: Row 3
- Columns detected:
- Column A: Field Name
- Column B: Data Type
- Column C: Nullable
- Column D: Description
>
Does this look right? If any columns are wrong, let me know and I'll adjust the mapping."
If user says it's wrong or detection failed:
Ask them to describe the template structure:
- Which row contains the headers?
- What does each column represent?
- Should data go in one sheet or multiple sheets (one per entity)?
Build a manual user_column_mapping dict:
user_column_mapping = {
1: 'field_name', # Column A
2: 'sql_type', # Column B
3: 'nullable', # Column C
4: 'description', # Column D
# ... etc
}Step 4B-3: Fill the Template
Use scripts/fill_template.py to fill the template:
from scripts.fill_template import fill_template
output_path = fill_template(
entities=entities,
template_path='path/to/user_template.xlsx',
output_path='path/to/filled_template.xlsx',
user_column_mapping=None # Or pass the manual mapping if needed
)How it works:
For Excel templates: 1. Auto-detects header row by scanning first 10 rows for recognizable column names 2. Maps template columns to entity fields using fuzzy matching:
- "Field Name", "Column", "Name" →
field_name - "Type", "Data Type", "SQL Type" →
sql_type - "Null", "Nullable", "Allow Null" →
nullable - "PK", "Primary Key" →
pk - "FK", "Foreign Key", "Relation" →
relation - "Default", "Default Value" →
default - "Description", "Note", "Comment" →
description - (see
scripts/fill_template.pyfor full list)
3. For 3 or fewer entities: fills all in one sheet 4. For 4+ entities: creates one sheet per entity (or uses existing sheets) 5. Preserves all existing formatting, formulas, and sheets
For Word templates: 1. Detects if template uses a table format or sections format 2. If table: fills rows into the existing table 3. If sections: creates a heading + table for each entity 4. Preserves existing styles and formatting
Step 4B-4: Handle Special Cases
Multi-sheet Excel workbooks (like database documentation templates):
If the template has special sheets (like "Changelog", "Table of Contents", "Index"):
- The skill preserves all existing sheets
- Only modifies data sheets or creates new entity sheets
- Ask user if they want you to update index/changelog sheets
Custom column requirements:
If the template has columns the skill doesn't recognize (e.g., "Business Owner", "Approval Status"):
- The skill leaves those columns blank
- Ask user how to fill them: "Some columns like 'Business Owner' aren't in the entity data. Should I leave them blank, or would you like to provide values?"
Existing data in template:
If template already has data rows:
- The skill appends new rows after existing data
- Never overwrites existing data
- Ask: "Your template has existing data. Should I append new entities below, or create new sheets for them?"
Column Matching Logic
The skill uses fuzzy matching with these patterns (case-insensitive):
| Field Type | Matches headers containing... |
|---|---|
field_name | field, attribute, property, name, column name |
column_name | column, col name, db column, database column |
sql_type | type, data type, datatype, sql type, db type |
pk | pk, primary, key, primary key, is pk |
nullable | null, nullable, allow null, not null, nullability |
unique | unique, distinct, is unique, unique key |
relation | fk, foreign, relation, table link, reference, foreign key |
length | length, size, max length, maxlength |
default | default, data default, default value, initial value |
description | description, note, comment, remark, notes |
validation | validation, constraint, rule, check |
Match threshold: 60% similarity required.
Error Handling
If auto-detection fails completely:
raise ValueError(
"Could not auto-detect template structure. Please describe your template format: "
"Which row has headers? What do the columns represent?"
)Then build user_column_mapping manually based on user's description and retry.
---
Step 4C — Generate SQL (DDL)
Use references/sql-generation.md for dialect-specific syntax.
General structure per entity:
-- ==========================================
-- Table: users (Entity: User)
-- ==========================================
DROP TABLE IF EXISTS `users`;
CREATE TABLE `users` (
`id` BIGINT NOT NULL AUTO_INCREMENT,
`email` VARCHAR(255) NOT NULL,
`created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `uq_users_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='User account entity';Rules:
- Respect the chosen dialect's quoting style (backtick /
"/[]) - Apply
NOT NULLwhennullable = false - Apply
UNIQUEconstraints from@Column(unique=true)or@UniqueConstraint - Apply FK
REFERENCESclauses when requested (and target table is in the extracted set) - Append
-- [Javadoc description]inline comment for fields that have descriptions - Emit all tables first, then FK
ALTER TABLE ... ADD CONSTRAINT ...statements at the end
(avoids forward-reference issues)
- Write output to a
.sqlfile
---
Step 5 — Save and Present
1. Print a brief summary:
- Entities found / processed
- Total fields
- Any fields skipped or warnings (unrecognised annotations, missing types, etc.)
2. Save output to /mnt/user-data/outputs/<descriptive_filename>.<ext> 3. Call present_files with the output path.
---
Spring JPA Edge Cases
| Situation | Handling |
|---|---|
@Inheritance(strategy=JOINED/TABLE_PER_CLASS) | Note strategy; flatten parent fields into child, mark [inherited] |
@MappedSuperclass | Treat as abstract; merge its fields into all subclasses |
@Embedded / @Embeddable | Flatten into parent entity; prefix Description with [embedded: EmbeddedClass] |
@OneToMany / @ManyToMany (no column) | Include row; SQL Type = (relation — no column) |
@JoinTable | Note join table name + join columns in FK/Relation column |
Lombok @Data / @Builder | Parse normally; note "Lombok" in entity header row |
No @Column annotation | Infer: column name = snake_case(fieldName), nullable = true, no length |
@Enumerated(EnumType.STRING) | SQL Type = VARCHAR(enum) |
@Enumerated(EnumType.ORDINAL) | SQL Type = INT(enum) |
@Lob on String | SQL Type = TEXT / LONGTEXT |
@Lob on byte[] | SQL Type = BLOB / LONGBLOB |
| 50+ entities | Ask: one sheet per entity vs. combined sheet |
ORM Frameworks — Parsing Reference
TypeORM (TypeScript)
Annotations
| Decorator | Meaning |
|---|---|
@Entity('table_name') | Entity + table name |
@PrimaryGeneratedColumn() | Auto PK |
@PrimaryColumn() | Manual PK |
@Column({ type, nullable, default, length, unique }) | Field mapping |
@CreateDateColumn / @UpdateDateColumn | Audit timestamps |
@OneToMany(() => Target, t => t.field) | Relationship |
@ManyToOne(() => Target) | Relationship |
@ManyToMany(() => Target) | Relationship |
@JoinColumn({ name }) | FK column name |
@JoinTable | Join table for M2M |
@Index(['col1', 'col2'], { unique }) | Index |
TypeScript → SQL Mapping
| TS Type | SQL Type |
|---|---|
string | VARCHAR |
number | INT or FLOAT |
boolean | BOOLEAN |
Date | TIMESTAMP |
'uuid' column type | UUID |
'text' column type | TEXT |
'decimal' | DECIMAL |
'enum' | ENUM |
Example
@Entity('product')
@Index(['sku'], { unique: true })
export class Product {
@PrimaryGeneratedColumn()
id: number;
@Column({ length: 100, nullable: false })
name: string;
@Column({ type: 'decimal', precision: 10, scale: 2 })
price: number;
@ManyToOne(() => Category, cat => cat.products)
@JoinColumn({ name: 'category_id' })
category: Category;
}---
Django ORM (Python)
Field Types → SQL Mapping
| Django Field | SQL Type |
|---|---|
CharField(max_length=N) | VARCHAR(N) |
TextField() | TEXT |
IntegerField() | INTEGER |
BigIntegerField() / BigAutoField | BIGINT |
FloatField() | FLOAT |
DecimalField(max_digits, decimal_places) | DECIMAL |
BooleanField() | BOOLEAN |
DateField() | DATE |
DateTimeField() | TIMESTAMP |
EmailField() | VARCHAR(254) |
UUIDField() | UUID |
JSONField() | JSON / JSONB |
FileField() / ImageField() | VARCHAR (path) |
ForeignKey(Model, on_delete=...) | FK (BIGINT ref) |
ManyToManyField(Model) | Join table |
OneToOneField(Model) | FK + UNIQUE |
Key kwargs to extract
null=True→nullable = trueblank=True→ application-level (note in validation)default=...→ default valueunique=True→ unique constraintdb_column='...'→ column name overridedb_index=True→ creates indexchoices=[...]→ enum-like, list values in constraintsverbose_name='...'→ descriptionhelp_text='...'→ description
Meta class
class Meta:
db_table = 'my_table' # → table_name
unique_together = [['a','b']] # → composite unique
indexes = [models.Index(fields=['name'])]Example
class Order(models.Model):
customer = models.ForeignKey(Customer, on_delete=models.CASCADE, related_name='orders')
status = models.CharField(max_length=20, choices=[('PENDING','Pending'),('DONE','Done')])
total = models.DecimalField(max_digits=10, decimal_places=2)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
db_table = 'orders'---
SQL DDL (CREATE TABLE)
Parse raw SQL. Support:
CREATE TABLE [IF NOT EXISTS] schema.table_name ( ... );column_name data_type [NOT NULL] [DEFAULT val] [UNIQUE] [PRIMARY KEY]CONSTRAINT name PRIMARY KEY (cols)CONSTRAINT name UNIQUE (cols)CONSTRAINT name FOREIGN KEY (col) REFERENCES other_table(col) [ON DELETE ...]CREATE [UNIQUE] INDEX name ON table (cols);- Comments:
-- commentor/* comment */above column defs as description COMMENT ON COLUMN table.col IS '...'(PostgreSQL)COMMENT '...'(MySQL inline)
SQL Type pass-through: keep the raw SQL type as-is.
---
Prisma Schema
Keywords
| Keyword | Meaning |
|---|---|
model ModelName { } | Entity; table name = snake_case of ModelName |
@@map("table_name") | Explicit table name |
@id | Primary key |
@default(...) | Default value |
@unique | Unique constraint |
@map("col_name") | Column name override |
@relation(...) | Relationship |
@@unique([...]) | Composite unique |
@@index([...]) | Index |
Prisma → SQL Type Mapping
| Prisma | SQL |
|---|---|
String | VARCHAR / TEXT |
Int | INTEGER |
BigInt | BIGINT |
Float | FLOAT |
Decimal | DECIMAL |
Boolean | BOOLEAN |
DateTime | TIMESTAMP |
Json | JSON |
Bytes | BYTEA / BLOB |
@db.VarChar(N) | VARCHAR(N) |
field? (trailing ?) | nullable = true |
---
SQLAlchemy (Python)
Column definitions
Column('name', String(100), nullable=False, unique=True, default='x', comment='...')
Column(Integer, primary_key=True, autoincrement=True)
Column(ForeignKey('other_table.id'))Relationship
relationship('Target', back_populates='...', cascade='all, delete')Table name
__tablename__ = 'table_name'on the model class
---
ActiveRecord / Rails
Parse db/schema.rb:
create_table "users", force: :cascade do |t|
t.string "email", null: false
t.integer "age"
t.timestamps
t.index ["email"], name: "index_users_on_email", unique: true
endParse migration files as fallback if schema.rb not present. Ruby type → SQL: string→VARCHAR, integer→INTEGER, text→TEXT, boolean→BOOLEAN, datetime→TIMESTAMP, decimal→DECIMAL, float→FLOAT, binary→BLOB.
---
Hibernate XML Mappings
<class name="com.example.Customer" table="customer">
<id name="id" type="long"><generator class="native"/></id>
<property name="fullName" column="full_name" type="string" not-null="true" length="200"/>
<many-to-one name="category" class="Category" column="category_id" not-null="true"/>
<bag name="orders" inverse="true" cascade="all">
<key column="customer_id"/>
<one-to-many class="Order"/>
</bag>
</class>Extract: class name → entity, table attr → table_name, property elements → fields, many-to-one → ManyToOne relationship, bag/set/list with one-to-many → OneToMany.
Generic Template Support — Design & Usage
This document explains how the entity-reader skill's generic template filling works.
---
Philosophy
Any template, any format — the skill should work with whatever Excel or Word template the user brings, without requiring them to conform to a specific structure.
Traditional approach (before):
- Hard-coded to specific template format (MO Vietnamese DB template)
- Required exact column names and sheet structure
- Failed if template was different
New approach:
- Auto-detection: Scans template to understand structure
- Fuzzy matching: Maps columns intelligently using similarity scoring
- Flexible: Works with single-sheet, multi-sheet, table-based, or section-based formats
- Fallback: Asks user for clarification if auto-detection is uncertain
---
How Auto-Detection Works
Excel Templates
Step 1: Find the header row
Scans rows 1-10 looking for a row that:
- Has at least 3 non-empty text cells
- At least 2 cells match known column patterns (using fuzzy matching)
Step 2: Map columns to field types
For each column header, calculates similarity score against known patterns:
- "Field Name" vs "field_name" patterns → score = 0.85 → match!
- "Type" vs "sql_type" patterns → score = 0.65 → match!
- "Random Column" vs any pattern → score < 0.60 → no match
Threshold: 60% similarity required.
Step 3: Determine layout
- 1-3 entities → single-sheet layout (all entities in one sheet)
- 4+ entities → multi-sheet layout (one sheet per entity)
Word Templates
Step 1: Check for tables
Scans all tables in the document:
- Checks first row for recognizable headers
- If found → uses table-based format
Step 2: Fallback to sections
If no recognizable table found:
- Uses sections format
- Creates heading + table for each entity
---
Column Matching Patterns
The skill recognizes these common column header variations:
| What user writes | Skill recognizes as |
|---|---|
| Field Name, Column Name, Attribute, Property | field_name |
| Type, Data Type, DataType, SQL Type, DB Type | sql_type |
| Null, Nullable, Allow Null, Not Null, Is Null | nullable |
| PK, Primary Key, Is Primary, Key (if contains "primary") | pk |
| FK, Foreign Key, Relation, Reference, Table Link | relation |
| Length, Size, Max Length, MaxLength | length |
| Default, Default Value, Initial Value, Data Default | default |
| Description, Note, Comment, Remark, Notes, Desc | description |
| Validation, Constraint, Rule, Check | validation |
| Unique, Distinct, Is Unique, Unique Key | unique |
Case-insensitive, whitespace-tolerant, fuzzy matching.
Examples that work:
- "field_name", "Field Name", "FIELD NAME", "fieldName", "Field-Name"
- "data_type", "Data Type", "DataType", "TYPE", "Sql Type"
- "Not Null", "not null", "Nullable", "NULLABLE", "Allow Null"
---
Formatting Preservation
The skill preserves:
- ✅ All existing sheets in workbook
- ✅ Cell formatting (fonts, colors, borders)
- ✅ Column widths
- ✅ Formulas
- ✅ Existing data rows (appends below them)
- ✅ Protected sheets (if not write-protected)
- ✅ Conditional formatting rules
- ✅ Data validation rules
The skill does NOT:
- ❌ Overwrite existing data
- ❌ Delete sheets
- ❌ Modify cells outside the data area
- ❌ Change workbook-level settings
---
Example: User Brings Custom Template
User's template: company_db_docs.xlsx
Row 1: [blank]
Row 2: Database Documentation
Row 3: [blank]
Row 4: Column Name | SQL Data Type | Required? | Primary? | Comments
Row 5: [data would go here]Skill's detection:
Header Row: 4
Column Mapping:
- Column A (1): "Column Name" → field_name (similarity: 0.72)
- Column B (2): "SQL Data Type" → sql_type (similarity: 0.81)
- Column C (3): "Required?" → nullable (similarity: 0.63)
- Column D (4): "Primary?" → pk (similarity: 0.71)
- Column E (5): "Comments" → description (similarity: 0.68)
Data Start Row: 5Skill fills:
Row 5: id | BIGINT | Yes | Yes | Primary key
Row 6: username | VARCHAR(100) | Yes | No | User login name
Row 7: email | VARCHAR(255) | Yes | No | Email address
...Result: Template filled correctly without any manual configuration!
---
When Auto-Detection Fails
Scenario 1: Ambiguous headers
Template has: "Info", "Data", "Value" (too generic)
Solution: Skill asks user:
"I found columns 'Info', 'Data', 'Value' but couldn't determine what they represent.
Which column should contain:
- Field names?
- Data types?
- Descriptions?"
Scenario 2: Non-standard layout
Template uses vertical format (fields in column A, values in column B)
Solution: Skill detects this is not a table format and asks:
"Your template uses a vertical layout which I can't auto-fill.
Would you like me to:
A) Create a new sheet with a standard table format
B) Generate a new file in your template's style (you'll need to describe the format)"
Scenario 3: Multiple tables per sheet
Template has separate tables for different entity groups on same sheet
Solution: Skill uses the first recognizable table and notifies user:
"I found multiple tables in your template. I'll fill the first one (starting at row 4).
If you need data in other tables, please extract them to separate sheets first."
---
Advanced: Custom Column Mapping
If user has unique columns the skill doesn't recognize, they can provide a manual mapping:
user_column_mapping = {
1: 'field_name', # Column A: "Database Column"
2: 'sql_type', # Column B: "Type"
3: 'nullable', # Column C: "Mandatory?"
4: 'description', # Column D: "Business Description"
5: 'custom_business_owner', # Column E: Not in standard fields
6: 'custom_approval_status' # Column F: Not in standard fields
}For custom columns not in entity data, skill leaves them blank and asks how to fill them.
---
Multi-Sheet Workbooks
Common pattern: DB documentation workbooks
- Sheet: Changelog (tracks changes)
- Sheet: Index (table of contents)
- Sheet: users (entity data)
- Sheet: orders (entity data)
- Sheet: products (entity data)Skill behavior:
1. Detects which sheets have data tables vs. metadata sheets 2. Fills or creates entity data sheets 3. Preserves Changelog, Index, and other special sheets 4. Offers to update Index/Changelog if user wants:
"I've filled the entity sheets. Would you like me to also:
- Add entries to the Index sheet?
- Add a change record to the Changelog?"
---
Supported Template Patterns
Pattern 1: Simple Single-Sheet Table
| Field | Type | Nullable | Description |
|-------|------|----------|-------------|
| ... | ... | ... | ... |Use case: Small projects, quick documentation
Pattern 2: Multi-Sheet (One Per Entity)
Sheet: users
Sheet: orders
Sheet: productsUse case: Large projects with many entities
Pattern 3: Grouped Sections
## Users Entity
[table]
## Orders Entity
[table]Use case: Word documents, narrative-style docs
Pattern 4: Complex Workbook
Sheet: Index
Sheet: Changelog
Sheet: entity_name_1
Sheet: entity_name_2
...Use case: Enterprise DB documentation, compliance requirements
---
Migration Guide (MO Template → Generic)
If you previously used the MO Vietnamese template format:
Old way:
from scripts.fill_mo_template import fill_mo_template
fill_mo_template(entities, 'MO-Mô_tả_database.xlsx', 'output.xlsx')New way:
from scripts.fill_template import fill_template
fill_template(entities, 'ANY_template.xlsx', 'output.xlsx')Benefits:
- ✅ Works with any template, not just MO format
- ✅ No need to remember MO-specific column positions
- ✅ Auto-detects structure
- ✅ Same API, more flexible
Compatibility:
- MO templates still work (detected as multi-sheet format)
- No need to change your templates
- Skill detects Vietnamese column names automatically
---
Best Practices
For users: 1. Use clear, descriptive column headers ("Field Name" not "Col1") 2. Put headers in first 10 rows of sheet 3. Keep table structure consistent within a sheet 4. For Word: use actual tables, not space-aligned text
For skill developers: 1. Always show detected structure to user before filling 2. Provide escape hatch for manual mapping 3. Preserve ALL existing template content 4. Handle missing columns gracefully (leave blank, don't error)
---
Future Enhancements
Potential improvements:
- Support CSV templates (delimiter auto-detection)
- Support Markdown tables (GitHub-style)
- Learn from user corrections (improve matching algorithm)
- Detect and update related sheets (Index, Changelog) automatically
- Support template validation (check for required columns)
- Template library (common formats users can choose from)
Java JPA / Hibernate Entity Parsing
Scanning for Entity Files
import pathlib, re
def find_entity_files(root: str) -> list[pathlib.Path]:
return [p for p in pathlib.Path(root).rglob("*.java")
if re.search(r'@Entity\b', p.read_text(encoding="utf-8", errors="ignore"))]Extracting Class / Table Names
CLASS_PATTERN = re.compile(r'(?:public|protected)?\s*class\s+(\w+)')
TABLE_PATTERN = re.compile(r'@Table\s*\(\s*(?:[^)]*\s)?name\s*=\s*"([^"]+)"')
SCHEMA_PATTERN = re.compile(r'@Table\s*\(\s*(?:[^)]*\s)?schema\s*=\s*"([^"]+)"')
def extract_class_info(src: str) -> dict:
cls = CLASS_PATTERN.search(src)
table = TABLE_PATTERN.search(src)
return {
"entity_name": cls.group(1) if cls else "Unknown",
"table_name": table.group(1) if table else camel_to_snake(cls.group(1)) if cls else "unknown"
}
def camel_to_snake(name: str) -> str:
s = re.sub(r'(?<=[a-z0-9])([A-Z])', r'_\1', name)
return s.lower()Extracting Fields
Parse field blocks. A field block looks like:
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "user_id", nullable = false, length = 36)
private Long id;FIELD_BLOCK_PATTERN = re.compile(
r'((?:@\w+(?:\([^)]*\))?\s*)+)' # annotations (group 1)
r'(?:private|protected|public)\s+'
r'([\w<>,\s]+?)\s+' # type (group 2)
r'(\w+)\s*;', # field name (group 3)
re.MULTILINE
)
COLUMN_ATTR = re.compile(r'(\w+)\s*=\s*(?:"([^"]*)"|(\w+))')
def parse_field(annotations: str, java_type: str, field_name: str) -> dict:
anns = annotations.strip()
col_match = re.search(r'@Column\(([^)]*)\)', anns)
col_attrs = {}
if col_match:
for m in COLUMN_ATTR.finditer(col_match.group(1)):
col_attrs[m.group(1)] = m.group(2) or m.group(3)
return {
"field_name": field_name,
"column_name": col_attrs.get("name", camel_to_snake(field_name)),
"code_type": java_type.strip(),
"sql_type": java_type_to_sql(java_type.strip()),
"pk": bool(re.search(r'@Id\b', anns)),
"nullable": col_attrs.get("nullable", "true").lower() != "false",
"unique": col_attrs.get("unique", "false").lower() == "true",
"length": col_attrs.get("length", ""),
"precision": col_attrs.get("precision", ""),
"scale": col_attrs.get("scale", ""),
"default": col_attrs.get("columnDefinition", ""),
"validation": extract_validations(anns),
"relation": extract_relation(anns),
"description": "", # filled from Javadoc separately
}Java → SQL Type Mapping
JAVA_SQL_MAP = {
"Long": "BIGINT", "long": "BIGINT",
"Integer": "INT", "int": "INT",
"Short": "SMALLINT", "short": "SMALLINT",
"Boolean": "BOOLEAN", "boolean": "BOOLEAN",
"Double": "DOUBLE", "double": "DOUBLE",
"Float": "FLOAT", "float": "FLOAT",
"BigDecimal": "DECIMAL",
"String": "VARCHAR",
"char": "CHAR",
"Date": "DATE",
"LocalDate": "DATE",
"LocalDateTime": "TIMESTAMP",
"ZonedDateTime": "TIMESTAMP WITH TIME ZONE",
"Instant": "TIMESTAMP",
"byte[]": "BLOB",
"UUID": "VARCHAR(36)",
}
def java_type_to_sql(java_type: str) -> str:
base = java_type.split("<")[0].split(".")[-1]
return JAVA_SQL_MAP.get(base, java_type)Validation Annotations
VALIDATION_ANNOTATIONS = [
"@NotNull", "@NotBlank", "@NotEmpty",
"@Size", "@Min", "@Max",
"@Pattern", "@Email", "@Positive",
"@PositiveOrZero", "@Negative", "@NegativeOrZero",
"@Past", "@PastOrPresent", "@Future", "@FutureOrPresent",
"@DecimalMin", "@DecimalMax", "@Digits",
]
def extract_validations(annotations: str) -> str:
found = []
for ann in VALIDATION_ANNOTATIONS:
m = re.search(re.escape(ann) + r'(?:\([^)]*\))?', annotations)
if m:
found.append(m.group(0))
return ", ".join(found)Relationship Annotations
REL_PATTERN = re.compile(
r'(@(?:OneToOne|OneToMany|ManyToOne|ManyToMany)(?:\([^)]*\))?)'
r'.*?(?:@JoinColumn\((?:[^)]*name\s*=\s*"([^"]+)")?[^)]*\))?'
r'.*?(?:private|protected|public)\s+(?:List<|Set<|Optional<)?(\w+)',
re.DOTALL
)
def extract_relation(annotations: str) -> str:
m = REL_PATTERN.search(annotations)
if m:
return f"{m.group(1)} → {m.group(3) or ''}"
return ""Javadoc Comments
JAVADOC_PATTERN = re.compile(r'/\*\*(.*?)\*/', re.DOTALL)
def extract_javadoc_before(src: str, field_pos: int) -> str:
"""Find the last Javadoc comment before field_pos."""
docs = [(m.start(), m.group(1)) for m in JAVADOC_PATTERN.finditer(src) if m.end() <= field_pos]
if docs:
text = docs[-1][1]
# Strip leading * from each line
return " ".join(line.strip().lstrip("*").strip() for line in text.splitlines()).strip()
return ""Hibernate XML Mapping (.hbm.xml)
import xml.etree.ElementTree as ET
def parse_hbm_xml(path: str) -> list[dict]:
tree = ET.parse(path)
root = tree.getroot()
ns = {"hbm": "urn:nhibernate-mapping-2.2"} # adjust if needed
entities = []
for cls_el in root.iter("class"):
entity = {
"entity_name": cls_el.get("name", "").split(".")[-1],
"table_name": cls_el.get("table", ""),
"fields": []
}
# ID
id_el = cls_el.find("id")
if id_el is not None:
entity["fields"].append({
"field_name": id_el.get("name"), "pk": True,
"column_name": id_el.get("column", id_el.get("name")),
"code_type": id_el.get("type", ""),
})
# Properties
for prop in cls_el.iter("property"):
entity["fields"].append({
"field_name": prop.get("name"),
"column_name": prop.get("column", prop.get("name")),
"code_type": prop.get("type", ""),
"nullable": prop.get("not-null", "false") == "false",
})
entities.append(entity)
return entitiesJava Spring JPA / Hibernate Annotations Reference
Key Annotations to Parse
Entity-level
| Annotation | Meaning | Extract |
|---|---|---|
@Entity | Marks class as JPA entity | entity name = class name |
@Table(name="...") | Override table name | table_name |
@Table(schema="...") | Schema prefix | note in description |
@Inheritance(strategy=...) | Inheritance mapping | note in description |
@DiscriminatorColumn | Discriminator for inheritance | note |
Field-level
| Annotation | Meaning | Extract |
|---|---|---|
@Id | Primary key | primary_key = true |
@GeneratedValue | Auto-generated PK | note strategy in constraints |
@Column(name="...", nullable=..., length=..., unique=..., precision=..., scale=...) | Column mapping | all attributes |
@Column(columnDefinition="...") | Raw SQL type | sql_type |
@Basic(optional=...) | Nullable override | nullable |
@Lob | Large object | data_type = CLOB/BLOB |
@Enumerated | Enum type | data_type = ENUM, list values from Java enum |
@Temporal(TemporalType.DATE/TIME/TIMESTAMP) | Date type | map to DATE/TIME/TIMESTAMP |
@Transient | Not persisted | skip this field |
@Version | Optimistic lock version | note in constraints |
@CreationTimestamp, @UpdateTimestamp | Audit fields | note in description |
Relationship annotations
| Annotation | type | Attributes to extract |
|---|---|---|
@OneToMany | OneToMany | mappedBy, cascade, fetch |
@ManyToOne | ManyToOne | optional (nullable), fetch |
@OneToOne | OneToOne | mappedBy, cascade, fetch |
@ManyToMany | ManyToMany | mappedBy, cascade |
@JoinColumn(name="...") | FK column name | column_name on relationship field |
@JoinTable | Join table for M2M | note join table name and columns |
Validation (Bean Validation / Hibernate Validator)
Capture these as validation field: @NotNull, @NotBlank, @NotEmpty, @Size(min=,max=), @Min, @Max, @Email, @Pattern, @Positive, @PositiveOrZero, @DecimalMin, @DecimalMax
Index annotations
@Table(indexes = {
@Index(name = "idx_email", columnList = "email", unique = true)
})Extract: index name, column list, unique flag.
---
Java Type → SQL Type Mapping
| Java Type | SQL Type |
|---|---|
String | VARCHAR(length) or TEXT |
Long / long | BIGINT |
Integer / int | INTEGER |
Double / double | DOUBLE |
BigDecimal | DECIMAL(precision, scale) |
Boolean / boolean | BOOLEAN |
LocalDate | DATE |
LocalDateTime / Date | TIMESTAMP |
LocalTime | TIME |
byte[] | BLOB |
UUID | UUID / VARCHAR(36) |
| Enum | VARCHAR or INTEGER depending on @Enumerated |
---
Parsing Example
@Entity
@Table(name = "customer",
indexes = @Index(name="idx_email", columnList="email", unique=true))
public class Customer {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
@Column(name = "full_name", nullable = false, length = 200)
@NotBlank
private String fullName;
@Column(unique = true, nullable = false, length = 320)
@Email
private String email;
@Column(name = "created_at")
@CreationTimestamp
private LocalDateTime createdAt;
@OneToMany(mappedBy = "customer", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
private List<Order> orders;
}Extract:
- Entity:
Customer→ table:customer - Fields:
id(PK, BIGINT, auto),fullName(VARCHAR(200), NOT NULL),email(VARCHAR(320), NOT NULL, UNIQUE),createdAt(TIMESTAMP, auto) - Relationship:
orders→ OneToMany →Order, cascade=ALL, fetch=LAZY - Index:
idx_emailonemailUNIQUE
MO Database Description Template — Format Reference
This is the user's standard template format (MO-Mô_tả_database.xlsx). When the user says "fill my template" or "add to my DB description file", use this spec exactly.
---
Workbook Structure
| Sheet | Purpose |
|---|---|
Changelog | History of changes — columns: Thời gian, Người chỉnh sửa, Loại thay đổi, Nội dung |
Mục lục | Index / table of contents — one row per table |
<table_name> | One sheet per database table, named exactly after the table |
---
Per-Table Sheet Layout
Row 0: [col0="Table name"] [col1=empty] [col2="public.<table_name>"] [rest=empty]
Row 1: (empty)
Row 2: (empty) ← some sheets skip rows 1-2, use row 2 directly as header
Row 3 (HEADER):
col0="No" col1="Name" col2="Type" col3="Null (Not/null)" col4="Key (PK/FK)"
col5=empty col6="Data default" col7="Description" col8="Note"
Row 4 (sub-header for Key col):
col4="Key" col5="Table link" (rest empty)
Row 5+: DATA ROWS — one row per fieldColumn Mapping (0-indexed)
| Col | Header | Maps to |
|---|---|---|
| 0 | No | Row number (sequential integer, or blank for sub-fields like JSON keys) |
| 1 | Name | Column/field name |
| 2 | Type | SQL data type (e.g. int8, varchar(255), timestamp, bool, text) |
| 3 | Null (Not/null) | NOT or Not null if NOT NULL; blank if nullable |
| 4 | Key (PK/FK) | PK if primary key; FK if foreign key; blank otherwise |
| 5 | (Table link) | Referenced table name for FK (e.g. public.merchant); blank if not FK |
| 6 | Data default | Default value (e.g. False, now(), nextval('...'), 9999-12-31 00:00:00.000) |
| 7 | Description | Vietnamese description of the field |
| 8 | Note | Additional notes (UNIQUE, INDEX, deprecation warnings, enum values, etc.) |
Examples from real sheets
No | Name | Type | Null | Key | Table link | Default | Description | Note
1 | id | int8 | NOT | PK | | | |
2 | merchant_id | int8 | NOT | FK | public.merchant | | Id của merchant |
3 | status | int8 | NOT | | | | Trạng thái: 0-inactive, 1-active |
4 | created_at | timestamp | | | | | Ngày tạo |
5 | is_deleted | bool | | | | False | Trạng thái xoá | true: Xóa mềm\nfalse: ActiveNull value convention
- NOT NULL → write
NOTin col3 (some sheets useNot null— match the existing style in the sheet) - Nullable → leave col3 blank (NaN)
Key conventions
- PK: col4=
PK, col5=blank - FK: col4=
FK, col5=referenced table (e.g.public.merchant(id)or justpublic.merchant) - Unique index only (no FK/PK): col4=blank, col8=
UNIQUEorUNIQUE INDEX USING BTREE - No constraint: col4=blank, col5=blank
---
"Mục lục" Sheet Columns
col0: sequential number
col1: TABLE_NAME
col2: TABLE_ID
col3: DESCRIPTION (Vietnamese table description)
col4: NOTE (e.g. "Không sử dụng", ticket reference)
col5: TABLE_PATH
col6: SOURCE
col7: SOURCE_SERVER_NAME
col8: SOURCE_SERVER_SCHEMA
... (remaining columns mostly blank for new tables)When adding a new table, append a row to Mục lục with at minimum: number, TABLE_NAME, DESCRIPTION.
---
Changelog Sheet Columns
col0: Thời gian (datetime)
col1: Người chỉnh sửa (editor name/code)
col2: Loại thay đổi (A=Add, M=Modify)
col3: Nội dung (change description in Vietnamese)When adding new tables or fields, append a row to Changelog.
---
Common SQL Types Used in This Template
| PostgreSQL type | Used as |
|---|---|
int4 / int | Integer 32-bit |
int8 | Integer 64-bit (BIGINT) |
serial4 | Auto-increment int (PK) |
bigserial | Auto-increment bigint (PK) |
varchar(n) | Variable string with length |
varchar | Variable string, no limit |
text | Unlimited text |
bool / boolean | Boolean |
timestamp | Timestamp without timezone |
timestamptz / timestamp with time zone | Timestamp with timezone |
json / jsonb | JSON |
numeric(p,s) | Decimal |
_int4 | Integer array |
---
Formatting Rules (match existing sheets exactly)
- No special cell background colors in data rows (plain white)
- Row 0 (table name row): plain, no formatting required
- Header row (row 3): plain — no bold, no color in original
- Column widths: vary per sheet, auto-fit is acceptable
- No borders required on data rows (original has no borders)
- Vietnamese text in Description/Note columns — preserve as-is
---
Notes on Special Rows
Some sheets include sub-field rows for JSON columns (e.g. person_info text field followed by indented sub-rows for name, birthDay, identificationType etc.). These have NaN in the No column (blank row number) and no type/key/null. Include them after the parent field row.
Some sheets have the table name in row 0 col0 as Table name, others omit it. Always write Table name in col0, row 0 to match the majority format.
Python ORM Parsing — SQLAlchemy & Django
---
SQLAlchemy (Core / ORM)
Declarative Base Pattern
class User(Base):
__tablename__ = "users"
__table_args__ = {"schema": "public"}
id = Column(Integer, primary_key=True)
email = Column(String(255), nullable=False, unique=True)
created_at = Column(DateTime, default=datetime.utcnow)
role_id = Column(Integer, ForeignKey("roles.id"), nullable=True)Parsing Strategy (regex-based, no import needed)
import re, pathlib
TABLE_NAME_RE = re.compile(r'__tablename__\s*=\s*["\']([^"\']+)["\']')
CLASS_RE = re.compile(r'^class\s+(\w+)\s*\(', re.MULTILINE)
COLUMN_RE = re.compile(
r'(\w+)\s*=\s*(?:mapped_column|Column)\s*\(([^)]*(?:\([^)]*\)[^)]*)*)\)',
re.MULTILINE
)
FK_RE = re.compile(r'ForeignKey\s*\(\s*["\']([^"\']+)["\']')
SQLALCHEMY_TYPE_MAP = {
"Integer": "INT", "BigInteger": "BIGINT", "SmallInteger": "SMALLINT",
"Float": "FLOAT", "Numeric": "DECIMAL", "Double": "DOUBLE",
"String": "VARCHAR", "Text": "TEXT", "Unicode": "NVARCHAR",
"Boolean": "BOOLEAN", "Date": "DATE", "DateTime": "TIMESTAMP",
"Time": "TIME", "Interval": "INTERVAL", "LargeBinary": "BLOB",
"JSON": "JSON", "UUID": "UUID", "Enum": "ENUM",
}
def parse_sqlalchemy_file(src: str) -> list[dict]:
entities = []
classes = list(CLASS_RE.finditer(src))
for i, cls_match in enumerate(classes):
cls_name = cls_match.group(1)
# Get the class body (up to next class or EOF)
body_start = cls_match.end()
body_end = classes[i + 1].start() if i + 1 < len(classes) else len(src)
body = src[body_start:body_end]
table_name_m = TABLE_NAME_RE.search(body)
if not table_name_m:
continue # Not a model class
fields = []
for col_m in COLUMN_RE.finditer(body):
field_name = col_m.group(1)
args_str = col_m.group(2)
# Extract type
type_m = re.search(r'\b([A-Z]\w+)\s*(?:\((\d+)(?:,\s*(\d+))?\))?', args_str)
code_type = type_m.group(1) if type_m else "Unknown"
length = type_m.group(2) if type_m and type_m.group(2) else ""
scale = type_m.group(3) if type_m and type_m.group(3) else ""
sql_type = SQLALCHEMY_TYPE_MAP.get(code_type, code_type)
if length:
sql_type = f"{sql_type}({length}" + (f",{scale})" if scale else ")")
fk_m = FK_RE.search(args_str)
fields.append({
"field_name": field_name,
"column_name": re.search(r'name\s*=\s*["\']([^"\']+)["\']', args_str) and
re.search(r'name\s*=\s*["\']([^"\']+)["\']', args_str).group(1) or field_name,
"code_type": code_type,
"sql_type": sql_type,
"pk": "primary_key=True" in args_str,
"nullable": "nullable=False" not in args_str,
"unique": "unique=True" in args_str,
"length": length,
"scale": scale,
"default": re.search(r'default\s*=\s*([^,)]+)', args_str).group(1).strip()
if re.search(r'default\s*=\s*([^,)]+)', args_str) else "",
"relation": f"FK → {fk_m.group(1)}" if fk_m else "",
"validation": "",
"description": "",
})
entities.append({
"entity_name": cls_name,
"table_name": table_name_m.group(1),
"fields": fields,
})
return entitiesSQLAlchemy 2.x mapped_column / Mapped[...]
Same regex works. Additionally scan:
MAPPED_RE = re.compile(r'(\w+)\s*:\s*Mapped\[([^\]]+)\]\s*=\s*mapped_column\(([^)]*)\)')- Type comes from the
Mapped[T]annotation — stripOptional[...]wrapper to get base type.
---
Django ORM
Model Pattern
class Product(models.Model):
class Meta:
db_table = "products"
name = models.CharField(max_length=200)
price = models.DecimalField(max_digits=10, decimal_places=2)
is_active = models.BooleanField(default=True)
category = models.ForeignKey("Category", on_delete=models.CASCADE, null=True)
created_at = models.DateTimeField(auto_now_add=True)Parsing
DB_TABLE_RE = re.compile(r'db_table\s*=\s*["\']([^"\']+)["\']')
FIELD_RE = re.compile(
r'(\w+)\s*=\s*models\.(\w+)\s*\(([^)]*(?:\([^)]*\)[^)]*)*)\)',
re.MULTILINE
)
DJANGO_TYPE_MAP = {
"AutoField": "INT AUTO_INCREMENT", "BigAutoField": "BIGINT AUTO_INCREMENT",
"IntegerField": "INT", "BigIntegerField": "BIGINT",
"SmallIntegerField": "SMALLINT", "PositiveIntegerField": "INT UNSIGNED",
"FloatField": "FLOAT", "DecimalField": "DECIMAL",
"CharField": "VARCHAR", "TextField": "TEXT",
"BooleanField": "BOOLEAN", "NullBooleanField": "BOOLEAN",
"DateField": "DATE", "DateTimeField": "TIMESTAMP", "TimeField": "TIME",
"DurationField": "INTERVAL",
"BinaryField": "BLOB", "ImageField": "VARCHAR", "FileField": "VARCHAR",
"JSONField": "JSON", "UUIDField": "UUID",
"EmailField": "VARCHAR(254)", "URLField": "VARCHAR(200)",
"SlugField": "VARCHAR(50)",
"ForeignKey": "INT (FK)", "OneToOneField": "INT (FK)",
"ManyToManyField": "(relation — join table)",
}
def parse_django_file(src: str) -> list[dict]:
entities = []
classes = list(CLASS_RE.finditer(src)) # reuse CLASS_RE from above
for i, cls_m in enumerate(classes):
cls_name = cls_m.group(1)
body_start = cls_m.end()
body_end = classes[i + 1].start() if i + 1 < len(classes) else len(src)
body = src[body_start:body_end]
if "models.Model" not in src[cls_m.start():body_end]:
continue
table_m = DB_TABLE_RE.search(body)
table_name = table_m.group(1) if table_m else f"{cls_name.lower()}s"
fields = []
for f in FIELD_RE.finditer(body):
fname, ftype, fargs = f.group(1), f.group(2), f.group(3)
if fname.startswith("_") or ftype == "Meta":
continue
mx = re.search(r'max_length\s*=\s*(\d+)', fargs)
dp = re.search(r'decimal_places\s*=\s*(\d+)', fargs)
md = re.search(r'max_digits\s*=\s*(\d+)', fargs)
to = re.search(r'^["\']?(\w+)["\']?', fargs)
fk_target = to.group(1) if ftype in ("ForeignKey", "OneToOneField", "ManyToManyField") and to else ""
sql = DJANGO_TYPE_MAP.get(ftype, ftype)
if mx:
sql = f"VARCHAR({mx.group(1)})"
elif md and dp:
sql = f"DECIMAL({md.group(1)},{dp.group(1)})"
fields.append({
"field_name": fname,
"column_name": fname if not fk_target else fname + "_id",
"code_type": ftype,
"sql_type": sql,
"pk": ftype in ("AutoField", "BigAutoField"),
"nullable": "null=True" in fargs,
"unique": "unique=True" in fargs,
"length": mx.group(1) if mx else "",
"default": re.search(r'default\s*=\s*([^,)]+)', fargs).group(1).strip()
if re.search(r'default\s*=\s*([^,)]+)', fargs) else "",
"relation": f"FK → {fk_target}" if fk_target else "",
"validation": "",
"description": "",
})
entities.append({"entity_name": cls_name, "table_name": table_name, "fields": fields})
return entitiesSQL DDL Parsing
Approach: Use sqlparse (preferred) or regex fallback
With sqlparse
import sqlparse
from sqlparse.sql import Statement, Parenthesis, Identifier
from sqlparse.tokens import Keyword, DDL
def parse_ddl(sql_text: str) -> list[dict]:
entities = []
statements = sqlparse.parse(sql_text)
for stmt in statements:
if stmt.get_type() != "CREATE":
continue
flat = [t for t in stmt.flatten()]
tokens = [t for t in stmt.tokens if not t.is_whitespace]
# Find table name
table_name = None
for i, tok in enumerate(tokens):
if tok.ttype is DDL and tok.normalized.upper() == "CREATE":
# Look for TABLE keyword then name
for j in range(i+1, len(tokens)):
if tokens[j].ttype is Keyword and tokens[j].normalized.upper() == "TABLE":
if j+1 < len(tokens):
table_name = str(tokens[j+1]).strip().strip('"').strip('`')
break
break
if not table_name:
continue
# Find column definitions in parentheses
for tok in stmt.tokens:
if isinstance(tok, Parenthesis):
fields = parse_column_defs(str(tok)[1:-1])
entities.append({"entity_name": table_name, "table_name": table_name, "fields": fields})
return entitiesRegex Fallback (no sqlparse)
import re
CREATE_TABLE_RE = re.compile(
r'CREATE\s+(?:TEMPORARY\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?'
r'(?:`?[\w.]+`?\.)?`?(\w+)`?\s*\(([^;]+?)\)\s*'
r'(?:ENGINE|DEFAULT|COMMENT|;|$)',
re.IGNORECASE | re.DOTALL
)
def parse_ddl_regex(sql_text: str) -> list[dict]:
entities = []
for m in CREATE_TABLE_RE.finditer(sql_text):
table_name = m.group(1)
body = m.group(2)
fields = parse_column_defs(body)
entities.append({"entity_name": table_name, "table_name": table_name, "fields": fields})
return entitiesColumn Definition Parser
COL_DEF_RE = re.compile(
r'^\s*`?(\w+)`?\s+' # column name
r'(\w+)' # data type keyword
r'(?:\(([^)]+)\))?' # optional (length) or (precision,scale)
r'(.*?)$', # rest of definition
re.IGNORECASE | re.MULTILINE
)
SQL_KEYWORDS = {"PRIMARY", "UNIQUE", "INDEX", "KEY", "CONSTRAINT", "FOREIGN",
"CHECK", "FULLTEXT", "SPATIAL"}
def parse_column_defs(body: str) -> list[dict]:
# Split on commas not inside parens
lines = split_columns(body)
fields = []
pks = set()
for line in lines:
line = line.strip()
if not line:
continue
upper = line.upper().lstrip()
# Table constraints
if any(upper.startswith(k) for k in SQL_KEYWORDS):
if "PRIMARY KEY" in upper:
pk_cols = re.findall(r'`?(\w+)`?', line.split("(", 1)[-1].rstrip(")"))
pks.update(pk_cols)
continue
m = COL_DEF_RE.match(line)
if not m:
continue
col_name = m.group(1)
sql_type = m.group(2).upper()
size_str = m.group(3) or ""
rest = m.group(4) or ""
precision, scale, length = "", "", ""
if "," in size_str:
parts = size_str.split(",")
precision, scale = parts[0].strip(), parts[1].strip()
elif size_str:
length = size_str.strip()
sql_type = f"{sql_type}({length})"
# Comment
comment_m = re.search(r"COMMENT\s+'([^']*)'", rest, re.IGNORECASE)
default_m = re.search(r"DEFAULT\s+((?:'[^']*'|\S+))", rest, re.IGNORECASE)
fk_target = "" # filled in second pass from FOREIGN KEY constraints
fields.append({
"field_name": col_name,
"column_name": col_name,
"code_type": sql_type,
"sql_type": sql_type + (f"({precision},{scale})" if precision else ""),
"pk": "PRIMARY KEY" in rest.upper(),
"nullable": "NOT NULL" not in rest.upper(),
"unique": "UNIQUE" in rest.upper(),
"length": length,
"precision": precision,
"scale": scale,
"default": default_m.group(1).strip("'") if default_m else "",
"relation": fk_target,
"auto_increment": "AUTO_INCREMENT" in rest.upper() or "AUTOINCREMENT" in rest.upper(),
"validation": "",
"description": comment_m.group(1) if comment_m else "",
})
# Apply table-level PRIMARY KEY
for f in fields:
if f["field_name"] in pks:
f["pk"] = True
return fields
def split_columns(body: str) -> list[str]:
"""Split column definitions on commas, ignoring commas inside parentheses."""
parts, depth, current = [], 0, []
for ch in body:
if ch == "(":
depth += 1
elif ch == ")":
depth -= 1
if ch == "," and depth == 0:
parts.append("".join(current).strip())
current = []
else:
current.append(ch)
if current:
parts.append("".join(current).strip())
return partsMulti-dialect Notes
| Dialect | Auto-increment syntax | Quote char |
|---|---|---|
| MySQL / MariaDB | AUTO_INCREMENT | backtick ` `` |
| PostgreSQL | SERIAL / BIGSERIAL or GENERATED ALWAYS AS IDENTITY | "double quotes" |
| SQLite | AUTOINCREMENT | none / " |
| SQL Server | IDENTITY(1,1) | [square brackets] |
| Oracle | GENERATED BY DEFAULT AS IDENTITY | "double quotes" |
Strip [, ], ` ` and "` from identifiers before storing.
FOREIGN KEY Resolution
FK_CONSTRAINT_RE = re.compile(
r'FOREIGN\s+KEY\s*\(`?(\w+)`?\)\s*REFERENCES\s+`?(\w+)`?\s*\(`?(\w+)`?\)',
re.IGNORECASE
)
def resolve_fks(fields: list[dict], body: str) -> list[dict]:
fk_map = {}
for m in FK_CONSTRAINT_RE.finditer(body):
fk_map[m.group(1)] = f"FK → {m.group(2)}.{m.group(3)}"
for f in fields:
if f["field_name"] in fk_map:
f["relation"] = fk_map[f["field_name"]]
return fieldsSQL DDL Generation — Dialect Reference
Dialect Quick Reference
| Feature | MySQL/MariaDB | PostgreSQL | SQL Server | Oracle | SQLite | H2 |
|---|---|---|---|---|---|---|
| Quote char | ` `` | " | [ ] | " | " | " |
| Auto-increment | AUTO_INCREMENT | SERIAL / GENERATED ALWAYS AS IDENTITY | IDENTITY(1,1) | GENERATED ALWAYS AS IDENTITY | AUTOINCREMENT | AUTO_INCREMENT / IDENTITY |
| String type | VARCHAR(n) | VARCHAR(n) | NVARCHAR(n) | VARCHAR2(n) | TEXT | VARCHAR(n) |
| Boolean | TINYINT(1) | BOOLEAN | BIT | NUMBER(1) | INTEGER | BOOLEAN |
| Timestamp | TIMESTAMP | TIMESTAMP | DATETIME2 | TIMESTAMP | TEXT | TIMESTAMP |
| Table suffix | ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 | (none) | (none) | (none) | (none) | (none) |
---
Java → SQL Type Map by Dialect
| Java Type | MySQL | PostgreSQL | SQL Server | Oracle |
|---|---|---|---|---|
Long / long | BIGINT | BIGINT | BIGINT | NUMBER(19) |
Integer / int | INT | INTEGER | INT | NUMBER(10) |
Short / short | SMALLINT | SMALLINT | SMALLINT | NUMBER(5) |
Boolean / boolean | TINYINT(1) | BOOLEAN | BIT | NUMBER(1) |
Double / double | DOUBLE | DOUBLE PRECISION | FLOAT | BINARY_DOUBLE |
Float / float | FLOAT | REAL | REAL | BINARY_FLOAT |
BigDecimal | DECIMAL(p,s) | NUMERIC(p,s) | DECIMAL(p,s) | NUMBER(p,s) |
String | VARCHAR(n) | VARCHAR(n) | NVARCHAR(n) | VARCHAR2(n) |
LocalDate | DATE | DATE | DATE | DATE |
LocalDateTime | DATETIME | TIMESTAMP | DATETIME2 | TIMESTAMP |
ZonedDateTime | TIMESTAMP | TIMESTAMP WITH TIME ZONE | DATETIMEOFFSET | TIMESTAMP WITH TIME ZONE |
Instant | TIMESTAMP | TIMESTAMP | DATETIME2 | TIMESTAMP |
UUID | VARCHAR(36) | UUID | UNIQUEIDENTIFIER | VARCHAR2(36) |
byte[] + @Lob | LONGBLOB | BYTEA | VARBINARY(MAX) | BLOB |
String + @Lob | LONGTEXT | TEXT | NVARCHAR(MAX) | CLOB |
---
MySQL / MariaDB Template
-- ==========================================
-- Table: {table_name} (Entity: {entity_name})
-- ==========================================
DROP TABLE IF EXISTS `{table_name}`;
CREATE TABLE `{table_name}` (
{column_defs},
PRIMARY KEY (`{pk_col}`){unique_constraints}{index_defs}
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='{entity_comment}';
{fk_alters}Column definition format:
`{col_name}` {sql_type} [NOT NULL] [DEFAULT {val}] [AUTO_INCREMENT] [COMMENT '{description}']Unique constraint (inline or separate UNIQUE KEY):
UNIQUE KEY `uq_{table}_{col}` (`{col}`)FK alter (emitted after all CREATE TABLEs):
ALTER TABLE `{child_table}`
ADD CONSTRAINT `fk_{child}_{parent}`
FOREIGN KEY (`{fk_col}`) REFERENCES `{parent_table}` (`{parent_pk}`)
ON DELETE RESTRICT ON UPDATE CASCADE;---
PostgreSQL Template
-- ==========================================
-- Table: {table_name} (Entity: {entity_name})
-- ==========================================
DROP TABLE IF EXISTS "{table_name}" CASCADE;
CREATE TABLE "{table_name}" (
{column_defs},
CONSTRAINT "pk_{table_name}" PRIMARY KEY ("{pk_col}"){unique_constraints}
);
COMMENT ON TABLE "{table_name}" IS '{entity_comment}';
{column_comments}
{fk_alters}Column definition:
"{col_name}" {sql_type} [NOT NULL] [DEFAULT {val}]Column comment:
COMMENT ON COLUMN "{table}"."{col}" IS '{description}';Unique:
CONSTRAINT "uq_{table}_{col}" UNIQUE ("{col}")FK:
ALTER TABLE "{child_table}"
ADD CONSTRAINT "fk_{child}_{parent}"
FOREIGN KEY ("{fk_col}") REFERENCES "{parent_table}" ("{parent_pk}");---
SQL Server Template
-- ==========================================
-- Table: {table_name} (Entity: {entity_name})
-- ==========================================
IF OBJECT_ID(N'[dbo].[{table_name}]', N'U') IS NOT NULL
DROP TABLE [dbo].[{table_name}];
GO
CREATE TABLE [dbo].[{table_name}] (
{column_defs},
CONSTRAINT [PK_{table_name}] PRIMARY KEY CLUSTERED ([{pk_col}] ASC){unique_constraints}
);
GO
{fk_alters}Column definition:
[{col_name}] {sql_type} [NOT NULL | NULL] [DEFAULT ({val})] [IDENTITY(1,1)]---
Python Code: DDL Generator
import re
def camel_to_snake(name: str) -> str:
s = re.sub(r'(?<=[a-z0-9])([A-Z])', r'_\1', name)
return s.lower()
DIALECT_TYPES = {
"mysql": {
"Long": "BIGINT", "Integer": "INT", "Short": "SMALLINT",
"Boolean": "TINYINT(1)", "Double": "DOUBLE", "Float": "FLOAT",
"BigDecimal": "DECIMAL({p},{s})", "String": "VARCHAR({n})",
"LocalDate": "DATE", "LocalDateTime": "DATETIME",
"ZonedDateTime": "TIMESTAMP", "Instant": "TIMESTAMP",
"UUID": "VARCHAR(36)", "default": "VARCHAR(255)",
},
"postgresql": {
"Long": "BIGINT", "Integer": "INTEGER", "Short": "SMALLINT",
"Boolean": "BOOLEAN", "Double": "DOUBLE PRECISION", "Float": "REAL",
"BigDecimal": "NUMERIC({p},{s})", "String": "VARCHAR({n})",
"LocalDate": "DATE", "LocalDateTime": "TIMESTAMP",
"ZonedDateTime": "TIMESTAMP WITH TIME ZONE", "Instant": "TIMESTAMP",
"UUID": "UUID", "default": "TEXT",
},
"sqlserver": {
"Long": "BIGINT", "Integer": "INT", "Short": "SMALLINT",
"Boolean": "BIT", "Double": "FLOAT", "Float": "REAL",
"BigDecimal": "DECIMAL({p},{s})", "String": "NVARCHAR({n})",
"LocalDate": "DATE", "LocalDateTime": "DATETIME2",
"ZonedDateTime": "DATETIMEOFFSET", "Instant": "DATETIME2",
"UUID": "UNIQUEIDENTIFIER", "default": "NVARCHAR(255)",
},
"h2": { # Good for Spring test environments
"Long": "BIGINT", "Integer": "INT", "Short": "SMALLINT",
"Boolean": "BOOLEAN", "Double": "DOUBLE", "Float": "FLOAT",
"BigDecimal": "DECIMAL({p},{s})", "String": "VARCHAR({n})",
"LocalDate": "DATE", "LocalDateTime": "TIMESTAMP",
"ZonedDateTime": "TIMESTAMP WITH TIME ZONE", "Instant": "TIMESTAMP",
"UUID": "UUID", "default": "VARCHAR(255)",
},
}
def resolve_sql_type(java_type: str, length: str, precision: str, scale: str,
dialect: str = "mysql") -> str:
base = java_type.split("<")[0].split(".")[-1]
type_map = DIALECT_TYPES.get(dialect, DIALECT_TYPES["mysql"])
sql = type_map.get(base, type_map["default"])
# Clean up empty strings - use defaults instead of ''
n = length.strip() if length and length.strip() else "255"
p = precision.strip() if precision and precision.strip() else "10"
s = scale.strip() if scale and scale.strip() else "2"
# Only apply formatting if the type template actually needs it
if "{n}" in sql:
return sql.format(n=n)
elif "{p}" in sql and "{s}" in sql:
return sql.format(p=p, s=s)
else:
return sql
def get_quote(dialect: str) -> tuple[str, str]:
if dialect == "mysql": return "`", "`"
if dialect == "sqlserver": return "[", "]"
return '"', '"'
def generate_ddl(entities: list[dict], dialect: str = "mysql",
drop_table: bool = True, include_fk: bool = True,
include_index: bool = False) -> str:
q0, q1 = get_quote(dialect)
lines = []
fk_lines = []
for entity in entities:
tbl = entity["table_name"]
name = entity["entity_name"]
lines.append(f"-- {'=' * 42}")
lines.append(f"-- Table: {tbl} (Entity: {name})")
lines.append(f"-- {'=' * 42}")
if drop_table:
if dialect == "mysql":
lines.append(f"DROP TABLE IF EXISTS {q0}{tbl}{q1};")
elif dialect == "postgresql":
lines.append(f'DROP TABLE IF EXISTS "{tbl}" CASCADE;')
elif dialect == "sqlserver":
lines.append(f"IF OBJECT_ID(N'[dbo].[{tbl}]', N'U') IS NOT NULL")
lines.append(f" DROP TABLE [dbo].[{tbl}];")
lines.append("GO")
else:
lines.append(f'DROP TABLE IF EXISTS "{tbl}";')
col_defs = []
pk_cols = []
uq_cols = []
for f in entity.get("fields", []):
if "(relation" in f.get("sql_type", ""):
continue # skip virtual relation fields
col = f.get("column_name", "")
if not col or not col.strip(): # Skip fields with no column name
continue
jtype = f.get("code_type", "String")
sql_type = resolve_sql_type(
jtype,
f.get("length", ""),
f.get("precision", ""),
f.get("scale", ""),
dialect
)
parts = [f" {q0}{col}{q1} {sql_type}"]
if not f.get("nullable", True):
parts.append("NOT NULL")
if f.get("pk") and dialect in ("mysql", "h2"):
parts.append("AUTO_INCREMENT" if dialect == "mysql" else "AUTO_INCREMENT")
desc = f.get("description", "")
if desc and desc.strip() and dialect == "mysql": # Only add comment if description exists
safe = desc.replace("'", "''")[:100]
parts.append(f"COMMENT '{safe}'")
col_defs.append(" ".join(parts))
if f.get("pk"):
pk_cols.append(col)
if f.get("unique") and not f.get("pk"):
uq_cols.append(col)
# FK alter statements
if include_fk and f.get("relation"):
rel = f.get("relation", "")
if "FK →" in rel or "@ManyToOne" in rel or "@OneToOne" in rel:
# Extract reference table from relation string
ref = rel.replace("FK →", "").replace("@ManyToOne →", "").replace("@OneToOne →", "").strip()
if ref and ref.strip(): # Only create FK if reference exists
ref_table = camel_to_snake(ref.split(".")[0]) if "." in ref else camel_to_snake(ref)
ref_col = ref.split(".")[-1] if "." in ref else "id"
if ref_table and ref_table.strip(): # Verify ref_table is not empty
fk_lines.append(
f"ALTER TABLE {q0}{tbl}{q1}\n"
f" ADD CONSTRAINT {q0}fk_{tbl}_{col}{q1}\n"
f" FOREIGN KEY ({q0}{col}{q1}) REFERENCES {q0}{ref_table}{q1} ({q0}{ref_col}{q1});"
)
if pk_cols:
pk_str = ", ".join(f"{q0}{c}{q1}" for c in pk_cols)
col_defs.append(f" PRIMARY KEY ({pk_str})")
for uq_col in uq_cols:
col_defs.append(f" UNIQUE KEY {q0}uq_{tbl}_{uq_col}{q1} ({q0}{uq_col}{q1})")
suffix = " ENGINE=InnoDB DEFAULT CHARSET=utf8mb4" if dialect == "mysql" else ""
lines.append(f"CREATE TABLE {q0}{tbl}{q1} (")
lines.append(",\n".join(col_defs))
lines.append(f"){suffix};")
lines.append("")
# PostgreSQL column comments
if dialect == "postgresql":
for f in entity.get("fields", []):
desc = f.get("description", "")
if desc and desc.strip(): # Only add comment if description exists and is not empty
safe = desc.replace("'", "''")[:200]
col_name = f.get("column_name", "")
if col_name and col_name.strip(): # Verify column name exists
lines.append(f"COMMENT ON COLUMN \"{tbl}\".\"{col_name}\" IS '{safe}';")
if any(f.get("description", "").strip() for f in entity.get("fields", [])):
lines.append("") # Only add blank line if we added comments
if dialect == "sqlserver":
lines.append("GO")
lines.append("")
if fk_lines:
lines.append("-- ==========================================")
lines.append("-- Foreign Key Constraints")
lines.append("-- ==========================================")
lines.extend(fk_lines)
return "\n".join(lines)TypeScript ORM Parsing — TypeORM, Prisma, Sequelize
---
TypeORM
Entity Pattern
@Entity("orders")
export class Order {
@PrimaryGeneratedColumn("uuid")
id: string;
@Column({ type: "varchar", length: 100, nullable: false })
name: string;
@Column({ type: "decimal", precision: 10, scale: 2, default: 0 })
total: number;
@ManyToOne(() => Customer, customer => customer.orders)
@JoinColumn({ name: "customer_id" })
customer: Customer;
}Python Parsing (regex)
import re
ENTITY_RE = re.compile(r'@Entity\s*\(\s*(?:["\']([^"\']+)["\'])?\s*\)')
CLASS_RE = re.compile(r'export\s+class\s+(\w+)')
COLUMN_RE = re.compile(
r'(@(?:Primary(?:Generated)?Column|Column|CreateDateColumn|UpdateDateColumn|DeleteDateColumn)'
r'(?:\([^)]*(?:\([^)]*\)[^)]*)*\))?)\s*\n\s*(\w+)\s*:\s*([\w<>\[\]|]+)',
re.MULTILINE
)
REL_RE = re.compile(
r'(@(?:OneToOne|OneToMany|ManyToOne|ManyToMany)\s*\([^)]+\))\s*'
r'(?:@JoinColumn\s*\(\s*\{[^}]*name\s*:\s*["\']([^"\']+)["\'][^}]*\}\s*\))?\s*'
r'\n\s*(\w+)\s*:\s*([\w<>\[\]]+)',
re.MULTILINE
)
TYPEORM_TYPE_MAP = {
"string": "VARCHAR", "number": "INT", "boolean": "BOOLEAN",
"Date": "TIMESTAMP", "Buffer": "BLOB",
}
TS_EXPLICIT = {
"varchar": "VARCHAR", "text": "TEXT", "char": "CHAR",
"int": "INT", "integer": "INT", "bigint": "BIGINT", "smallint": "SMALLINT",
"float": "FLOAT", "double": "DOUBLE", "decimal": "DECIMAL", "numeric": "DECIMAL",
"boolean": "BOOLEAN", "bool": "BOOLEAN",
"date": "DATE", "datetime": "DATETIME", "timestamp": "TIMESTAMP",
"time": "TIME", "json": "JSON", "jsonb": "JSONB", "uuid": "UUID",
"blob": "BLOB", "bytea": "BYTEA", "enum": "ENUM",
}
def parse_typeorm_file(src: str) -> list[dict]:
entities = []
entity_m = ENTITY_RE.search(src)
class_m = CLASS_RE.search(src)
if not entity_m or not class_m:
return []
table_name = entity_m.group(1) or camel_to_snake(class_m.group(1))
fields = []
for col_m in COLUMN_RE.finditer(src):
ann, fname, ts_type = col_m.group(1), col_m.group(2), col_m.group(3)
# Extract explicit type from annotation options
type_opt = re.search(r'type\s*:\s*["\']([^"\']+)["\']', ann)
sql_type = TS_EXPLICIT.get(type_opt.group(1), type_opt.group(1).upper()) if type_opt \
else TYPEORM_TYPE_MAP.get(ts_type, ts_type)
length = re.search(r'length\s*:\s*(\d+)', ann)
prec = re.search(r'precision\s*:\s*(\d+)', ann)
scale = re.search(r'scale\s*:\s*(\d+)', ann)
fields.append({
"field_name": fname,
"column_name": re.search(r'name\s*:\s*["\']([^"\']+)["\']', ann) and
re.search(r'name\s*:\s*["\']([^"\']+)["\']', ann).group(1) or camel_to_snake(fname),
"code_type": ts_type,
"sql_type": sql_type,
"pk": "PrimaryGeneratedColumn" in ann or "PrimaryColumn" in ann,
"nullable": "nullable: true" in ann,
"unique": "unique: true" in ann,
"length": length.group(1) if length else "",
"precision": prec.group(1) if prec else "",
"scale": scale.group(1) if scale else "",
"default": re.search(r'default\s*:\s*([^,}]+)', ann).group(1).strip()
if re.search(r'default\s*:\s*([^,}]+)', ann) else "",
"relation": "",
"validation": "",
"description": "",
})
for rel_m in REL_RE.finditer(src):
fields.append({
"field_name": rel_m.group(3),
"column_name": rel_m.group(2) or camel_to_snake(rel_m.group(3)) + "_id",
"code_type": rel_m.group(4),
"sql_type": "(relation — no column)" if "OneToMany" in rel_m.group(1) or "ManyToMany" in rel_m.group(1) else "INT (FK)",
"pk": False, "nullable": True, "unique": False,
"length": "", "precision": "", "scale": "", "default": "",
"relation": f"{rel_m.group(1).split('(')[0].strip()} → {rel_m.group(4)}",
"validation": "", "description": "",
})
entities.append({"entity_name": class_m.group(1), "table_name": table_name, "fields": fields})
return entities---
Prisma
Schema Pattern
model User {
id Int @id @default(autoincrement())
email String @unique @db.VarChar(255)
name String?
createdAt DateTime @default(now()) @map("created_at")
posts Post[]
@@map("users")
}Python Parsing
MODEL_RE = re.compile(r'model\s+(\w+)\s*\{([^}]+)\}', re.DOTALL)
MAP_RE = re.compile(r'@@map\s*\(\s*"([^"]+)"\s*\)')
FIELD_RE = re.compile(
r'^\s*(\w+)\s+([\w\[\]]+)(\?)?(.*)$',
re.MULTILINE
)
PRISMA_TYPE_MAP = {
"Int": "INT", "BigInt": "BIGINT", "Float": "FLOAT", "Decimal": "DECIMAL",
"Boolean": "BOOLEAN", "String": "VARCHAR", "DateTime": "TIMESTAMP",
"Json": "JSON", "Bytes": "BLOB",
}
def parse_prisma(src: str) -> list[dict]:
entities = []
for model_m in MODEL_RE.finditer(src):
model_name = model_m.group(1)
body = model_m.group(2)
map_m = MAP_RE.search(body)
table_name = map_m.group(1) if map_m else camel_to_snake(model_name)
fields = []
for f in FIELD_RE.finditer(body):
fname = f.group(1)
ftype = f.group(2).rstrip("[]")
optional = bool(f.group(3)) # "?" suffix
attrs = f.group(4) or ""
if fname.startswith("@@") or fname.startswith("//"):
continue
is_list = "[]" in f.group(2)
db_attr = re.search(r'@db\.(\w+)(?:\((\d+)(?:,\s*(\d+))?\))?', attrs)
sql_type = (db_attr.group(1) + (f"({db_attr.group(2)}" + (f",{db_attr.group(3)})" if db_attr.group(3) else ")") if db_attr.group(2) else ""))
if db_attr else PRISMA_TYPE_MAP.get(ftype, ftype))
col_map = re.search(r'@map\s*\(\s*"([^"]+)"\s*\)', attrs)
default = re.search(r'@default\s*\(([^)]+)\)', attrs)
fields.append({
"field_name": fname,
"column_name": col_map.group(1) if col_map else camel_to_snake(fname),
"code_type": f.group(2),
"sql_type": "(relation — no column)" if is_list else sql_type,
"pk": "@id" in attrs,
"nullable": optional,
"unique": "@unique" in attrs,
"length": db_attr.group(2) if db_attr and db_attr.group(2) else "",
"precision": "",
"scale": db_attr.group(3) if db_attr and db_attr.group(3) else "",
"default": default.group(1) if default else "",
"relation": f"→ {ftype}" if is_list or (ftype[0].isupper() and ftype not in PRISMA_TYPE_MAP) else "",
"validation": "",
"description": "",
})
entities.append({"entity_name": model_name, "table_name": table_name, "fields": fields})
return entities---
Sequelize
Model Pattern (JS/TS)
const User = sequelize.define('User', {
id: { type: DataTypes.INTEGER, primaryKey: true, autoIncrement: true },
email: { type: DataTypes.STRING(255), allowNull: false, unique: true },
score: { type: DataTypes.DECIMAL(10, 2), defaultValue: 0 },
}, { tableName: 'users' });Python Parsing
SEQ_MODEL_RE = re.compile(
r"(?:sequelize\.define|new\s+Sequelize\.Model)\s*\(\s*['\"](\w+)['\"].*?tableName['\"]?\s*:\s*['\"]([^'\"]+)['\"]",
re.DOTALL
)
SEQ_FIELD_RE = re.compile(
r"(\w+)\s*:\s*\{([^}]+)\}",
re.MULTILINE
)
DATATYPE_RE = re.compile(r"DataTypes\.(\w+)(?:\(([^)]+)\))?")
SEQUELIZE_TYPE_MAP = {
"STRING": "VARCHAR", "TEXT": "TEXT", "CITEXT": "CITEXT",
"INTEGER": "INT", "BIGINT": "BIGINT", "FLOAT": "FLOAT",
"DOUBLE": "DOUBLE", "DECIMAL": "DECIMAL", "BOOLEAN": "BOOLEAN",
"DATE": "TIMESTAMP", "DATEONLY": "DATE", "TIME": "TIME",
"BLOB": "BLOB", "UUID": "UUID", "JSON": "JSON", "JSONB": "JSONB",
"ENUM": "ENUM", "ARRAY": "ARRAY",
}"""
Detect duplicate or highly similar table structures in parsed entity metadata.
Used by entity-reader skill to prevent redundant documentation.
"""
from typing import List, Dict, Tuple, Set
from dataclasses import dataclass
@dataclass
class TableSimilarity:
"""Represents similarity between two tables."""
table1: str
table2: str
similarity_percent: float
shared_columns: List[str]
table1_only: List[str]
table2_only: List[str]
column_count1: int
column_count2: int
def extract_column_names(entity: Dict) -> Set[str]:
"""Extract column names from an entity's field list."""
columns = set()
for field in entity.get('fields', []):
col_name = field.get('column_name', '').lower().strip()
if col_name:
columns.add(col_name)
return columns
def calculate_similarity(cols1: Set[str], cols2: Set[str]) -> float:
"""
Calculate similarity percentage between two column sets.
Uses the smaller table as the denominator to catch subset relationships.
"""
if not cols1 or not cols2:
return 0.0
shared = cols1.intersection(cols2)
smaller_table_size = min(len(cols1), len(cols2))
if smaller_table_size == 0:
return 0.0
return (len(shared) / smaller_table_size) * 100
def detect_duplicate_tables(entities: List[Dict], threshold: float = 90.0) -> List[TableSimilarity]:
"""
Detect duplicate or highly similar table structures.
Args:
entities: List of entity dictionaries with 'name', 'table_name', and 'fields'
threshold: Similarity percentage threshold (default 90%)
Returns:
List of TableSimilarity objects for pairs exceeding the threshold
"""
duplicates = []
# Compare each pair of tables
for i in range(len(entities)):
for j in range(i + 1, len(entities)):
entity1 = entities[i]
entity2 = entities[j]
table1_name = entity1.get('table_name', entity1.get('name', 'unknown'))
table2_name = entity2.get('table_name', entity2.get('name', 'unknown'))
cols1 = extract_column_names(entity1)
cols2 = extract_column_names(entity2)
if not cols1 or not cols2:
continue
similarity = calculate_similarity(cols1, cols2)
if similarity >= threshold:
shared = sorted(cols1.intersection(cols2))
only1 = sorted(cols1 - cols2)
only2 = sorted(cols2 - cols1)
duplicates.append(TableSimilarity(
table1=table1_name,
table2=table2_name,
similarity_percent=similarity,
shared_columns=shared,
table1_only=only1,
table2_only=only2,
column_count1=len(cols1),
column_count2=len(cols2)
))
return duplicates
def format_duplicate_report(similarity: TableSimilarity) -> str:
"""Format a human-readable duplicate detection report."""
report = []
report.append(f"\n{'='*70}")
report.append(f"POTENTIAL DUPLICATE DETECTED")
report.append(f"{'='*70}")
report.append(f"Table 1: {similarity.table1} ({similarity.column_count1} columns)")
report.append(f"Table 2: {similarity.table2} ({similarity.column_count2} columns)")
report.append(f"Similarity: {similarity.similarity_percent:.1f}%")
report.append(f"\nShared columns ({len(similarity.shared_columns)}):")
for col in similarity.shared_columns[:10]: # Show first 10
report.append(f" - {col}")
if len(similarity.shared_columns) > 10:
report.append(f" ... and {len(similarity.shared_columns) - 10} more")
if similarity.table1_only:
report.append(f"\nOnly in {similarity.table1}:")
for col in similarity.table1_only[:5]:
report.append(f" - {col}")
if len(similarity.table1_only) > 5:
report.append(f" ... and {len(similarity.table1_only) - 5} more")
if similarity.table2_only:
report.append(f"\nOnly in {similarity.table2}:")
for col in similarity.table2_only[:5]:
report.append(f" - {col}")
if len(similarity.table2_only) > 5:
report.append(f" ... and {len(similarity.table2_only) - 5} more")
report.append(f"{'='*70}\n")
return "\n".join(report)
def filter_entities_by_user_choice(
entities: List[Dict],
duplicates: List[TableSimilarity],
user_choices: Dict[str, str]
) -> List[Dict]:
"""
Filter entity list based on user's duplicate handling choices.
Args:
entities: Original entity list
duplicates: Detected duplicates
user_choices: Dict mapping duplicate pair IDs to choices ('keep_first', 'keep_second', 'keep_both', 'keep_both_annotated')
Returns:
Filtered entity list
"""
tables_to_skip = set()
tables_to_annotate = {} # table_name -> other_table_name
for i, dup in enumerate(duplicates):
choice_key = f"dup_{i}"
choice = user_choices.get(choice_key, 'keep_both')
if choice == 'keep_first':
tables_to_skip.add(dup.table2)
elif choice == 'keep_second':
tables_to_skip.add(dup.table1)
elif choice == 'keep_both_annotated':
tables_to_annotate[dup.table1] = dup.table2
tables_to_annotate[dup.table2] = dup.table1
# Filter entities
filtered = []
for entity in entities:
table_name = entity.get('table_name', entity.get('name', ''))
if table_name in tables_to_skip:
continue
# Add annotation if needed
if table_name in tables_to_annotate:
other_table = tables_to_annotate[table_name]
annotation = f"[Note: Similar structure to {other_table}]"
# Add annotation to first field's description or create a note field
if entity.get('fields'):
first_field = entity['fields'][0]
current_desc = first_field.get('description', '')
if current_desc:
first_field['description'] = f"{annotation} {current_desc}"
else:
first_field['description'] = annotation
# Also store at entity level for summary purposes
entity['duplicate_note'] = annotation
filtered.append(entity)
return filtered
if __name__ == '__main__':
# Example usage
test_entities = [
{
'name': 'User',
'table_name': 'users',
'fields': [
{'column_name': 'id'},
{'column_name': 'email'},
{'column_name': 'created_at'}
]
},
{
'name': 'UserBackup',
'table_name': 'users_backup',
'fields': [
{'column_name': 'id'},
{'column_name': 'email'},
{'column_name': 'created_at'},
{'column_name': 'backup_date'}
]
},
{
'name': 'Product',
'table_name': 'products',
'fields': [
{'column_name': 'id'},
{'column_name': 'name'},
{'column_name': 'price'}
]
}
]
duplicates = detect_duplicate_tables(test_entities, threshold=90.0)
for dup in duplicates:
print(format_duplicate_report(dup))
#!/usr/bin/env python3
"""
entity_to_excel.py
------------------
Generic helper: takes a list of entity dicts and writes a formatted Excel workbook.
Usage (called from Claude's generated code):
from scripts.entity_to_excel import write_excel
write_excel(entities, output_path="output.xlsx")
Entity format:
[
{
"entity_name": "User",
"table_name": "users",
"fields": [
{
"field_name": "id", "column_name": "id",
"code_type": "Long", "sql_type": "BIGINT",
"pk": True, "nullable": False, "unique": True,
"length": "", "precision": "", "scale": "",
"default": "", "relation": "", "validation": "", "description": ""
}, ...
]
}, ...
]
"""
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
HEADER_FILL = PatternFill("solid", start_color="4472C4")
ALT_FILL = PatternFill("solid", start_color="EBF3FB")
WHITE_FILL = PatternFill("solid", start_color="FFFFFF")
PK_FILL = PatternFill("solid", start_color="E2EFDA") # light green
FK_FILL = PatternFill("solid", start_color="FFF2CC") # light yellow
HEADER_FONT = Font(bold=True, color="FFFFFF", name="Arial", size=10)
NORMAL_FONT = Font(name="Arial", size=10)
THIN_BORDER = Border(
left=Side(style="thin"), right=Side(style="thin"),
top=Side(style="thin"), bottom=Side(style="thin")
)
COLUMNS = [
("Field Name", "field_name", 18),
("Column Name", "column_name", 18),
("Code Type", "code_type", 16),
("SQL Type", "sql_type", 16),
("PK", "pk", 6),
("Nullable", "nullable", 9),
("Unique", "unique", 8),
("FK / Relation","relation", 22),
("Length", "length", 8),
("Precision", "precision", 9),
("Scale", "scale", 7),
("Default", "default", 14),
("Validation", "validation", 22),
("Description", "description", 30),
]
def _write_entity_sheet(wb: Workbook, entity: dict) -> None:
name = entity["entity_name"][:31] # Excel sheet name limit
ws = wb.create_sheet(title=name)
# Sub-title row
ws.merge_cells(start_row=1, start_column=1, end_row=1, end_column=len(COLUMNS))
cell = ws.cell(row=1, column=1,
value=f"{entity['entity_name']} → table: {entity['table_name']}")
cell.font = Font(bold=True, name="Arial", size=11, color="1F3864")
cell.fill = PatternFill("solid", start_color="D9E1F2")
cell.alignment = Alignment(horizontal="center", vertical="center")
ws.row_dimensions[1].height = 20
# Header row
for col_idx, (header, _, width) in enumerate(COLUMNS, start=1):
c = ws.cell(row=2, column=col_idx, value=header)
c.font = HEADER_FONT
c.fill = HEADER_FILL
c.alignment = Alignment(horizontal="center", vertical="center", wrap_text=True)
c.border = THIN_BORDER
ws.column_dimensions[get_column_letter(col_idx)].width = width
ws.row_dimensions[2].height = 18
ws.freeze_panes = "A3"
for row_idx, field in enumerate(entity.get("fields", []), start=3):
alt = (row_idx % 2 == 1)
is_pk = bool(field.get("pk"))
is_fk = bool(field.get("relation"))
row_fill = PK_FILL if is_pk else (FK_FILL if is_fk else (ALT_FILL if alt else WHITE_FILL))
for col_idx, (_, key, _) in enumerate(COLUMNS, start=1):
val = field.get(key, "")
if isinstance(val, bool):
val = "✓" if val else ""
c = ws.cell(row=row_idx, column=col_idx, value=val)
c.font = NORMAL_FONT
c.fill = row_fill
c.alignment = Alignment(horizontal="left", vertical="center", wrap_text=True)
c.border = THIN_BORDER
def write_excel(entities: list, output_path: str = "entity_dictionary.xlsx",
combined_threshold: int = 5) -> str:
wb = Workbook()
wb.remove(wb.active) # remove default sheet
# Summary sheet
ws_sum = wb.create_sheet(title="Summary", index=0)
sum_headers = ["Entity Name", "Table Name", "Field Count", "PK Field(s)"]
for ci, h in enumerate(sum_headers, 1):
c = ws_sum.cell(row=1, column=ci, value=h)
c.font = HEADER_FONT; c.fill = HEADER_FILL
c.alignment = Alignment(horizontal="center"); c.border = THIN_BORDER
widths = [24, 24, 12, 30]
for ci, w in enumerate(widths, 1):
ws_sum.column_dimensions[get_column_letter(ci)].width = w
ws_sum.freeze_panes = "A2"
for ri, entity in enumerate(entities, start=2):
pks = [f["field_name"] for f in entity.get("fields", []) if f.get("pk")]
ws_sum.cell(row=ri, column=1, value=entity["entity_name"]).border = THIN_BORDER
ws_sum.cell(row=ri, column=2, value=entity["table_name"]).border = THIN_BORDER
ws_sum.cell(row=ri, column=3, value=len(entity.get("fields", []))).border = THIN_BORDER
ws_sum.cell(row=ri, column=4, value=", ".join(pks)).border = THIN_BORDER
fill = ALT_FILL if ri % 2 == 1 else WHITE_FILL
for ci in range(1, 5):
ws_sum.cell(row=ri, column=ci).fill = fill
ws_sum.cell(row=ri, column=ci).font = NORMAL_FONT
if len(entities) <= combined_threshold:
# All entities on one "All Fields" sheet
ws_all = wb.create_sheet(title="All Fields")
all_cols = [("Entity", "entity_name", 18), ("Table", "table_name", 18)] + COLUMNS
for ci, (h, _, w) in enumerate(all_cols, 1):
c = ws_all.cell(row=1, column=ci, value=h)
c.font = HEADER_FONT; c.fill = HEADER_FILL
c.alignment = Alignment(horizontal="center"); c.border = THIN_BORDER
ws_all.column_dimensions[get_column_letter(ci)].width = w
ws_all.freeze_panes = "A2"
row_idx = 2
for entity in entities:
for field in entity.get("fields", []):
alt = row_idx % 2 == 0
is_pk = bool(field.get("pk"))
is_fk = bool(field.get("relation"))
row_fill = PK_FILL if is_pk else (FK_FILL if is_fk else (ALT_FILL if alt else WHITE_FILL))
ws_all.cell(row=row_idx, column=1, value=entity["entity_name"]).fill = row_fill
ws_all.cell(row=row_idx, column=2, value=entity["table_name"]).fill = row_fill
for ci, (_, key, _) in enumerate(COLUMNS, start=3):
val = field.get(key, "")
if isinstance(val, bool):
val = "✓" if val else ""
c = ws_all.cell(row=row_idx, column=ci, value=val)
c.fill = row_fill; c.border = THIN_BORDER; c.font = NORMAL_FONT
c.alignment = Alignment(wrap_text=True)
row_idx += 1
else:
for entity in entities:
_write_entity_sheet(wb, entity)
wb.save(output_path)
return output_path
if __name__ == "__main__":
# Quick smoke test
sample = [
{
"entity_name": "User", "table_name": "users",
"fields": [
{"field_name": "id", "column_name": "id", "code_type": "Long",
"sql_type": "BIGINT", "pk": True, "nullable": False, "unique": True,
"length": "", "precision": "", "scale": "", "default": "",
"relation": "", "validation": "@NotNull", "description": "Primary key"},
{"field_name": "email", "column_name": "email", "code_type": "String",
"sql_type": "VARCHAR(255)", "pk": False, "nullable": False, "unique": True,
"length": "255", "precision": "", "scale": "", "default": "",
"relation": "", "validation": "@Email @NotBlank", "description": "User email address"},
]
}
]
path = write_excel(sample, "/tmp/test_entity.xlsx")
print(f"Written: {path}")
#!/usr/bin/env python3
"""
fill_template.py
----------------
Generic template filler for entity metadata - works with any Excel or Word template format.
This script intelligently detects template structure and maps entity fields to template
columns/sections automatically. Supports both Excel (.xlsx) and Word (.docx) templates.
Usage:
from scripts.fill_template import fill_template
result = fill_template(
entities=entities,
template_path='path/to/template.xlsx',
output_path='path/to/output.xlsx',
user_column_mapping=None # Optional: specify if auto-detection fails
)
"""
from openpyxl import load_workbook
from openpyxl.styles import Font, PatternFill, Alignment
from docx import Document
from docx.shared import Pt, RGBColor
from datetime import datetime
from typing import List, Dict, Optional, Tuple
import re
from difflib import SequenceMatcher
class TemplateMapper:
"""Intelligent mapper from entity fields to template columns."""
# Fuzzy matching keywords for common column types
COLUMN_PATTERNS = {
'field_name': ['field', 'attribute', 'property', 'name', 'column name', 'field name'],
'column_name': ['column', 'col name', 'db column', 'database column', 'db name'],
'sql_type': ['type', 'data type', 'datatype', 'sql type', 'db type'],
'pk': ['pk', 'primary', 'key', 'primary key', 'is pk', 'is primary'],
'nullable': ['null', 'nullable', 'allow null', 'not null', 'nullability', 'is null'],
'unique': ['unique', 'distinct', 'is unique', 'unique key'],
'relation': ['fk', 'foreign', 'relation', 'table link', 'reference', 'foreign key', 'ref table'],
'length': ['length', 'size', 'max length', 'maxlength'],
'default': ['default', 'data default', 'default value', 'initial value'],
'description': ['description', 'note', 'comment', 'remark', 'notes', 'desc'],
'validation': ['validation', 'constraint', 'rule', 'check'],
}
@staticmethod
def similarity(a: str, b: str) -> float:
"""Calculate similarity between two strings."""
return SequenceMatcher(None, a.lower().strip(), b.lower().strip()).ratio()
@classmethod
def fuzzy_match_column(cls, header: str, threshold: float = 0.6) -> Optional[str]:
"""
Match a template column header to a known field type using fuzzy matching.
Args:
header: Column header from template
threshold: Minimum similarity score (0.0 to 1.0)
Returns:
Matched field type or None
"""
if not header or not isinstance(header, str):
return None
best_match = None
best_score = threshold
for field_type, patterns in cls.COLUMN_PATTERNS.items():
for pattern in patterns:
score = cls.similarity(header, pattern)
if score > best_score:
best_score = score
best_match = field_type
return best_match
@classmethod
def detect_excel_structure(cls, ws) -> Dict:
"""
Auto-detect Excel template structure.
Returns dict with:
- header_row: Row index of headers
- column_mapping: Dict mapping column indices to field types
- data_start_row: Where data rows begin
- structure_type: 'single_sheet' or 'multi_sheet'
"""
structure = {
'header_row': None,
'column_mapping': {},
'data_start_row': None,
'structure_type': 'single_sheet'
}
# Scan first 10 rows to find headers
for row_idx in range(1, min(11, ws.max_row + 1)):
row_cells = [ws.cell(row=row_idx, column=col).value
for col in range(1, ws.max_column + 1)]
# A header row typically has multiple non-empty text values
non_empty = [c for c in row_cells if c and isinstance(c, str)]
if len(non_empty) >= 3: # At least 3 columns
# Try to match these to our known patterns
matches = sum(1 for cell in non_empty if cls.fuzzy_match_column(cell))
if matches >= 2: # At least 2 recognizable columns
structure['header_row'] = row_idx
structure['data_start_row'] = row_idx + 1
# Build column mapping
for col_idx, cell_value in enumerate(row_cells, start=1):
field_type = cls.fuzzy_match_column(cell_value)
if field_type:
structure['column_mapping'][col_idx] = field_type
break
return structure
@classmethod
def detect_word_structure(cls, doc: Document) -> Dict:
"""
Auto-detect Word template structure.
Returns dict with:
- format: 'table' or 'sections'
- table_index: Index of the table containing field data (if format='table')
- column_mapping: Dict mapping column indices to field types
"""
structure = {
'format': None,
'table_index': None,
'column_mapping': {}
}
# Check for tables
for idx, table in enumerate(doc.tables):
if table.rows:
# Check first row for headers
header_cells = [cell.text.strip() for cell in table.rows[0].cells]
matches = sum(1 for cell in header_cells if cls.fuzzy_match_column(cell))
if matches >= 2:
structure['format'] = 'table'
structure['table_index'] = idx
# Build column mapping
for col_idx, cell_text in enumerate(header_cells):
field_type = cls.fuzzy_match_column(cell_text)
if field_type:
structure['column_mapping'][col_idx] = field_type
break
# If no table found, assume sections format
if not structure['format']:
structure['format'] = 'sections'
return structure
def fill_excel_template(entities: List[Dict], template_path: str, output_path: str,
user_column_mapping: Optional[Dict] = None) -> str:
"""
Fill an Excel template with entity metadata.
Args:
entities: List of entity dicts with 'name', 'table_name', 'fields'
template_path: Path to template .xlsx file
output_path: Path to save filled template
user_column_mapping: Optional manual column mapping if auto-detection fails
Returns:
Path to saved file
"""
# Load template
wb = load_workbook(template_path, data_only=False)
# Get first sheet as the working sheet (or create one if empty workbook)
if wb.sheetnames:
ws = wb.active
else:
ws = wb.create_sheet("Entities")
# Detect structure
structure = TemplateMapper.detect_excel_structure(ws)
# Use user mapping if provided
if user_column_mapping:
structure['column_mapping'] = user_column_mapping
if not structure['header_row']:
raise ValueError(
"Could not auto-detect template structure. Please provide user_column_mapping. "
"Example: {'field_name': 1, 'sql_type': 2, 'nullable': 3}"
)
# Determine if we're doing single-sheet (all entities) or multi-sheet (one per entity)
if len(entities) > 3:
# Multi-sheet approach: one sheet per entity
for entity in entities:
sheet_name = entity.get('table_name', entity['name'])[:31] # Excel max sheet name length
# Create or get sheet
if sheet_name in wb.sheetnames:
entity_ws = wb[sheet_name]
# Find last row
last_row = entity_ws.max_row
start_row = last_row + 1
else:
entity_ws = wb.create_sheet(title=sheet_name)
# Copy header structure from template
if structure['header_row']:
for col_idx, field_type in structure['column_mapping'].items():
# Copy header
header_cell = ws.cell(row=structure['header_row'], column=col_idx)
entity_ws.cell(row=1, column=col_idx, value=header_cell.value)
start_row = 2
# Fill data
_fill_entity_rows(entity_ws, entity, structure['column_mapping'], start_row)
else:
# Single-sheet approach: all entities in one sheet
current_row = structure['data_start_row']
for entity in entities:
# Optionally add entity name row
if 'field_name' in structure['column_mapping'].values():
ws.cell(row=current_row, column=1, value=f"Entity: {entity['name']}")
current_row += 1
# Fill data
current_row = _fill_entity_rows(ws, entity, structure['column_mapping'], current_row)
current_row += 1 # Blank row between entities
# Save
wb.save(output_path)
return output_path
def _fill_entity_rows(ws, entity: Dict, column_mapping: Dict, start_row: int) -> int:
"""
Fill entity field rows into worksheet.
Returns the next available row index.
"""
row_idx = start_row
for field in entity.get('fields', []):
for col_idx, field_type in column_mapping.items():
value = _get_field_value(field, field_type)
ws.cell(row=row_idx, column=col_idx, value=value)
row_idx += 1
return row_idx
def _get_field_value(field: Dict, field_type: str) -> str:
"""Extract the appropriate value from a field dict based on field type."""
mapping = {
'field_name': field.get('field_name', ''),
'column_name': field.get('column_name', field.get('field_name', '')),
'sql_type': field.get('sql_type', ''),
'pk': 'PK' if field.get('pk') else '',
'nullable': '' if field.get('nullable', True) else 'NOT NULL',
'unique': 'UNIQUE' if field.get('unique') else '',
'relation': field.get('relation', ''),
'length': field.get('length', ''),
'default': field.get('default', ''),
'description': field.get('description', ''),
'validation': field.get('validation', ''),
}
return str(mapping.get(field_type, ''))
def fill_word_template(entities: List[Dict], template_path: str, output_path: str) -> str:
"""
Fill a Word template with entity metadata.
Args:
entities: List of entity dicts
template_path: Path to template .docx file
output_path: Path to save filled template
Returns:
Path to saved file
"""
doc = Document(template_path)
structure = TemplateMapper.detect_word_structure(doc)
if structure['format'] == 'table' and structure['table_index'] is not None:
# Fill existing table
table = doc.tables[structure['table_index']]
for entity in entities:
# Add entity section heading if not in table
if len(doc.paragraphs) > 0:
doc.add_heading(entity.get('table_name', entity['name']), level=2)
for field in entity.get('fields', []):
row_cells = table.add_row().cells
for col_idx, field_type in structure['column_mapping'].items():
value = _get_field_value(field, field_type)
row_cells[col_idx].text = value
else:
# Sections format: add entities as separate sections
for entity in entities:
# Add heading
doc.add_heading(f"{entity.get('table_name', entity['name'])}", level=2)
# Add a table for fields
num_fields = len(entity.get('fields', []))
if num_fields > 0:
# Create table with headers
table = doc.add_table(rows=1, cols=5)
table.style = 'Light Grid Accent 1'
headers = ['Field Name', 'Type', 'Nullable', 'Key', 'Description']
for idx, header in enumerate(headers):
table.rows[0].cells[idx].text = header
# Add field rows
for field in entity.get('fields', []):
row_cells = table.add_row().cells
row_cells[0].text = field.get('column_name', field.get('field_name', ''))
row_cells[1].text = field.get('sql_type', '')
row_cells[2].text = '' if field.get('nullable', True) else 'NOT NULL'
key_info = []
if field.get('pk'):
key_info.append('PK')
if field.get('relation'):
key_info.append('FK')
row_cells[3].text = ', '.join(key_info)
row_cells[4].text = field.get('description', '')
doc.add_paragraph() # Blank paragraph between entities
doc.save(output_path)
return output_path
def fill_template(entities: List[Dict], template_path: str, output_path: str,
user_column_mapping: Optional[Dict] = None) -> str:
"""
Main entry point: auto-detect template type and fill it.
Args:
entities: List of entity dicts
template_path: Path to template file (.xlsx or .docx)
output_path: Path to save filled template
user_column_mapping: Optional manual column mapping for Excel
Returns:
Path to saved file
Raises:
ValueError: If template type is not supported or structure cannot be detected
"""
if template_path.endswith('.xlsx'):
return fill_excel_template(entities, template_path, output_path, user_column_mapping)
elif template_path.endswith('.docx'):
return fill_word_template(entities, template_path, output_path)
else:
raise ValueError(f"Unsupported template format: {template_path}. Must be .xlsx or .docx")
def print_detected_structure(template_path: str) -> None:
"""
Helper function to print the detected structure of a template.
Useful for debugging or showing users what was detected.
"""
if template_path.endswith('.xlsx'):
wb = load_workbook(template_path, data_only=False)
ws = wb.active
structure = TemplateMapper.detect_excel_structure(ws)
print("=== Excel Template Structure ===")
print(f"Header Row: {structure['header_row']}")
print(f"Data Start Row: {structure['data_start_row']}")
print(f"Column Mapping:")
for col_idx, field_type in sorted(structure['column_mapping'].items()):
col_letter = ws.cell(row=1, column=col_idx).column_letter
header_text = ws.cell(row=structure['header_row'], column=col_idx).value
print(f" Column {col_letter} (#{col_idx}): '{header_text}' → {field_type}")
elif template_path.endswith('.docx'):
doc = Document(template_path)
structure = TemplateMapper.detect_word_structure(doc)
print("=== Word Template Structure ===")
print(f"Format: {structure['format']}")
if structure['format'] == 'table':
print(f"Table Index: {structure['table_index']}")
print(f"Column Mapping:")
for col_idx, field_type in sorted(structure['column_mapping'].items()):
print(f" Column {col_idx}: {field_type}")
if __name__ == "__main__":
# Example usage
sample_entities = [
{
"name": "User",
"table_name": "users",
"fields": [
{
"field_name": "id",
"column_name": "id",
"sql_type": "BIGINT",
"pk": True,
"nullable": False,
"unique": True,
"description": "Primary key"
},
{
"field_name": "email",
"column_name": "email",
"sql_type": "VARCHAR(255)",
"pk": False,
"nullable": False,
"unique": True,
"length": "255",
"description": "User email address"
}
]
}
]
print("Generic template filler - ready to use")
print("Supports: .xlsx and .docx templates")
print("Auto-detects column structure intelligently")