Article Extractor

MQL5 Article Extractor

Extract technical trading articles from mql5.com for training data collection. Scope limited to mql5.com domain only.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use This Skill

Use this skill when:

• Extracting articles from mql5.com for reference or training data
• Downloading MQL5 documentation and tutorials
• Collecting trading articles from specific MQL5 users
• Building a corpus of MQL5 programming examples

Scope Boundaries

VALID requests:

• "Extract this mql5.com article: <https://www.mql5.com/en/articles/19625>"
• "Get all articles from MQL5 user 29210372"
• "Download trading articles from mql5.com"
• "Extract 5 MQL5 articles for testing"

OUT OF SCOPE:

• "Extract from yahoo.com" - NOT SUPPORTED (mql5.com only)
• "Scrape news from reuters" - NOT SUPPORTED (mql5.com only)
• "Get stock data from Bloomberg" - NOT SUPPORTED (mql5.com only)

If user requests non-mql5.com extraction, respond: "This skill extracts articles from mql5.com ONLY. For other sites, use different tools."

Repository Location

Working directory: $HOME/eon/mql5-local (adjust path for your environment)

Always execute commands from this directory:

cd "$HOME/eon/mql5-local"

Valid Input Types

1. Article URL (Most Specific)

Format: https://www.mql5.com/en/articles/[ID] Example: https://www.mql5.com/en/articles/19625 Action: Extract single article

2. User ID (Numeric or Username)

Format: Numeric (e.g., 29210372) or username (e.g., jslopes) Source: From mql5.com profile URL Action: Auto-discover and extract all user's articles

3. URL List File

Format: Text file with one URL per line Action: Batch process multiple articles

4. Vague Request

If user says "extract mql5 articles" without specifics, prompt for:

1. Article URL OR User ID 1. Quantity limit (for testing) 1. Output location preference

---

Reference Documentation

For detailed information, see:

• Extraction Modes - Single, batch, auto-discovery, official docs modes
• Data Sources - User collections and official documentation
• Troubleshooting - Common issues and solutions
• Examples - Usage examples and patterns

---

Troubleshooting

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

Skill: MQL5 Article Extractor

Topic Collections

TICK Data Research (mql5_articles/tick_data/)

• Official Docs: 3 Python MetaTrader5 TICK functions
• copy_ticks_from, copy_ticks_range, symbol_info_tick
• User Articles: 9 articles on TICK data handling, storage, and analysis
• Research Documentation: /docs/tick_research/ (4 files)

Python Integration (mql5_articles/python_integration/)

• Official Docs: 32 complete Python MetaTrader5 API functions
• Connection, account info, symbol management, historical data, orders, positions
• User Articles: 15 implementation guides
• ML/AI integration, trading automation, data analysis, socket communication
• Research Documentation: /docs/python_research/ (1 file)

Input Validation Workflow

When user makes vague request:

1. Ask for domain confirmation (if ambiguous):

• "Is this for mql5.com articles? (This tool only works with mql5.com)"

1. Request specific input:

• "Please provide one of:
• Article URL: https://www.mql5.com/en/articles/{ID}
• User ID: numeric or username from mql5.com profile
• Path to file with article URLs"

1. Suggest scope limits:

• "For testing, use --max-articles N to limit quantity"
• "For production, omit limit to extract all"

1. Confirm before large operations:

• If user says "all articles" → ask "User X has Y articles. Proceed with full extraction?"

Common Usage Examples

Example 1: Vague → Specific (Guided)

User: "Extract some trading articles" Skill: "This extracts from mql5.com only. Please specify:

1. Article URL (https://www.mql5.com/en/articles/{ID}) 1. MQL5 user ID (e.g., 29210372) 1. Path to URL file

Which do you have?"

User: "User 29210372" Skill:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py discover-and-extract --user-id 29210372

Example 2: URL Provided (Direct)

User: "Extract <https://www.mql5.com/en/articles/19625>" Skill:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py single https://www.mql5.com/en/articles/19625

Example 3: Out of Scope (Reject)

User: "Extract articles from reuters.com" Skill: "This skill extracts from mql5.com ONLY. Reuters is not supported. Use general web scraping tools instead."

Example 4: Testing (Limited)

User: "Test with 5 articles from user jslopes" Skill:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py discover-and-extract --user-id jslopes --max-articles 5

Output Structure

All extractions go to:

mql5_articles/
├── 29210372/                 # User collections (numeric ID or username)
│   └── article_[ID]/
│       ├── article_[ID].md
│       ├── metadata.json
│       └── images/
├── tick_data/                # Topic collections
│   ├── official_docs/        # 3 Python MT5 TICK functions
│   │   ├── copy_ticks_from.md

Evolution Log

Convention: Reverse chronological order (newest on top, oldest at bottom). Prepend new entries.

---

2026-02-26: Initial Evolution Log

Status: Skill is in use and maintained. Track improvements here.

Purpose

This evolution log tracks updates to the skill. Each entry should note:

• What changed (content, structure, tooling)
• Why it changed (bug fix, feature request, best practice)
• Files affected

How to Use

1. When updating SKILL.md or references, add an entry here with the date 2. Keep entries reverse-chronological (newest first) 3. Link to ADRs or GitHub issues when relevant 4. Reference specific line changes when helpful

---

Skill: MQL5 Article Extractor

MQL5 Article Extractor - Examples

Example 1: Vague Request → Bounded Guidance

User Input: "I need to extract some trading articles"

Skill Response:

This skill extracts articles from mql5.com ONLY (not other trading sites).

Please specify one of:
1. MQL5 article URL: https://www.mql5.com/en/articles/[ID]
2. MQL5 user ID: numeric (29210372) or username (jslopes)
3. Path to text file containing article URLs

What would you like to extract?

---

Example 2: Direct URL (Valid)

User Input: "Extract <https://www.mql5.com/en/articles/19625>"

Skill Actions:

1. Recognizes valid mql5.com URL 1. Executes single extraction:

   cd $HOME/eon/mql5-local
   .venv/bin/python mql5_extract.py single https://www.mql5.com/en/articles/19625

Output:

✅ Extraction successful!
   Article ID: 19625
   Title: Automating Trading Strategies in MQL5 (Part 34)...
   Author: Allan Munene Mutiiria
   Word count: 2,435
   Code blocks: 10
   Images: 9
   Output: $HOME/eon/mql5-local/mql5_articles/29210372/article_19625/

---

Example 3: Out-of-Scope Domain (Rejected)

User Input: "Can you extract articles from Yahoo Finance?"

Skill Response:

❌ OUT OF SCOPE

This skill extracts from mql5.com ONLY. Yahoo Finance is not supported.

For general web scraping, use different tools or skills.

Valid inputs for this skill:
- mql5.com article URLs
- mql5.com user IDs
- Files containing mql5.com URLs

---

Example 4: User ID Discovery (Valid)

User Input: "Get all articles from MQL5 user 29210372"

Skill Actions:

1. Validates user ID format 1. Confirms scope (mql5.com) 1. Checks article count 1. Prompts for confirmation if >10 articles

Skill Response:

User 29210372 has 77 articles on mql5.com.

Proceed with full extraction? This will:
- Take ~3-5 minutes (2s rate limit between articles)
- Download all images
- Create 77 article folders
- Generate ~400MB of data

Confirm? (Or use --max-articles 5 to test first)

User: "Yes"

Skill Executes:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py discover-and-extract --user-id 29210372

---

Example 5: Testing Limit (Bounded Operation)

User Input: "Test extraction with 3 articles from user jslopes"

Skill Actions:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py discover-and-extract --user-id jslopes --max-articles 3

Output:

Discovered 12 articles for user jslopes
Limited to 3 articles

[Extraction progress...]

✅ Batch processing completed
Total: 3, Successful: 3, Failed: 0
Output: $HOME/eon/mql5-local/mql5_articles/jslopes/

---

Example 6: Ambiguous Keywords (Prompt for Clarification)

User Input: "Extract trading strategies"

Skill Response:

"Trading strategies" could mean different sources.

This skill works with mql5.com ONLY.

Are you looking for:
1. Specific MQL5 article? → Provide URL
2. All articles from MQL5 author? → Provide user ID
3. Articles from different site? → Out of scope

Please clarify your request.

---

Example 7: Batch File Processing (Valid)

User Input: "I have a file with 10 mql5 URLs, extract them all"

Skill Actions:

1. Ask for file path 1. Verify file exists 1. Preview first few URLs 1. Confirm all are mql5.com domain 1. Execute batch

Skill Response:

Please provide the file path containing URLs.

User: "/tmp/mql5_urls.txt"

Skill Verifies:

head -3 /tmp/mql5_urls.txt
# https://www.mql5.com/en/articles/19625
# https://www.mql5.com/en/articles/19624
# https://www.mql5.com/en/articles/19623

Skill Executes:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py batch /tmp/mql5_urls.txt

---

Example 8: Mixed Domain File (Reject Invalid)

User Input: "Extract from this file: mixed_urls.txt"

File Contents:

https://www.mql5.com/en/articles/19625
https://www.yahoo.com/finance/news/article
https://www.mql5.com/en/articles/19624

Skill Response:

❌ INVALID FILE

Line 2 contains non-mql5.com URL: https://www.yahoo.com/finance/news/article

This skill ONLY processes mql5.com URLs.

Options:
1. Remove non-mql5.com lines from file
2. Create mql5-only file
3. Extract valid URLs individually

Would you like me to filter and create mql5-only file?

---

Example 9: Resume Interrupted Extraction (Valid)

User Input: "My batch extraction stopped at article 50, can I resume?"

Skill Response:

Yes! The checkpoint system saved your progress.

Resume command:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py batch [original_file] --resume

This will skip the 50 already-processed articles and continue from article 51.

Checkpoint file: .extraction_checkpoint.json

---

Example 10: Custom Output Location (Valid)

User Input: "Extract article 19625 to /tmp/test_output/"

Skill Executes:

cd $HOME/eon/mql5-local
.venv/bin/python mql5_extract.py --output /tmp/test_output single https://www.mql5.com/en/articles/19625

Output:

✅ Extraction successful!
Output: /tmp/test_output/29210372/article_19625/

---

Bounding Summary

---

Skill Activation Keywords

The skill activates on:

• "mql5", "MQL5", "mql5.com"
• "MetaTrader", "MT5"
• "trading articles", "algorithmic trading"
• "extract mql5", "scrape mql5"
• URLs containing "mql5.com"

The skill rejects on:

• Other domains (yahoo, google, reuters, bloomberg, etc.)
• General "extract articles" without mql5 context
• Non-trading content requests

Skill: MQL5 Article Extractor

Extraction Modes

Mode 1: Single Article

When: User provides one article URL Command:

.venv/bin/python mql5_extract.py single https://www.mql5.com/en/articles/[ID]

Output: mql5_articles/[user_id]/article_[ID]/

Mode 2: Batch from File

When: User has URL file or wants multiple specific articles Command:

.venv/bin/python mql5_extract.py batch urls.txt

Checkpoint: Auto-saves progress, resumable with --resume

Mode 3: Auto-Discovery

When: User provides MQL5 user ID or username Command:

.venv/bin/python mql5_extract.py discover-and-extract --user-id [USER_ID]

Discovers: All published articles for that user

Official Documentation Extraction

Mode 4: Official Docs (Single Page)

When: User wants official MQL5/Python MetaTrader5 documentation (not user articles)

Scripts Location: /scripts/official_docs_extractor.py

Command:

cd $HOME/eon/mql5-local
curl -s "https://www.mql5.com/en/docs/python_metatrader5/mt5copyticksfrom_py" > page.html
.venv/bin/python scripts/official_docs_extractor.py page.html "URL"

Output: Markdown file with source URL, HTML auto-deleted

Mode 5: Batch Official Docs

When: User wants all Python MetaTrader5 API documentation

Scripts Location: /scripts/extract_all_python_docs.sh

Command:

cd $HOME/eon/mql5-local
./scripts/extract_all_python_docs.sh

Result: 32 official API function docs extracted

Key Differences from User Articles

• Different HTML structure (div.docsContainer vs div.content)
• Inline tables and code examples preserved
• No images (documentation only)
• Simpler file naming (function_name.md)
• Source URLs embedded in markdown
• HTML files auto-deleted after conversion

Data Sources

User Collections

• Primary Source: <https://www.mql5.com/en/users/29210372/publications>
• Author: Allan Munene Mutiiria (77 technical articles)
• Content Type: MQL5 trading strategy implementations

Skill: MQL5 Article Extractor

│ │ ├── copyticks_range.md │ │ └── symbol_info_tick.md │ └── user_articles/ # 9 articles by author │ ├── artmedia70/article[ID]/ │ ├── lazymesh/article[ID]/ │ └── ... ├── python_integration/ # Topic collections │ ├── official_docs/ # 32 MT5 Python API functions │ │ ├── mt5initialize_py.md │ │ ├── mt5copyticksfrom_py.md │ │ └── ... │ └── user_articles/ # 15 implementation articles │ ├── dmitrievsky/article[ID]/ │ ├── koshtenko/article\_[ID]/ │ └── ... ├── extraction_summary.json └── extraction.log

`````

Content Organization:

• User Collections (e.g., 29210372/): Articles by specific authors
• Topic Collections (e.g., tick_data/, python_integration/): Organized by research area
• official_docs/: Official MQL5 documentation pages
• user_articles/: Community-contributed articles by author

Quality Verification

After extraction, verify outputs:

````bash

Count articles extracted

find mql5_articles/ -name "article_*.md" | wc -l

Check MQL5 code blocks

grep -r "```mql5" mql5_articles/ | wc -l

View summary

cat mql5_articles/extraction_summary.json `````

Error Handling

If extraction fails:

1. Check logs: tail -f logs/extraction.log 1. Verify URL is mql5.com domain 1. Check internet connection 1. For batch: use --resume to continue from checkpoint

CLI Options Reference

Global options (before subcommand):

• --output DIR - Custom output directory
• --config FILE - Custom config file
• --verbose - Debug logging
• --quiet - Error-only logging

Batch options:

• --resume - Continue from checkpoint
• --no-checkpoint - Disable checkpoint system
• --max-articles N - Limit to N articles

Discovery options:

• --user-id ID - MQL5 user ID or username
• --save-urls FILE - Save discovered URLs to file
• --max-articles N - Limit extraction

Input Bounding Rules

Rule 1: Domain Validation Only accept mql5.com URLs. Reject all other domains immediately.

Rule 2: Input Type Classification Classify user input as:

• URL pattern → single extraction
• Numeric/username → discovery
• File path → batch
• Ambiguous → prompt for clarification

Rule 3: Scope Enforcement If user mentions keywords like "yahoo", "google", "reuters", "bloomberg" → respond with scope limitation message.

Rule 4: Confirmation for Large Operations If discovery would extract >10 articles, confirm with user before proceeding.

Security Notes

• Only executes within $HOME/eon/mql5-local
• Uses virtual environment .venv/bin/python
• No network tools allowed (uses Playwright internally)
• Rate limiting enforced (2s between articles)
• Checkpoint files in project root only

Typical Interaction Flow

1. User mentions MQL5 or trading articles 1. Skill activates and bounds request to mql5.com 1. If input vague → prompt for specifics (URL, user ID, or file) 1. Validate input type and domain 1. Execute appropriate command 1. Show output location and verification commands

Success Indicators

After execution, report:

• Number of articles extracted
• Total word count
• Code blocks found
• Images downloaded
• Output directory location
• Link to extraction summary

---

Remember: This skill ONLY works with mql5.com. Any request for other domains is out of scope and should be rejected with a clear message.