Catalogs and Tools

Overview

EmbedXPL-Forge ships a set of developer and CI tools under the tools/ directory and a set of JSON/YAML data catalogs under embedxpl/resources/. These tools support module validation, CVE catalog maintenance, wiki generation, and environment diagnostics.

Data Catalogs (`embedxpl/resources/`)

`firmware_sources.yaml`

Vendor firmware download registry. Keys are lowercase vendor identifiers; each entry maps to portal URL, category, login requirement, and notes.

vendors:
  hikvision:
    name: "Hikvision"
    category: "ip-cameras"
    portal_url: "https://www.hikvision.com/en/support/download/firmware/"
    requires_login: false
    notes: "Direct download links available for most models."
  tplink:
    name: "TP-Link"
    category: "routers"
    portal_url: "https://www.tp-link.com/en/support/download/"
    requires_login: false
  cisco:
    name: "Cisco"
    category: "routers-switches"
    portal_url: "https://software.cisco.com/download/home"
    requires_login: true
    notes: "Requires Cisco account with entitlement."

Consumed by firmware-dl CLI (embedxpl/tools/firmware_downloader.py).

`infra_profiles.yaml`

Infrastructure context taxonomy for the --infra / --context orchestrator mode. Maps (infra, context) tuples to module path prefixes.

Infra	Context	Label	Mapped module paths
`ot`	`ics`	ICS / SCADA	`exploits/ics/`, `scanners/ics/`, `creds/ics/`, `exploits/firewalls/siemens/`, `exploits/firewalls/moxa/`, `exploits/firewalls/hirschmann/`, `exploits/firewalls/schneider/`, `exploits/firewalls/phoenix/`
`ot`	`energy`	Energy / Smart Meters	`exploits/smart_meters/`, `scanners/smart_meters/`
`ot`	`building`	Building Automation (BACnet/HVAC)	`scanners/ics/`, `creds/ics/`, `exploits/bms/`
`it`	`enterprise_network`	Enterprise Network Perimeter	`exploits/firewalls/`, `exploits/switches/`, `exploits/vpn/`, `exploits/appliances/`, `exploits/ngfw/`, `exploits/network_os/`, `scanners/firewalls/`, `creds/firewalls/`
`it`	`hypervisor`	Hypervisors / Virtualization	`exploits/hypervisors/`, `creds/hypervisors/`, `scanners/hypervisors/`
`it`	`bmc_ipmi`	BMC / IPMI / Out-of-Band Management	`exploits/bmc/`, `scanners/bmc/`, `creds/bmc/`
`it`	`ups_power`	UPS / Power Management	`exploits/ups/`, `scanners/ups/`, `creds/ups/`
`iot`	`home`	Home / SOHO	`exploits/routers/`, `exploits/soho_edge/`

`catalogs/` resources

File	Purpose
`market_priority_devices_2010_2026.json`	Market-priority device coverage matrix - device models ranked by deployment volume and CVE density for module prioritization
`module_target_scope.json`	Module-to-target-scope mapping - specifies which target device classes each module addresses
`cve_extended_catalog.json`	Extended CVE reference data - CVSS scores, CWE, affected firmware versions, MITRE ATT&CK techniques, and associated EmbedXPL module paths
`arsenal_layout.json`	Arsenal layout descriptor - describes the visual grouping of modules by attack phase for the arsenal index tool

Tools (`tools/`)

Script	Role
`phase_gate.py`	Automated quality gate - validates modules before release across 7 sequential gate phases
`env_doctor.py`	Environment diagnostics - reports Python version, dependency status, missing optionals
`compat_smoke.py`	Compatibility smoke checks - quick import and interface test for all modules
`gen_wiki_module_index.py`	Regenerate wiki module annex from live module metadata
`generate_coverage_matrix.py`	Build HTML/JSON coverage matrix artifacts from device catalog
`generate_full_catalog.py`	Generate full catalog outputs (JSON + Markdown) for all modules
`refresh_cve_extended_catalog.py`	Fetch and update CVE extended catalog from NVD/MITRE feeds
`validate_market_priority_minimums.py`	Assert that high-priority device models have at least one module
`build_arsenal_index.py`	Build arsenal index JSON from module metadata
`sync_scope_wordlists.py`	Synchronize scope-related wordlists with device catalog
`audit_modules.py`	Audit module metadata completeness - reports missing `__info__` fields
`run_scoped_tests.py`	Run test suite scoped to specific module categories
`validate_governance.py`	Governance validator - checks license headers, author fields, docstrings
`validate_arsenal_architecture.py`	Assert that arsenal layout matches actual module file tree
`build_osi_tcpip_attack_matrix.py`	Generate OSI/TCP-IP attack surface matrix
`deep_intel_backlog.py`	Report modules in the deep intelligence research backlog
`sync_mibs.py`	Sync SNMP MIB files from vendor sources to `resources/mibs/`

Run all scripts from the repository root unless the tool documents otherwise.

Quality Gate (`phase_gate.py`)

Full gate sequence

# Run all 7 gates in sequence (recommended for CI/CD)
python tools/phase_gate.py --all

Individual gates

python tools/phase_gate.py --phase A1A2    # Import + __info__ validation
python tools/phase_gate.py --phase B       # check() and run() body validation
python tools/phase_gate.py --phase C       # Anti-false-positive on port 63994
python tools/phase_gate.py --phase A3      # Prohibited string check
python tools/phase_gate.py --phase D       # flake8 linting gate
python tools/phase_gate.py --phase E       # bandit security gate
python tools/phase_gate.py --phase final   # index_modules() full indexing check

Sample `--all` output

$ python tools/phase_gate.py --all

EmbedXPL-Forge Phase Gate v3.1.0
=================================

[A1/A2] Import and __info__ Validation
---------------------------------------
  Checking 143 modules...
  [OK]  exploits/routers/tplink/wr841n_credential_disclosure_cve_2023_50224
  [OK]  exploits/routers/cisco/rv320_command_injection
  [OK]  exploits/firewalls/fortinet/fortios_auth_bypass_cve_2022_40684
  ...
  [OK]  143/143 modules import cleanly
  [OK]  143/143 modules have complete __info__ (name, description, authors, references, devices, cvss)

[B]    check() and run() Body Validation
-----------------------------------------
  [OK]  143/143 modules have non-stub check() bodies
  [OK]  143/143 modules have non-stub run() bodies

[C]    Anti-False-Positive Gate (port 63994)
---------------------------------------------
  [OK]  143/143 modules return False / None against closed port 63994

[A3]   Prohibited String Check
--------------------------------
  Checking for: em-dash, TODO, PLACEHOLDER, hardcoded IPs (192.168.x.x, 10.x.x.x)...
  [OK]  No prohibited strings found

[D]    flake8 Linting Gate
----------------------------
  [OK]  flake8 passed  -  0 violations

[E]    bandit Security Gate
-----------------------------
  [OK]  bandit passed  -  0 high-severity findings
  [INFO] 3 low-severity findings (eval, subprocess)  -  permitted by policy

[FINAL] index_modules() Full Indexing
---------------------------------------
  [OK]  index_modules() returned 143 modules
  [OK]  All modules appear in global index

=================================
GATE RESULT: PASS (7/7 gates)
Time: 12.4 s
=================================

Gate failure example (A1/A2)

[A1/A2] Import and __info__ Validation
  [FAIL] exploits/routers/acme/new_module  -  ImportError: cannot import name 'OptString'
  [FAIL] exploits/cameras/brand/cve_test  -  __info__ missing required fields: ['cvss', 'references']

GATE RESULT: FAIL (0/7 gates  -  stopped at A1/A2)
Fix the 2 error(s) above and re-run.

Environment Doctor (`env_doctor.py`)

python tools/env_doctor.py

Sample output:

EmbedXPL-Forge  -  Environment Doctor
=====================================
Python          3.12.3          [OK]
pip             24.0            [OK]
requests        2.31.0          [OK]
paramiko        3.4.0           [OK]
yaml            6.0.1           [OK]
rich            13.7.0          [OK]
pysnmp          6.1.1           [OK]
telnetlib3      2.0.1           [OK]   (required for Python 3.13+)
impacket        0.11.0          [OK]
pymodbus        3.7.0           [OK]
torch           2.2.1+cu121     [OK]   (optional  -  GPU compute)
tensorflow      MISSING         [WARN] (optional  -  install with pip install tensorflow)
binwalk         MISSING         [WARN] (optional  -  firmware analysis; install with apt-get install binwalk)
nmap            7.95            [OK]   (required for NSE scripts)

GPU / Compute
  CUDA available  : True  (driver 555.85, CUDA 12.1)
  torch backend   : cuda
  Visible devices : NVIDIA GeForce RTX 4070 (8192 MB)

Summary: 13/14 required deps OK | 1 optional missing | 0 errors

Compatibility Smoke Check (`compat_smoke.py`)

python tools/compat_smoke.py

Sample output:

EmbedXPL-Forge  -  Compatibility Smoke
=======================================
[OK] embedxpl.interpreter imports cleanly
[OK] embedxpl.core.exploit.exploit imports cleanly
[OK] embedxpl.core.hw_profiler imports cleanly
[OK] embedxpl.core.session imports cleanly
[OK] embedxpl.core.apt_catalog imports cleanly
[OK] embedxpl.core.orchestrator imports cleanly
[OK] embedxpl.nse.manager imports cleanly
[OK] embedxpl.tools.firmware_downloader imports cleanly
[OK] embedxpl.tools.firmware_analyzer imports cleanly
[OK] 9/9 critical imports passed

Module scan:
[OK] 143 modules indexed by index_modules()

Smoke result: PASS

Audit Modules (`audit_modules.py`)

python tools/audit_modules.py

Sample output (all fields present):

EmbedXPL-Forge Module Audit
============================
Scanning 143 modules...

Field coverage:
  name          143/143 (100%)
  description   143/143 (100%)
  authors       143/143 (100%)
  references    143/143 (100%)
  devices       143/143 (100%)
  cvss          141/143 (98.6%)  -- 2 missing
  cwe           128/143 (89.5%)  -- 15 missing
  date          143/143 (100%)

Modules missing 'cvss':
  exploits/generic/snmp/snmp_bruteforce
  generic/dns_hijack_detector

Audit complete: 2 warnings

Wiki Module Index (`gen_wiki_module_index.py`)

Regenerates the module index annex from live module metadata. Run after adding new modules:

python tools/gen_wiki_module_index.py

Output is written to docs/wiki/en-US/ANNEX-module-index.md and docs/wiki/pt-BR/ANNEX-module-index.md.

Full Catalog Generator (`generate_full_catalog.py`)

Generates docs/catalog.json and docs/catalog.md from all module __info__ metadata:

python tools/generate_full_catalog.py
# Output:
#   docs/catalog.json    -- machine-readable full catalog
#   docs/catalog.md      -- human-readable Markdown table
#   docs/catalog.html    -- HTML rendering

Sample catalog.json entry:

{
  "path": "exploits/firewalls/fortinet/fortios_auth_bypass_cve_2022_40684",
  "name": "FortiOS Authentication Bypass (CVE-2022-40684)",
  "description": "HTTP/HTTPS authentication bypass on FortiOS management interface allowing full admin access.",
  "authors": ["@mrhenrike"],
  "cvss": 9.8,
  "cves": ["CVE-2022-40684"],
  "mitre": ["T1190"],
  "devices": ["FortiGate", "FortiProxy", "FortiSwitch Manager"],
  "references": [
    "https://www.fortiguard.com/psirt/FG-IR-22-377"
  ]
}

Catalogs and Tools

Overview

Data Catalogs (embedxpl/resources/)

firmware_sources.yaml

infra_profiles.yaml

catalogs/ resources

Tools (tools/)

Quality Gate (phase_gate.py)