Extending the Artemis Scanner

Crawling pipeline

This is the dependency the whole Period 2 plan sat on. A detector can only test the URLs it gets handed, so whatever coverage the crawler produced was the ceiling on every injection module I was going to write later - which is why it had to be rebuilt before any of them.

Artemis used to do single-page HTML parsing for crawling - no JS execution, no following links, no dedup. M2738 replaced that with Katana (run with -jc so it picks up endpoints buried in JS files) piped through uro for structural deduplication, cached in Redis behind a distributed lock so ten modules scanning the same host trigger exactly one crawl instead of ten, with a shorter cache TTL when the crawl times out so partial results don’t stick around for a full day. Four consumer modules migrated onto it in the same PR.

My original proposal had a third stage - gf for vulnerability-class filtering - and I dropped it. gf filters URLs by matching parameter names against English-language wordlists, but Artemis scans a lot of Polish infrastructure, where parameter names don’t reliably match those wordlists. A genuinely vulnerable parameter with an unusual name could get silently filtered out before any detector ever saw it - an optimization that can hide a real vulnerability isn’t a safe default.

Direct URL scanning

Artemis works out what to do with a target by classifying whatever you hand it, and each class drops into a fixed pipeline of modules. Direct URL scanning just means being able to hand it one exact address and have that address scanned as-is. I pulled it forward because it shortens the loop for everyone - my own testing for the rest of the project, and anyone who already knows the single endpoint they want checked.

Before this, Artemis’s classifier only understood domains, IPs, IP ranges, and ASNs. If we submitted a domain, it went through subdomain enumeration, then port scanning, then fingerprinting before any HTTP module ever saw a task - even if we already knew the exact URL we wanted scanned. M2899 taught the classifier to parse URLs directly and emit the TaskType.SERVICE task without the detour, so every HTTP module picks it up with zero changes - it’s the same task type the port scanner would eventually have produced anyway.

CPE tagging for detected technologies

This started small. Wappalyzer - the fingerprinting library Artemis uses to work out what software a site is running - detects things like Django or MongoDB on a target, but that result only went into Postgres for the report, so no downstream module could ever see it. M2978, fixed that: structured Technology objects carrying CPE (Common Platform Enumeration), version, and categories on the task payload, so downstream modules can route on what’s running on a target instead of reading it back out of the report. A CPE is the standardized name for one specific product at one specific version, and it’s the key vulnerability databases are indexed by - without it, “MongoDB 4.2” is just a string.

This is the tagging work only, with no vulnerability lookup yet, and splitting it out was deliberate. Same pattern as the crawler: the tags had to actually reach downstream modules through the task queue before anything could use them, so this had to land first - without it, the CVE lookup below would have no reliable way to know what product or version it was even checking.

CVE lookup for detected technologies

A CVE (Common Vulnerabilities and Exposures) is the public identifier for one specific known flaw in one specific piece of software, and it’s what makes the CPE strings from the previous deliverable worth carrying around. Building on that tagging from M2978, M2842 is an actual CVE discovery flow: switch the Go Wappalyzer wrapper to FingerprintWithInfo to get CPE strings, then query NVD (National Vulnerability Database) for known CVEs against them.

This is the payoff the tagging work was for, and it’s cheap in exactly the way the rest of the scanner isn’t: the target has already been fingerprinted, so turning “this host runs X 1.2” into “this host runs a version with known reported flaws” is a database question, not more probing traffic aimed at someone’s server.

OS command injection detector

With the crawler and the tagging in place, Period 2 was the detectors. I started with this one because it has the least ambiguous proof of the three: a command either executed on the target or it didn’t, so I could get the shared scaffolding right against a signal I trusted before moving to detectors that have to infer a result from error text.

M2947 implements two detection methods. Output-based: inject an echo of a random per-request marker through each shell separator (;, |, newline, $(...), backticks) and check whether the marker comes back - that proves execution actually happened, not just that the app returned a suspicious error string. Time-based blind: inject sleep N through the same separators and confirm a reproducible delay across several rounds measured against a baseline, for cases where there’s no visible output to check.

Since merging, the detector has already returned a real finding during production scanning - confirmation that the two detection methods hold up against live targets, not just the test suite.

Extracting shared logic

M3028 removed code instead of adding it. The SQL, ORM, and LFI detectors each ran the same sequence at the top of their run() method - crawl the target, add the seed URL back in, union in query-stripped variants, filter out static assets, cap the result - and the three copies had already started drifting apart from each other. I moved it into one function, get_links_to_scan(), in crawling.py, and all three detectors call it instead of carrying their own copy.

A further piece of sharing - a common result-building helper in injection_helpers.py and crawling.collect_parameters - landed inside O3051 (the NoSQL injection PR) instead of its own PR.