A pixel verified website cloning and machine readiness pipeline, explained in full: 16 stages in the order they must run, 106 configured numbers with what each one protects, 40 module contracts, 24 tests, the failures that shaped all of it, and the whole starter kit downloadable from this page. No signup, no email, nothing held back.
Every number below came out of one real run against the test site that ships inside the kit on this page. You can run it yourself and get your own.
Eight readiness axes measured before and after the rebuild. Site score moved from 42 to 100.
That fidelity average is 99.88 and not 100, on a test site with no third party scripts and no fonts to load. This is the honest ceiling: capture is not deterministic, it moves about two points run to run, and the correct response to a score inside that band is to leave it alone.
Rebuilding somebody's website by hand takes days, and the rebuild is never quite the same page. Meanwhile the thing that decides whether that business gets found has changed: people ask an assistant instead of scrolling a results list, and assistants read structure, not layout.
This takes one web address and produces a working copy of that site that looks the same, plus the machine-readable structure the original was missing. It proves the copy looks the same by taking a picture of both and comparing them pixel by pixel, and it refuses to ship a page that failed that comparison.
A folder you can open. The pages, their assets, a score for how closely each page matched, a score for how ready the site is to be read by a machine, and a report that shows both numbers with the failures listed rather than summarized.
If that is enough, the whole thing is downloadable from the kit section with no form and no email. The rest of this page is how it works and why each part is shaped the way it is.
These are published early on purpose. Anyone can show you a passing run. The useful part of a system is the list of ways it already broke, because that is what the numbers in the config are actually defending against.
A network request that arrives while the browser context is already closing rejects when you fulfill it AND rejects when you let it continue. Catching one leaves the other, and an unhandled rejection takes the whole process down.
Capture is not deterministic. Fonts, animation timing and lazy loading land differently run to run, and the fidelity score moves by around two points for reasons that have nothing to do with the clone being better or worse.
Many sites serve mobile from the same document as desktop. Capturing a separate mobile tree on one of those produces a duplicate that immediately starts to drift, and then the diff compares against the wrong baseline.
Past a certain rendered height the browser hands back stale or blank frames instead of the page. The diff then compares two mostly empty images and reports an excellent score.
The heading rewrite runs by default, because it is usually right. It also moves pixels, which is exactly what the fidelity gate exists to catch. Two correct systems disagreeing looks identical to a capture bug.
You cannot select pages you have not surveyed, price a job you have not sized, or verify a page you have not captured. The sequence is a dependency chain, not a preference.
blocked-budget and crashed are not stages. They are the two ways a job exits early, and counting them as steps teaches a pipeline that does not exist. A crash writes the phase it died in so a resume knows where to pick up.
This is the arithmetic the pipeline runs at the verify stage, executing in your browser on two real capture screenshots from the run above. Nothing is fetched and nothing is precomputed.
Read what these two images are before you read the number. They are captures of two different pages of the test site, not an original and its clone. That is deliberate, because it gives the demo both controls a measurement needs: comparing a capture with itself must return exactly 100, and comparing two genuinely different pages must return something poor. A tool that cannot produce both answers is not measuring anything.
Both images are downscaled here to keep this page's weight sane, so the number this demo computes is the arithmetic, not the official score from the run. The official score is in the first section and it came from the full resolution frames.
A live pixel comparison of two captured screenshots, computed in the browser.
Every stage below carries the same five fields, filled in every time. A blank in a table like this is where a system stops being explainable.
You cannot clone what you have not counted.
survey.maxPages 10000, survey.maxDepth 4, survey.requestDelayMs 350, survey.requestTimeoutMs 15000, and 2 more in the table belowA bounded page set is a promise you can keep.
Price the job before the job prices you.
pricePerMTokIn 3, pricePerMTokOut 15, levels.1.label "Verbatim + optimization", levels.1.tokensInPerPage 0, and 11 more in the table belowCapture the rendered page, not the source.
capture.concurrency 4, capture.perHostDelayMs 500, capture.navTimeoutMs 30000, capture.settleExtraMs 800, and 12 more in the table belowRemove what phones home. Change nothing that shows.
trackerDomains ["google-analytics.com","googletagmanager.com","connect.facebook.net","hotjar.com","clarity.ms"], keepEmbedDomains ["youtube.com","youtube-nocookie.com","player.vimeo.com","google.com/maps"], bannerSelectors ["#cookie-banner",".cookie-consent","[id*=\"gdpr\"]","[class*=\"consent\"]"], residualScriptPatterns ["dataLayer","gtag(","fbq(","_paq.push"]A page that machines cannot read is invisible to the machines people now ask.
meta.section "Metadata and Open Graph", meta.evidence "DATA", meta.note "Complete titles, descriptions and OG tags feed classic snippets and AI answer cards. Titles are never rewritten; over-60-character titles become audit items.", schema.section "Structured Data", and 41 more in the table belowA clone that looks right and does nothing is a screenshot.
Rewriting words is cheaper than rewriting layout, and safer.
levels.2.label "Refresh", levels.2.tokensInPerPage 6000, levels.2.tokensOutPerPage 2500Keep the facts, replace the frame.
levels.3.label "Full redesign", levels.3.tokensInPerPage 12000, levels.3.tokensOutPerPage 8000The clone is only as good as the number that compares it.
detection.sameDocElementRatio 0.85, detection.sameDocTagSimilarity 0.9, detection.sameDocTextSimilarity 0.8, detection.ambiguousBand 0.03, and 5 more in the table belowA fix that moves pixels is not a fix. It is a redesign nobody asked for.
verify.passThreshold 99Click it. A widget that renders is not a widget that works.
capture.viewports.desktop.width 1440, capture.viewports.desktop.height 900, capture.viewports.mobile.width 390, capture.viewports.mobile.height 844Fast is a feature agents measure.
Ship a folder someone else can open without you.
package.zipName "{jobId}-{domain}-clone.zip"Every job should make the next job cheaper.
Done is a state the machine writes, not a feeling the operator has.
Where these came from, and where they do not apply. Every number here was tuned on brochure style sites: a marketing site of tens to low hundreds of pages, server rendered or lightly hydrated, with a normal amount of imagery. Other classes of site behave differently. A heavy single page application needs a longer hydration wait and will still fail the architecture check. A media site will hit the asset ceilings long before the page ceiling. A store is gated out at intake rather than tuned for.
How to check them against your own site, without guessing. Run one job with the defaults and read three things: the survey count against what you believe the site has, the per page fidelity spread rather than the average, and the asset byte total against the ceiling. A ceiling you never reach is the wrong ceiling to tune. Change one number, re-run the same site, and compare the same three readings. Changing two at once tells you nothing about either.
One rule outranks all of them: never raise a threshold to clear a gate. The threshold is the product. Raising it converts a known failure into an unknown one.
| Key | Value | Stage | What it protects |
|---|---|---|---|
version |
1 | Structural. Not a tuning knob. | |
port |
3434 | Structural. Not a tuning knob. | |
survey.maxPages |
10000 | SURVEY | The machine. A faceted filter or a calendar generates URLs forever; this is the stop. |
survey.maxDepth |
4 | SURVEY | Relevance. Past four hops from the home page you are crawling archives, not the business. |
survey.requestDelayMs |
350 | SURVEY | Somebody else's server. This is the politeness budget and it is not a performance setting. |
survey.requestTimeoutMs |
15000 | SURVEY | The queue. One hanging request must not stall the crawl. |
survey.userAgent |
"YourCloner/0.1 (+https://example.com/crawler; site rebuild tool)" | SURVEY | Your reputation. Identify yourself and point at a page explaining what you are doing. |
survey.quickPreviewPages |
5 | SURVEY | Serves the SURVEY stage. You cannot clone what you have not counted. |
capture.concurrency |
4 | CAPTURE | Your own machine. Each unit is a browser tab holding a full page in memory. |
capture.perHostDelayMs |
500 | CAPTURE | The origin server, again, this time under parallel load. |
capture.navTimeoutMs |
30000 | CAPTURE | The job. A page that will not load is a page you skip, not a job you lose. |
capture.settleExtraMs |
800 | CAPTURE | Fidelity. Animations and late layout land in this window. |
capture.maxHydrationWaitMs |
5000 | CAPTURE | Correctness on framework sites. Serialize before hydration and you capture empty containers. |
capture.viewports.desktop.width |
1440 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.viewports.desktop.height |
900 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.viewports.mobile.width |
390 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.viewports.mobile.height |
844 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.tabletHarvest.width |
834 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.tabletHarvest.height |
1112 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.tabletHarvest.settleMs |
400 | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
capture.maxAssetBytes |
26214400 | CAPTURE | Disk, per asset. A single video can outweigh an entire site. |
capture.maxTotalAssetBytes |
524288000 | CAPTURE | Disk, per job. The ceiling that actually binds. |
capture.maxScreenshotHeightPx |
12000 | CAPTURE | The diff. Beyond this height the browser returns stale or blank frames and the comparison silently lies. |
capture.retryLadder |
[0,1500] | CAPTURE | Serves the CAPTURE stage. Capture the rendered page, not the source. |
detection.sameDocElementRatio |
0.85 | VERIFY | The architecture verdict: whether mobile is the same document or a separate one. |
detection.sameDocTagSimilarity |
0.9 | VERIFY | Serves the VERIFY stage. The clone is only as good as the number that compares it. |
detection.sameDocTextSimilarity |
0.8 | VERIFY | Serves the VERIFY stage. The clone is only as good as the number that compares it. |
detection.ambiguousBand |
0.03 | VERIFY | Serves the VERIFY stage. The clone is only as good as the number that compares it. |
detection.enforce |
true | VERIFY | Whether the verdict blocks or merely advises. |
verify.passThreshold |
99 | VERIFY | The promise. Below this a page is not a clone, and this number outranks every optimizer. |
verify.warnThreshold |
95 | VERIFY | The escalation line. Between warn and pass a human looks. |
verify.pixelmatchThreshold |
0.1 | VERIFY | Signal. Too tight and antialiasing reads as failure; too loose and real breakage reads as success. |
verify.tileHeightPx |
2000 | VERIFY | Memory during the diff of a tall page. |
package.zipName |
"{jobId}-{domain}-clone.zip" | PACKAGE | Serves the PACKAGE stage. Ship a folder someone else can open without you. |
booking.enabled |
false | Structural. Not a tuning knob. | |
booking.endpoint |
"" | Structural. Not a tuning knob. | |
booking.successMessage |
"Thanks. We got your request and will call you back shortly." | Structural. Not a tuning knob. |
| Key | Value | Stage | What it protects |
|---|---|---|---|
version |
1 | Structural. Not a tuning knob. | |
note |
"AI spend model per optimization level. Level 1 uses no AI. Token estimates per page; prices per million tokens." | Structural. Not a tuning knob. | |
model |
"claude-sonnet-5" | Structural. Not a tuning knob. | |
pricePerMTokIn |
3 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
pricePerMTokOut |
15 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.1.label |
"Verbatim + optimization" | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.1.tokensInPerPage |
0 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.1.tokensOutPerPage |
0 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.2.label |
"Refresh" | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.2.tokensInPerPage |
6000 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.2.tokensOutPerPage |
2500 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.3.label |
"Full redesign" | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.3.tokensInPerPage |
12000 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
levels.3.tokensOutPerPage |
8000 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
phase2Level1.tokensInPerPage |
3000 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
phase2Level1.tokensOutPerPage |
900 | ESTIMATE | Serves the ESTIMATE stage. Price the job before the job prices you. |
defaultBudgetUsd |
10 | ESTIMATE | The invoice. The gate runs before the spend, not after. |
warnAtBudgetRatio |
0.8 | ESTIMATE | Warning lead time before the budget gate fires. |
| Key | Value | Stage | What it protects |
|---|---|---|---|
version |
1 | Structural. Not a tuning knob. | |
trackerDomains |
["google-analytics.com","googletagmanager.com","connect.facebook.net","hotjar.com","clarity.ms"] | TRANSFORM | Serves the TRANSFORM stage. Remove what phones home. Change nothing that shows. |
keepEmbedDomains |
["youtube.com","youtube-nocookie.com","player.vimeo.com","google.com/maps"] | TRANSFORM | Serves the TRANSFORM stage. Remove what phones home. Change nothing that shows. |
bannerSelectors |
["#cookie-banner",".cookie-consent","[id*=\"gdpr\"]","[class*=\"consent\"]"] | TRANSFORM | Serves the TRANSFORM stage. Remove what phones home. Change nothing that shows. |
residualScriptPatterns |
["dataLayer","gtag(","fbq(","_paq.push"] | TRANSFORM | Serves the TRANSFORM stage. Remove what phones home. Change nothing that shows. |
| Key | Value | Stage | What it protects |
|---|---|---|---|
_source |
"https://[redacted]/ (SwarmSystem SEO + AI-visibility playbook). Evidence labels: DATA = measured correlation or platform documentation; MIXED = partial evidence; OPINION = forward-looking convention." | Structural. Not a tuning knob. | |
meta.section |
"Metadata and Open Graph" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
meta.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
meta.note |
"Complete titles, descriptions and OG tags feed classic snippets and AI answer cards. Titles are never rewritten; over-60-character titles become audit items." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
schema.section |
"Structured Data" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
schema.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
schema.note |
"Does not directly cause citations; feeds rich results, the local pack and the knowledge graph. One connected @graph per page, fact-gated fields only, no FAQPage (dead for general sites, May 2026)." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
semantic.section |
"Canonical and Semantics" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
semantic.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
semantic.note |
"Canonical hygiene prevents duplicate-content splits; lang and landmarks help parsers." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
robots.section |
"AI Crawler Access" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
robots.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
robots.note |
"Named Allow blocks for search and AI assistant crawlers; Bytespider blocked (scraper, no citation value)." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
llms.section |
"Not Recommended" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
llms.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
llms.note |
"Zero citation correlation measured; emitted as zero-cost bonus surface only. Scores 0 in readiness v2." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
agentsJson.section |
"Agent Surface" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
agentsJson.evidence |
"OPINION" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
agentsJson.note |
"Emerging convention; low cost, forward-looking capability manifest." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
mdTwins.section |
"Agent-Readable Content" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
mdTwins.evidence |
"MIXED" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
mdTwins.note |
"Markdown twins give agents a clean text surface; adoption evidence still early." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
webmcp.section |
"Agentic Actions (WebMCP)" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
webmcp.evidence |
"OPINION" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
webmcp.note |
"Google I/O 2026 direction; declarative form tools so agents know what actions the site supports." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
iconFonts.section |
"Icon Font Repair" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
iconFonts.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
iconFonts.note |
"Origin-absolute font-face URLs break glyph fonts offline and CORS-fail online; repaired to captured local assets so hamburger, social and detail icons render. Unresolved fonts (never fetched by the origin page) are audit items." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
images.section |
"Media Performance" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
images.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
images.note |
"Intrinsic dimensions remove layout shift; lazy below-fold plus hero preload move LCP. Core Web Vitals are ranking signals." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
links.section |
"Internal Linking" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
links.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
links.note |
"Orphan and deep pages crawl poorly; descriptive anchors carry relevance. Audit-only: visible changes need a human." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
headings.section |
"Heading Structure" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
headings.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
headings.note |
"Exactly one H1 per page; fixes are attempted and kept only when the pixel gate passes, auto-reverted otherwise." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
sitemap.section |
"Honest Sitemap" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
sitemap.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
sitemap.note |
"Real pages only, lastmod from capture time, no m/ duplicates, no priority or changefreq noise." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
canonical.section |
"Mobile Canonical" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
canonical.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
canonical.note |
"m/ pages canonicalize to the desktop URL so the duplicate tree never competes with the primary." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
architecture.section |
"Site Architecture" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
architecture.evidence |
"DATA" | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
architecture.note |
"Per-site single-document vs ua-split verdict from structural comparison of what the origin serves desktops and phones. One document ships as one file per page; two documents stay faithfully mirrored and loudly labeled, with Level 3 as the one-file conversion path." | OPTIMIZE | Serves the OPTIMIZE stage. A page that machines cannot read is invisible to the machines people now ask. |
A flat list of file names teaches nothing. Grouped by stage, the same list shows you where the weight sits: the optimize stage carries the most code because it is doing the most different jobs, and the four capture modules exist because desktop, mobile, harvest and extraction are genuinely different problems.
Each contract below is the module's own opening docblock, taken from source rather than written for this page. Where one reads oddly, that is the real state of the comment.
crawler.js298 linesSURVEY stage: discover every page on a site. Order of attack: sitemap.xml (incl. sitemap indexes) → BFS link crawl fallback.
buildExclude classifyPage extractLinks isExcluded looksLikeSoft404 surveycore-pages.js136 linesCORE PAGES, bounded root-page selection for the Siri Fix profile.
EXCLUDE_PATTERNS EXCLUDE_PATTERN_SOURCES selectCorePagesdetect-archetype.js101 linesARCHETYPE DETECTOR: decide whether a surveyed site is a directory/listing site (MyListing/Directorist-genre: a large repeated listing slug family + a category taxonomy + a listing archive) or a generic site. Pure function over the survey report; no...
detectArchetypecost-tracker.js83 linesCost estimation + guardrails. Estimates AI spend from the survey BEFORE submit and enforces a hard budget cap (warn at ratio, pause at cap). Level 1 uses no AI, so its estimate is $0 in Phase 1 (capture is local + free).
CostMeter checkBudget estimate estimateAll tokenCostcapturer.js498 linesCAPTURE stage. For each page, three sub-steps executed inside one Playwright page: (a) SANITIZE - block tracker domains at the network layer, strip cookie banners, freeze CSS animations. Snapshot + screenshot see the SAME stable state, so VERIFY is fair by...
assetExt autoScroll capture recordAsset rewriteHtml sanitizePage serializeDom waitForFontscapture-mobile.js99 linesMobile-variant capture. Site builders (Wix classic and kin) serve a DIFFERENT document per user agent: separate viewport meta (width=320), separate layout vars, even a reduced component tree. A desktop snapshot therefore cannot be pixel-right on real...
MOBILE_UA captureMobileVariantcapture-harvest.js317 linesCapture-time interaction harvesting. Framework sites (Radix/Headless/Next) lazy- render collapsed/portalled widgets, so their content is absent from the static DOM. Running on the LIVE page (React still active), we open each widget, harvest its content,...
harvestInteractions harvestMobileMenu harvestPopupMenu preExpandAccordions resolveAnchorLinkscapture-extract.js289 linesCapture-side extraction helpers: IR entities/structure from the live page, and CSS url() localization so the clone is CDN-independent. Split from capturer.js to keep files under 500 lines.
applyArchitectureVerdict assetExt computeArchitecture extractEntities extractMediaGeometry extractStructure localizeCssUrls promoteMobileShots recordAsset tabletAssetHarvesttransformer.js132 linesTRANSFORM stage (Level 1). Pure HTML->HTML passes with a no-visual-change contract: reads original/, writes clone/. `original/` stays immutable.
copyTree listFiles transformextract-listings.js201 linesLISTINGS EXTRACTION (Directory Mode): walk every captured /listing/ page in slug order and parse each once into a canonical record. listings.json is the single source of truth for the client-side engine, the agent surfaces, and the directory schema....
extractListings parseListingextract-taxonomy.js284 linesTAXONOMY INGESTION (Directory Mode): the captured taxonomy archive pages (listing-category/<term>/, single-tag/<term>/ and their page/N/ pagination) are membership lists. Harvest them, merge into listings.json, and reconcile.
applyTaxonomy extractMemberSlugs harvestTaxonomy normalizeSlug reconcileTaxonomy runTaxonomyStage walkArchiveTermoptimization-passes.js373 linesOPTIMIZE stage: makes the clone agentic-ready. Every pass is deterministic (no API key) and NON-VISUAL: it only touches <head>, appends JSON-LD, or emits sibling files (llms.txt, markdown twins, robots, agents.json). VERIFY runs after and must still pass,...
optimize optimizeAllTrees passAgentsJson passIconFonts passLlms passMdTwins passMeta passRobots passSchema passWebMcpoptimization-links.js109 linesLink graph analysis (Tier-3 audit: nothing visible is auto-fixed) and the deterministic heading plan (Tier-2: applied later behind the pixel gate).
applyHeadingPlan passHeadings passLinkGraphoptimization-media.js157 linesTier-1 media pass: image repairs that are provably non-visual, driven by capture-time geometry (page.media / page.mediaMobile from the IR).
assetFile passImagesreadiness-score.js200 linesAI Readiness Score v2. Rates how ready a page is for AI/agent discovery on a 0-100 scale across seven axes. Runs on a page's HTML (works for both the ORIGINAL captured page and the OPTIMIZED clone) plus site-level file checks, so the before/after delta is...
ALLOW BLOCK WEIGHTS scorePage scoreSite scoreTree treeSiteFactsschema-directory.js137 linesDIRECTORY SCHEMA: the archetype-gated branch of passSchema. When site.archetype === 'directory', pages get directory-correct types instead of the LocalBusiness-anchored graph: home WebSite + SearchAction (live client-side search) + Organization...
passSchemaDirectoryagent-surfaces.js69 linesAGENT SURFACES (Directory Mode): emit the machine-readable directory views from listings.json, the single source of truth. One parse, three surfaces: 1. /.well-known/directory.json - full corpus for agent consumption 2. /agents.json enrichment - adds the...
emitDirectorySurfacesicon-fix.js93 linesIcon-font repair pass (site scope). Phase 0 evidence ([redacted] [redacted]): origin (https://site.com/...), so glyph fonts fail offline outright and fail online without CORS. Two observed shapes: 1. the localizer mapped the file but emitted the origin base...
passIconFontsinteraction-detect.js262 linesInteraction recognizer. Given a page's cheerio $, detect HIGH-CONFIDENCE interaction patterns (mobile nav, FAQ/accordion, tabs, gallery) and annotate the DOM with data-bx-* hooks that behaviors.js wires at runtime.
annotate capturedHiddeninteraction-pass.js162 linesInteraction pass (publish-time behavior restore). Runs on a shipped tree (clone/ or clone-refresh/). For each page it: 1. Annotates high-confidence interaction patterns with data-bx-* hooks (lib/interaction-detect.js): non-destructive, no initial-pixel change.
guessType inertScripts reactivateSearchForms runInteractionPasswidget-registry.js309 linesWidget registry: Elementor-first recognition with the generic detector as fallback. ONE pure entry point, annotateDom($, opts), drives BOTH doors: capture-time annotation (fresh runs) and clone-side re-annotation (retrofit), so fixes learned in retrofit...
annotateDom parseSettingsredesign.js366 linesLevel 2 REFRESH. Keeps each page's DOM structure and layout, but rewrites the visible copy in the operator's voice and applies a conservative modern polish.
HUMANIZE RULES blockType collectBlocks extractReferenceFacts gate injectPolish refresh verifyRubricredesign-level3.js536 linesLevel 3 FULL REDESIGN. Keeps the CONTENT (headings, copy, images, links, NAP) but rebuilds the DESIGN from scratch: a clean, light, conversion-first template for a local home-services business. Emits a self-contained page (inline CSS, system font stack,...
detectTheme extractModel redesign renderPage verifyRubricverifier.js418 linesVERIFY stage. Serve clone/ locally, re-render each page in Playwright, and diff its screenshots against the CAPTURE-TIME ground-truth screenshots (never a live re-fetch of the source, which can change mid-job). Produces a per-page fidelity score plus a...
diffPng plainSummary renderHighlightedTree screenshotTree serveDir verify verifyPagesfeature-parity.js125 linesFEATURE-PARITY MANIFEST: enumerate every interactive surface of the original site and disposition each one against the built clone, so "feature-complete" is a checked list, not a feeling. Standard Directory Mode stage output.
buildManifestaffiliate-gate.js73 linesAFFILIATE INTEGRITY GATE: prove that every affiliate URL (?fpr= / &fpr=) captured from the original site survives the full pass chain into the emitted clone unchanged. Count equality AND set equality; any mutation, loss, or invention blocks the deploy.
affiliateGate collectAffiliateUrlsdoc-compare.js144 linesDocument comparator behind the SINGLE-DOCUMENT vs UA-SPLIT verdict.
DEFAULTS aggregateVerdict compareDocsbehavior-verify.js351 linesBehavior verification gate. Serves a shipped clone OFFLINE on an ephemeral port and drives its widgets with Playwright across 3 viewports, asserting the per-family state changes the runtime is supposed to produce. This is the proof layer: it turns "we...
DRIVERS VIEWPORTS familiesOnPage pickSample verifyBehaviorsspeed-proof.js146 linesSpeed proof ("3/3 agentic browser ready"). Runs Lighthouse against the clone served locally by the verifier's serveDir, desktop + mobile emulation, on 3 sampled pages. Records LCP / TBT (lab proxy for INP) / CLS + the SEO category score to...
GREEN samplePages speedProofpackager.js434 linesPACKAGE stage. Emits publishable artifacts from the clone tree: - sitemap.xml (regenerated from the final page map) - robots.txt (Phase 2 adds AI-crawler allowances) - report.html (self-contained brand-styled proof: fidelity, cost, log) -...
DEFECTS_16 buildRobots buildSitemap familyVerdicts isAffiliateUrl packageJob rebaseClone renderReportdeploy-manifest-gate.js134 linesDEPLOY-MANIFEST GATE (all archetypes, default-on): a built page must never reference an asset that is not in the deploy tree, and an injected runtime script must appear exactly once per page. This is the tree-integrity half of the never-again pair; the...
collectAssetRefs deployManifestGate verifyRefsbooking-adapter.js151 linesAgentic booking adapter (publish-time). When a booking endpoint is configured (configs/cloner.json booking.endpoint, env GHL_BOOKING_WEBHOOK, or per-job), this makes the downloaded static site actually bookable: 1. Rewires each contact/quote form to POST a...
resolveEndpoint wireBookingpattern-store.js139 linesPattern library. Every clone deposits a structured record per page (page type, heading shape, schema types, nav archetype, CTA/form/image counts, readability) to page-patterns.jsonl in the job dir. An aggregate index (data/pattern-index.json) is rebuilt...
deposit pageRecord rebuildIndexconfig.js40 linesCentral config loader. All moving-standards config lives in configs/*.json (versioned) so standards updates never require code edits.
CONFIG_DIR DATA_DIR jobDir loadConfig resetConfigCacheutils.js176 linesShared helpers: atomic writes (data-safety rules), URL normalization, page→file mapping, path sanitization, small async utilities.
appendJsonl contentHash normalizeUrl pageFilePath parseSrcset readJsonSafe readJsonl safeJoin sameSite sanitizeSegment sleep withTimeout writeJsonAtomicfuzzy.js71 linesVendored fuzzy matcher for the directory engine. Zero dependencies, shared verbatim between Node (extraction-time checks, tests) and the browser (public/directory-engine.js loads this via a <script> copy).
fuzzy subsequencellm.js137 linesLLM provider for the redesign levels. CLI-first: shells out to `claude -p` (subscription auth, no per-token billing, matches the operator's setup). Falls back to the Anthropic API only if the CLI is unavailable and a key is set.
available generate hasCli parseJsonjob-runner.js346 linesJob orchestration shared by CLI and server. Runs the pipeline and appends every event to progress.jsonl (the single source of truth). The server tails that file to the browser over WebSocket; reconnect/replay comes for free.
finalizeTier2 listJobs patchJob runJob siteFacts startSurveyem-dash-scan.js65 linesEM-DASH GATE: zero em dashes (raw or entity-encoded) in ARTIFACTS WE AUTHOR - lib code, gate reports, readiness copy, engine/runtime JS, agent-surface meta.
emDashScandescribe-clone.js107 linesThe self-describing deliverable. One data source (the architecture verdict + capture facts + extracted entities + an occurrence scan of the clone) emits every surface that tells the owner how to maintain the folder: - EDIT-ME.txt at the clone root (for the...
COPY describeCloneThe full build ships 24 test files. They are listed here by what each one holds in place rather than by what it calls, because the second list is the one that goes stale. The kit carries the same list with a note on which are worth writing first.
| File | What it defends |
|---|---|
behavior-verify.test.js | behavior verify |
behaviors-runtime.test.js | behaviors runtime |
copy.test.js | copy |
core-pages.test.js | core pages |
cost-package.test.js | cost package |
crawler.test.js | crawler |
describe.test.js | describe |
detection.test.js | detection |
directory-mode.test.js | directory mode |
doc-compare.test.js | doc compare |
geometry.test.js | geometry |
icon-fix.test.js | icon fix |
integration.test.js | integration |
interaction.test.js | interaction |
media-links.test.js | media links |
mobile-parity.test.js | mobile parity |
optimize.test.js | optimize |
phase3.test.js | phase3 |
scorer.test.js | scorer |
seo.test.js | seo |
srcset.test.js | srcset |
tablet-harvest.test.js | tablet harvest |
tier2-gate.test.js | tier2 gate |
widget-registry.test.js | widget registry |
The honest answer first. No dedicated skill and no dedicated agent runs this pipeline. It is run three ways and only three ways: a command line entry point, a local server with a browser interface on top of the same pipeline, and a dashboard module that calls the same server. Anyone telling you a swarm of agents clones the site is describing something else.
The word agent is doing three different jobs in this space, and conflating them is how people end up building the wrong thing. Here are all three, separated.
What executes the pipeline. Here that is plain code with a stage loop. Deterministic, resumable, and boring on purpose. This is the part people imagine is an agent. It is not.
Two optional stages hand a page to a language model and take structured output back. That is the only place a model touches the work, it is bounded by a prompt contract, and its output is still subject to the same pixel gate as everything else.
The finished site carries machine readable files so that other people's agents can read it. This is the opposite direction: not an agent running the system, but the system feeding agents.
Both optional stages use the same shape: a fixed list of facts the model may not invent, a block contract that says what structure must come back, and a rubric the output is graded against before it is accepted. The prompts below are templates carrying that shape, not our production prompts.
This is a TEMPLATE, not a production prompt. It carries the shape that matters and leaves
the voice to you. The shape is the transferable part: a block contract, a fixed fact list, and a
rubric the output is graded against before it is allowed to land.
## Why it is shaped this way
The refresh pass rewrites words while leaving layout alone. That only holds if the model returns
the same blocks it was given. Ask for prose and you get prose, and the layout is gone. So the
prompt hands over an array of blocks and demands the same array back.
Every fact the model is allowed to state is extracted from the source page first and passed in as
a closed list. A model with no fact list invents phone numbers.
## The template
ROLE
You rewrite website copy. You do not design, you do not restructure, and you do not add facts.
INPUT
blocks: [{ id, type, text }] the page, one entry per editable block
facts: { name, phone, address, services[], hours } the ONLY facts you may state
brief: { audience, tone, readingLevel, banned[] }
HARD CONSTRAINTS
1. Return exactly one entry per input block, same ids, same order.
2. Never state a fact that is not in `facts`. If a block needs one you do not have, keep the
original text for that block and set `kept: true`.
3. Stay within {minRatio} and {maxRatio} of the original character count per block
[ ... ] The full template ships in the kit. Use the copy button above for all of it.This is a TEMPLATE, not a production prompt. Same reasoning as the copy prompt: the shape
transfers, the taste does not.
## Why it is shaped this way
The redesign pass keeps the content and replaces the frame. The failure it exists to prevent is a
beautiful page that says something untrue, so the model never sees the original HTML. It sees an
extracted content model and a theme, and it renders from those.
Handing over the original markup invites the model to copy its structure, which defeats the point,
and to keep its stale facts, which is worse.
## The template
ROLE
You lay out a page from a content model and a theme. You render, you do not write.
INPUT
model: {
title, headings[], paragraphs[], images[{src,alt,width,height}],
links[{href,text}], nap: { name, address, phone }
}
theme: { palette[], displayFont, bodyFont, density, radius }
page: { type, purpose }
HARD CONSTRAINTS
1. Every string you output comes from `model`. You may reorder and you may omit. You may not
invent, and you may not paraphrase the NAP.
2. Use only colours in `theme.palette`. The palette is closed.
3. Every image keeps its width and height attributes. Layout shift is a scored defect.
4. Exactly one top level heading.
5. Self-contained output: no external stylesheets, fonts or scripts.
OUTPUT
One complete HTML docume
[ ... ] The full template ships in the kit. Use the copy button above for all of it.Every clone ships these so that a machine reading it does not have to infer structure from layout. This page emits the same three, because a page that teaches this and does not do it is not worth reading.
Typed structured data in the head, describing what the page is. Present in this page's head. Open the source and read it.
A plain text summary at a known path, for readers that want the shape of a site without crawling it. Served at /llms.txt and printed in full below.
A machine readable statement of what is here and what may be done with it. Served at /agents.json and printed in full below.
These are not screenshots of a good idea. They are the exact bytes served at those two paths, printed here from the same source that writes the files, plus a robots.txt that allows the readers this page keeps talking about. A page that scores sites on machine access and then blocks it would not be worth the reading.
# The Website Clone System > How a website cloning and agentic optimization pipeline works, stage by stage, with every threshold, every module contract, the failures that shaped it, and the whole starter kit downloadable from the page. Nothing gated. Home: https://clone-system.swarmsystem.ai/ A complete, ungated explanation of a website cloning and agentic optimization pipeline, plus a downloadable starter kit carried inside the page itself. ## Contents - The pipeline: 16 stages in dependency order, each with the principle it enforces - Thresholds: 106 configured values, each with what it protects - Modules: 40 library contracts grouped by the stage they serve - Tests: 24 test files with what each defends - Failures: five published failures with their cost and their fix - Operations: order states, concurrency governor, gates, delivery clock, alert types - The kit: 70 files, 184.4 KB, downloadable with no signup ## Boundaries - The kit is a templatized starter kit. Engine source is not included. - Four of 16 stages ship implemented; the rest ship as declared contracts. - No client data of any kind appears on this page or in the kit. - The tool contains no detection evasion and defaults to polite crawling. ## Author SwarmSystem
{
"version": "1.0",
"name": "The Website Clone System",
"description": "How a website cloning and agentic optimization pipeline works, stage by stage, with every threshold, every module contract, the failures that shaped it, and the whole starter kit downloadable from the page. Nothing gated.",
"url": "https://clone-system.swarmsystem.ai/",
"author": "SwarmSystem",
"license": "Documentation and starter kit are free to use, adapt and redistribute.",
"gated": false,
"contents": {
"stages": 16,
"thresholds": 106,
"modules": 40,
"tests": 24,
"kitFiles": 70,
"kitBytes": 188823
},
"downloads": [
{
"name": "clone-system-kit.zip",
"type": "application/zip",
"mechanism": "assembled in the browser from text embedded in this page"
},
{
"name": "clone-system.md",
"type": "text/markdown",
"mechanism": "assembled in the browser from text embedded in this page"
}
],
"boundaries": [
"Engine source is not included. This is a starter kit.",
"Four of sixteen stages ship implemented; twelve ship as declared contracts.",
"No client data appears anywhere on this page or in the kit.",
"No detection evasion features. Polite crawl defaults are in the shipped config."
],
"usage": "Rebuild sites you are authorized to rebuild. The tool does not decide that for you."
}If you do want to drive this from an assistant, the kit carries a skill definition and
an agent definition as templates. They are starting points shaped like the real thing, and they are
in the file list below under skills-and-agents/. What they are not is a claim that we
run it that way.
skills-and-agents/README.mdskills-and-agents/clone-system.skill.mdskills-and-agents/clone-operator.agent.mdThe moment more than one of these runs at a time you are no longer running a tool, you are running a queue. This is the layer above the pipeline: what state an order is in, what stops too many at once, and what has to be true before a person sees the result.
| State | Plain word | What it means |
|---|---|---|
RECEIVED |
WAITING | A paid order has arrived from your funnel. Nothing has been spent on it yet. |
SCANNED |
WAITING | It passed intake checks and is queued, waiting for a free clone slot. |
CLONING |
WORKING | A dedicated process is capturing and optimizing the site. |
QA_READY |
WORKING | The clone finished and passed the readiness gate. It is ready for a person to look at. |
TEAM_REVIEW |
IN REVIEW | It is live on an internal, non-indexed preview host and posted to your team channel for review. |
APPROVED |
APPROVED | A reviewer signed off. It is waiting to be delivered. |
DELIVERED |
DONE | It went to the customer and the evidence was written down. |
HELD |
HELD | The machine stopped it on purpose and said why. This one needs a human decision. |
ERROR |
ERROR | Something broke. Retry and escalation rules take over from here. |
| Ceiling | Value | What it protects |
|---|---|---|
| Machine ceiling | 4 clones at once | The whole machine, across all offers. The hard bound. |
| Per-offer concurrency | 4 (offer-a), 4 (offer-b) | One offer can never starve another |
| Daily cap | 8/day (offer-a), 20/day (offer-b) | The human QA and delivery side of the pipeline |
Two gates sit on this path: E-commerce gate at intake, and AI-readiness gate at clone complete. The first refuses work the pipeline is not honest about doing. The second refuses to show a person a result that has not cleared the floor, which is the difference between a review queue and a slush pile.
Every one of these is a thing the system will not do, stated before you find out the expensive way.
A site of roughly two hundred pages takes about 45 to 55 minutes end to end on one machine, and most of that is capture waiting for real pages to settle. A handful of pages is minutes. Anyone promising a large site in seconds is not doing the verification step.
Anything behind a login, a cart, a checkout, or a session belongs to the application and not to the page. Those are gated out at intake on purpose rather than half captured.
Navigation, accordions, tabs, dropdowns and galleries are recognized and re-attached. A bespoke widget outside those families ships inert, and the report says so rather than leaving you to discover it.
The original's server is not yours. Submissions are stopped rather than pointed somewhere new, and pointing them somewhere new is your job.
The score compares the clone to the pictures taken at capture time, not to the live site as it is today. If the original changes afterwards the number is stale rather than wrong, and the only fix is to capture again.
It is our weighting of things machines look for. It is useful because it is consistent and because every axis is visible, not because anyone else recognizes the number.
The tool has no opinion about permission. That question is yours, it has real answers, and the use policy in the kit states where we draw the line.
What is in the kit. The architecture in full, the four real config files with the real numbers, a runnable skeleton that walks the real stage order and genuinely implements four of the sixteen stages, 40 library stubs carrying the real function signatures and the real contract in each header, the test list with what each test defends, the two prompt templates, the operating policy, the skill and agent definitions, and a four page test site with a deliberate defect in it.
What is not in the kit, said plainly. The engine's own source is not included. This is a templatized starter kit, not a release of our implementation. Twelve of the sixteen stages ship as declared stubs with their contracts, not as working code. Saying "nothing is gated" is only honest if that boundary is stated out loud, so there it is: the knowledge is complete and ungated, the implementation of twelve stages is yours to write.
What that means in practice. A cold copy of this kit runs on the first try and produces a real scored report against the bundled test site, with zero configuration edits and zero dependencies to install. It is a working thing you extend, not a diagram.
| Path | Bytes | Get it |
|---|---|---|
ARCHITECTURE.md | 24,331 | |
LICENSE-AND-USE.md | 1,581 | |
QUICKSTART.md | 2,165 | |
README.md | 2,411 | |
TESTS.md | 1,825 | |
configs/README.md | 14,432 | |
configs/cloner.json | 1,339 | |
configs/playbook-map.json | 3,577 | |
configs/pricing.json | 711 | |
configs/strip-lists.json | 502 | |
fixture/assets/badge-400x300.png | 4,606 | binary |
fixture/assets/hero-1200x630.png | 30,276 | binary |
fixture/assets/site.css | 4,360 | |
fixture/assets/site.js | 2,089 | |
fixture/contact.html | 4,049 | |
fixture/gutter-guards.html | 5,238 | |
fixture/index.html | 5,028 | |
fixture/services.html | 4,514 | |
generate-doc.cjs | 4,705 | |
policy/README.md | 3,293 | |
policy/policy.json | 1,344 | |
prompts/copy-rewrite.md | 2,237 | |
prompts/redesign.md | 2,018 | |
scaffold/cli.js | 4,518 | |
scaffold/lib/affiliate-gate.js | 857 | |
scaffold/lib/agent-surfaces.js | 735 | |
scaffold/lib/behavior-verify.js | 1,099 | |
scaffold/lib/booking-adapter.js | 847 | |
scaffold/lib/capture-extract.js | 1,927 | |
scaffold/lib/capture-harvest.js | 1,309 | |
scaffold/lib/capture-mobile.js | 793 | |
scaffold/lib/capturer.js | 1,507 | |
scaffold/lib/config.js | 924 | |
scaffold/lib/core-pages.js | 690 | |
scaffold/lib/cost-tracker.js | 1,080 | |
scaffold/lib/crawler.js | 1,152 | |
scaffold/lib/deploy-manifest-gate.js | 1,009 | |
scaffold/lib/describe-clone.js | 769 | |
scaffold/lib/detect-archetype.js | 716 | |
scaffold/lib/doc-compare.js | 709 | |
scaffold/lib/em-dash-scan.js | 597 | |
scaffold/lib/extract-listings.js | 846 | |
scaffold/lib/extract-taxonomy.js | 1,524 | |
scaffold/lib/feature-parity.js | 680 | |
scaffold/lib/fuzzy.js | 732 | |
scaffold/lib/icon-fix.js | 702 | |
scaffold/lib/interaction-detect.js | 799 | |
scaffold/lib/interaction-pass.js | 1,139 | |
scaffold/lib/job-runner.js | 1,232 | |
scaffold/lib/llm.js | 958 | |
scaffold/lib/optimization-links.js | 884 | |
scaffold/lib/optimization-media.js | 715 | |
scaffold/lib/optimization-passes.js | 1,873 | |
scaffold/lib/packager.js | 1,474 | |
scaffold/lib/pattern-store.js | 937 | |
scaffold/lib/readiness-score.js | 1,247 | |
scaffold/lib/redesign-level3.js | 1,201 | |
scaffold/lib/redesign.js | 1,424 | |
scaffold/lib/schema-directory.js | 724 | |
scaffold/lib/speed-proof.js | 860 | |
scaffold/lib/transformer.js | 808 | |
scaffold/lib/utils.js | 1,935 | |
scaffold/lib/verifier.js | 1,406 | |
scaffold/lib/widget-registry.js | 838 | |
scaffold/package.json | 611 | |
scaffold/serve.cjs | 2,353 | |
scaffold/stages.js | 7,636 | |
skills-and-agents/README.md | 2,046 | |
skills-and-agents/clone-operator.agent.md | 1,785 | |
skills-and-agents/clone-system.skill.md | 1,585 |
The first two are the whole quickstart. If step two does not produce a scored report on your machine, stop there and fix that before going further, because everything after it assumes a working loop.
unzip clone-system-kit.zip && cd clone-system-kit
Everything below runs from the kit root.
cd scaffold && npm run demo
Serves the bundled test site on a free port, walks all sixteen stages, and writes a scored report. No network, no dependencies.
node generate-doc.cjs
Renders the kit's own markdown into a single self contained HTML file.
node cli.js clone https://example.com --out=job1
Only against a site you are authorized to rebuild. Read the use policy first.
npm install && npx playwright install chromium
Needed once you replace the scaffold capture with a real browser capture.
node --test tests/*.test.js
The full build ships 24 test files. The kit lists every one and what it defends.
Node >=20 is the only requirement for the demo. A full build additionally needs 11 packages, and the kit lists them by name with what each one is for.
The tool has no opinion about permission and this page will not pretend otherwise. It is built for rebuilding sites you are authorized to rebuild: your own, a client's with their agreement, an archive of something you own, or a local test fixture. Pointing it at a site you have no relationship with is not a technical question and the answer does not come from us.
Three things are deliberately absent from the kit and will stay absent. There is no evasion of any kind: no rotating identities, no anti detection behaviour, no working around a block. The crawl defaults are polite and they are in the config where you can see them, including a request delay that is a courtesy budget rather than a performance setting. And the user agent string is a template that asks you to identify yourself and point at a page explaining what you are doing.
If a site tells you not to, the correct behaviour is to stop. A tool that makes that easy to ignore is a different tool than this one.
Everything in this directory is published for you to take, change and build on, for any purpose, commercial included. No attribution required. No warranty of any kind. A templatized starter kit. It is the architecture, the thresholds, the stage contracts, the policy shape and a runnable scaffold. It is enough to build your own system and it is not a copy of ours. It is not the production engine source. The scaffold's module bodies are stubs with real signatures and real contracts and a TODO where the implementation goes. That boundary is stated here, on the page, and in the README, because "nothing is gated" is only honest when the edge is named out loud. This tooling exists to rebuild sites you are authorized to rebuild: your own, or a client's with their permission. That is the whole intended use. The crawl defaults that ship in `configs/cloner.json` are polite on purpose: a request delay, a per-host delay, a page ceiling and a depth ceiling. Leave them alone unless you have a reason, and if you raise them, raise them on a site you own. There are no evasion features here and none will be added. No proxy rotation, no fingerprint spoofing, no robots.txt bypass. If a site does not want to be crawled, that is an answer. Set `survey.userAgent` to a string that identifies YOU and points at a page explaining what you are doing. The template ships with a placeholder for exactly that reason. Running someone else's user agent is not anonymity, it is impersonation.
Everything above is a claim. This is where the claims get checked. The table is written by a verifier script, not by hand, and when the verifier has not run the table says so rather than showing green.
| Gate | What it checks | Result | Detail |
|---|---|---|---|
G1 | Runs from a local file with the network blocked | PASS | Standards mode, 70 file controls live, archive and twin enabled, live self check PASSED, 0 script errors, 0 network requests beyond the font sheet (1 blocked, page falls back to its declared faces); demo controls both correct (self 100.00, different pages 74.50 against a 99 threshold) |
G2 | Every teaching claim readable with JavaScript off | PASS | 72,857 characters of teaching content readable with scripting off, 0 elements left invisible |
G3 | The browser assembled archive is a valid ZIP | PASS | 195.4 KB archive assembled in the browser, python zipfile reports no bad entry, 70 members |
G4 | Unpacked archive is byte identical to the kit | PASS | all 70 files byte identical to the kit on disk, no extras |
G5 | Per file download and copy work as a fallback | PASS | 13 of 70 per file downloads sampled byte identical (both binaries included), copy confirms |
G6 | No client data anywhere on the page | PASS | Zero hits across 440,549 characters of text; 2 embedded images are byte identical to the downscale of the synthetic fixture captures; own published host clone-system.swarmsystem.ai masked as an exact literal, every other swarmsystem host still banned. Scope: 485 patterns: 27 client folders, 8 registry domains, 40 job ids, 189 secret values, plus the standing internal list |
G7 | Brand law: dashes, rules, tokens, one field, footer | PASS | 0 dashes, 0 rules, 13 token colours only, exactly 1 ambient field, centered footer, no horizontal scroll at 390 or 1440px, rail orb inside the viewport and descending, nothing lost under reduced motion |
G8 | Every stage, module, test and threshold present | PASS | 16 stages, 40 modules, 24 tests, 106 thresholds, 0 blanks |
G9 | Agent surfaces parse and the markdown twin downloads | PASS | JSON-LD parses as TechArticle, llms.txt block 1251 characters and byte identical to the file served, agents.json parses at version 1.0 and matches the file served, robots.txt allows all, deploy dir holds exactly 6 named files, markdown twin downloads at 45.3 KB |
G10 | Keyboard operation, focus, and contrast | PASS | 159 controls, all keyboard reachable, first tab lands on A:Get the kit, focus-visible styled; 111 text nodes under the contrast floor (lowest 3.46:1, the inherited --dim token WARN recorded in brand canon) |
G11 | The receipt matches real source state | PASS | manifest read 2026-08-06T16:33:44.347Z; 40 modules and 24 tests still on disk, all 16 stages still named in the entry point |
6 strings were removed from the extracted source before it reached this page, in 4 classes. The manifest records where each removal happened and what class it was, and deliberately does not record the string itself, because a redaction log that prints what it removed has not redacted anything.