Scraper Reliability Engineering for Bright Data Scraper Studio
Your scraper didn’t stop.It started lying.
When a scraper breaks loudly, you find out. When a numeric field quietly starts returning 0 instead of its real value, it fills on every row, passes every null and schema check you own, and poisons everything downstream. Molt baselines what healthy looked like, catches the drift against it, writes the heal prompt from the measured evidence — and refuses to ship the fix until a human has seen the diff.
- drift verdicts per field
- 7
- drift verdicts per field
- states in the incident machine
- 12
- states in the incident machine
- fixes shipped without a human
- 0
- fixes shipped without a human
The failure nobody gates
[01/06] · silent breakage
A field that returns zero is still a field.Every check you own says it is fine.
On the chaos target Molt watches, comment_count went from a typical value of 60.5 to a typical value of 0. Its fill rate never moved: it stayed at 100%, because zero is a value. Nothing in an ordinary pipeline is looking at the number itself, so nothing objected — and this is the bug the project found in its own UI first, which is why the grid above refuses to print a percentage for a zeroed field.
The missing memory
[02/06] · baseline
Your monitoring knows what just happened.It does not know what normal was.
You cannot detect drift without a memory to drift from. Every run Molt makes becomes a snapshot — per field, its fill rate, its value shape and its typical magnitude. One snapshot is pinned as the baseline, and every later run is classified against it field by field. That comparison is a pure function: the same two snapshots always produce the same verdict, which is why the detection rules are pinned by fixtures instead of tested against a live website.
comment_count
typical value
comment_count
fill rate
title
fill rate
Two rows describe the same field in the same run. Only the first one is a fault, and only a baseline makes it visible.
How it works
[03/06] · the loop
Six steps, one collector ID.Detect, diagnose, heal, verify.
- 01
Run
Runs the collector through the Bright Data CLI, projects the rows, and snapshots every field.
molt check chaos - 02
Detect
Classifies each field against the baseline: healthy, degraded, distorted, flatlined, collapsed, vanished, appeared. A fault opens an incident.
- 03
Diagnose
Turns measured drift into a heal prompt — naming the dead fields, their before-and-after numbers, and crucially the fields still working, so the healer knows what not to touch.
- 04
Heal
Heals the same collector in place. Same Collector ID before and after — not a regenerate.
bdata scraper heal <id> "<prompt>" - 05
Review
Stops. The incident sits at awaiting_approval until a human reads the baseline-versus-preview diff.
molt review - 06
Verify
Commits the fix, then runs the collector again and proves recovery is the negation of the fault, at the same threshold. Only then does the incident close.
molt approve
The human gate
[04/06] · awaiting_approval
A fix that returns data is not a fix.Molt stops and shows you the diff.
An AI heal confirms one thing: the field came back non-null. It cannot tell you the value is right. So the incident state machine has a state it will not leave on its own — awaiting_approval — and the only ways out are a person approving or rejecting it.
Approving in the browser spawns the identical bdata command molt approve would. The button is a window onto the terminal, not a second implementation of it.
| field | baseline | broken | preview | |
|---|---|---|---|---|
| comment_count | 60.5 | 0 | 18.5 | |
| download_count | 20,251.5 | 0 | 6,192 | |
| title | 100% | 100% | 100% | · |
| tags | 100% | 100% | 100% | · |
Built on Bright Data Scraper Studio
[05/06] · four commands
Molt does not replace Scraper Studio.It heals the collector you already have.
Generate the collector
Scraper Studio builds it from plain language. Description caps at 500 characters.
bdata scraper create <url> "<description>"Run it and take structured output
Rows in, snapshot out. This is the only step that happens on every check.
bdata scraper run <id> <url>Heal the same collector
In place, reusing the collector. Prompt caps at 1000 characters.
bdata scraper heal <id> "<prompt>"Approve, then verify
Molt runs the collector again afterwards and checks the fault is actually gone.
bdata scraper approve <id>Honest limits
[06/06] · stated, not hidden
A reliability tool that hides its own limitsis not a reliability tool.
Credit figures are estimates
Bright Data publishes no per-operation price list, so Molt weights commands by kind and says so everywhere a number appears. It is a relative signal, not a bill.
Healing is slow and rate-limited
Heal and create are AI-Flow jobs: five to twenty-five minutes each, behind a concurrent-job cap that returns 429. Molt serialises them through a single slot rather than pretending otherwise.
Targets must stay small
The intent analyser fails outright on large documents — a 1.63 MB page killed two collector builds and left two orphans that cannot be deleted programmatically. Molt measures Content-Length before it will onboard a target, and refuses above roughly 200 KB.
Bright Data cannot reach your laptop
Collectors run in Bright Data’s cloud. A target on localhost is not a target, which is why the chaos site used for demonstrations is deployed publicly.
A small preview cannot prove a magnitude
When a heal returns two rows against a sixty-row baseline, Molt says the sample is too small to compare sizes and tells you what it can still prove: a zeroed field is no longer zero.