About the role
Must Have:
- Strong Python experience required.
- Experience building and operating web scraping (with Scrapy) and data engineering systems in production is a plus - preferable.
- Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing failures in code you did not write, against sources you do not control.
- Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests, and understanding network and auth flows - not just inspecting elements.
- Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections.
- Strong understanding of ETL and data engineering concepts, including extraction, transformation, validation, deduplication, incremental processing, backfills, and data-quality checks.
- Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks overnight.
- Multi-format extraction, particularly PDF, where the structure is never where you want it.
- Git and collaborative development workflows, including code review.
Good to Have:
- Browser automation for JS-rendered sources (Playwright, Selenium).
- Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow.
- Hierarchical or citation-style document modelling.
- Proxy and anti-bot handling within legal and contractual limits.
Role in brief:
A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality.
Build:
- Design and own Scrapy spiders and ETL pipelines for global regulatory sources.
- Extract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and stable identifiers.
- Build item pipelines that persist to MongoDB and object storage.
- Reverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs.
Operate:
- Run scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.
- Improve reliability with incremental and full loads, output validation, and alerts that catch silent failures.
- Use MongoDB queries to inspect, backfill and correct data safely.
Debug:
- Own triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.
- Tackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).
- Write short postmortems and turn each into a validation check, test or fix.
Improve speed:
- Build reusable base spiders and parsers so a new source takes days, not weeks.
- Own metrics such as time-to-launch, failure rate and manual intervention hours.
- Use AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results.
Collaborate:
- Work with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.
- Document spider designs, source quirks and runbooks, and help raise code review and testing standards.