Builder Jobs — Scraper

Hourly pipeline that scrapes engineering jobs from company career pages, classifies each role with Claude, and publishes a curated index to zachproffitt/builder-jobs.

_{Last updated August 30, 2026 at 06:38 UTC}

Pipeline

fetch_jobs.py              fetch current listings from all companies
fetch_job_descriptions.py  fetch full description text for new jobs
classify_companies.py      generate company summaries via Claude
classify_jobs.py           classify roles, summarize, extract skills and comp
render_job_indexes.py      regenerate README.md, REMOTE.md, COMPANIES.md in builder-jobs

Actions

Workflow	Schedule	What it does
Jobs	Hourly	Fetch listings → classify → publish index to builder-jobs
Companies	Sundays	Discover new companies from YC, VC portfolios, and industry curation → detect ATS

Both workflows write a step summary visible in the Actions dashboard. Logs are committed with each run (data/jobs.log, data/companies.log).

Runs hourly via GitHub Actions. Commits to both repos automatically.

Monitoring

Pipeline metrics are pushed to Grafana Cloud via Prometheus. Dashboard →

Scope

The board covers remote and Colorado roles. That scope is enforced in pipeline/job_filters.py and applied before any spend: a job whose title matches the skip list, or whose location is neither remote nor Colorado, never gets a description fetch or an LLM call. Locations with no usable signal ("United States", "Multiple Locations", blank) still go to Claude, since those often hide a remote role whose remote-ness is only stated in the description.

Pre-filter skips are cached with a skip_reason field. To widen the scope, edit the regexes in job_filters.py, then reopen the affected jobs:

PYTHONPATH=. python pipeline/classify_jobs.py --purge-skips location

Classification

Each job that survives the pre-filters is sent to Claude with a structured prompt that extracts:

• BUILDER — does this role primarily write code?
• SUMMARY — 1–2 sentence description
• SKILLS — up to 8 specific technologies, languages, or tools
• LEVEL — intern / junior / mid / senior / staff / principal / manager
• COMP — base salary range with original currency symbol
• HYBRID / CONTRACT — work arrangement flags
• REGION — us / canada / international / unclear

Non-engineering, contract, and international (outside US/Canada) roles are filtered out, as are any that Claude resolves to a location outside the remote + Colorado scope.

Supported ATS

ATS	Companies	Scraper
Ashby	595	scrapers/ats_ashby.py
Greenhouse	579	scrapers/ats_greenhouse.py
Lever	194	scrapers/ats_lever.py
Workday	181	scrapers/ats_workday.py
BambooHR	76	scrapers/ats_bamboo.py
Breezy	51	scrapers/ats_breezy.py
Workable	79	scrapers/ats_workable.py
SmartRecruiters	13	scrapers/ats_smartrecruiters.py
Eightfold	8	scrapers/ats_eightfold.py
Total	1776

Company sources

~5,700 companies in data/companies.json, sourced from:

• Y Combinator — all active batches via Algolia (pipeline/fetch_yc_companies.py)
• VC portfolios — Founders Fund, Khosla Ventures, Greylock, Sequoia (pipeline/fetch_vc_companies.py)
• Industry curation — Claude-enumerated top companies across 20+ sectors (pipeline/fetch_industry_companies.py)

ATS detection (pipeline/fetch_companies.py) resolves any entry with "status": "new" and updates it in place.

To add a company manually: add a stub to data/companies.json and trigger the Companies workflow (or run fetch_companies.py locally):

{"name": "Acme Corp", "website": "https://acme.com", "status": "new"}

To remove a company: set "status": "inactive" in data/companies.json. It will stop being fetched immediately and won't be re-added by discovery.

More ATS scrapers and company sources are actively being added.

Rolling window

Jobs older than 14 days are dropped and their .md files deleted. seen_jobs.json is a permanent ID registry — prevents re-surfacing long-running postings that age out and reappear.

New companies are archived on first fetch so their existing backlog doesn't flood the board. Only jobs that appear on subsequent runs are treated as new.

Setup

Requires Python 3.11+. Clone both repos as siblings:

git clone https://github.com/zachproffitt/builder-jobs-scraper
git clone https://github.com/zachproffitt/builder-jobs

pip install -r requirements.txt
export ANTHROPIC_API_KEY=...

Run the full pipeline:

PYTHONPATH=. python pipeline/fetch_jobs.py
PYTHONPATH=. python pipeline/fetch_job_descriptions.py
PYTHONPATH=. python pipeline/classify_companies.py
PYTHONPATH=. python pipeline/classify_jobs.py
PYTHONPATH=. python pipeline/render_job_indexes.py ../jobs

Data files

File	Description
data/companies.json	All companies: name, website, ATS, slug, status
data/jobs_seen.json	Permanent ID registry: {job_id: first_seen_timestamp}
data/companies_seen.json	First-fetch registry per company
data/jobs_raw.json	Rolling 14-day window of listings with descriptions (not committed)
data/jobs_classified.json	Claude inference results per job ID
data/companies_classified.json	Company summaries used in rendered output
data/jobs.log	Log for the Jobs workflow (committed each run)
data/companies.log	Log for the Companies workflow (committed each run)