Job discovery on Raven

How discovery works

Mon, 01 Jan 0001 00:00:00 +0000

How job discovery works #

Deep dive: architecture, data flow, and what Raven does not do.

Module: jobs/discover.mjs · CLI: raven discover

TL;DR #

raven discover does not search Google or scrape arbitrary websites. It runs up to four parallel tiers that pull structured job data from public APIs and feeds, filters client-side, deduplicates by canonical apply URL, and saves to data/jobs.json.

flowchart LR
 A[raven discover] --> B[ATS APIs]
 A --> C[Board feeds]
 A --> D[Local SQLite]
 A --> E[hiring.cafe optional]
 B --> F[merge + dedup]
 C --> F
 D --> F
 E --> F
 F --> G[data/jobs.json]

What “search” means #

Mechanism	How listings are found	Default discover?
ATS reverse scan	Public JSON API per company board	Yes
Board feeds	Aggregator APIs (RemoteOK, Remotive, …)	Yes
openjobdata index	SQL on `data/jobs.db`	Yes (needs sync)
hiring.cafe	POST search API	Opt-in
WebSearch / `search_queries`	Google-style `site:` queries	Not implemented
`scan_method: websearch`	Per-company handoff	Log only

Zero LLM, zero browser by default.

Sources & tiers

Mon, 01 Jan 0001 00:00:00 +0000

Job sources #

Raven orchestrates discovery tiers in parallel via jobs/discover.mjs. Each tier uses public APIs or feeds — not web search engines.

Full architecture: How discovery works · WebSearch status: Scan strategies

Tier 1 — Live ATS APIs (`--sources ats`) #

Reverse-scans public company directories and hits each platform’s zero-auth JSON API.

Platform	Careers URL pattern
Greenhouse	`job-boards.greenhouse.io/{slug}`
Lever	`jobs.lever.co/{slug}`
Ashby	`jobs.ashbyhq.com/{slug}`
Workday	`{tenant}.wd{N}.myworkdayjobs.com/{site}`
Rippling	`ats.rippling.com/{slug}/jobs`
Workable	`apply.workable.com/{slug}`
BambooHR	`{tenant}.bamboohr.com/careers`
SmartRecruiters	`careers.smartrecruiters.com/{slug}`
Recruitee	`{slug}.recruitee.com`
Pinpoint	`{slug}.pinpointhq.com`
Teamtailor	`{slug}.teamtailor.com`
Personio	`{slug}.jobs.personio.de`

Company slug lists come from openjobdata (after raven sync-jobs) or bundled caches in data/cache/ats-companies/.

Scan strategies

Mon, 01 Jan 0001 00:00:00 +0000

Scan strategies #

portals.example.yml documents a 4-level strategy for job discovery beyond default tiers. This page maps each level to what Raven runs automatically today.

Four levels (from portals.example.yml) #

Level 0. local_parser → custom script per company
Level 1. Playwright → browser scrape of careers_url
Level 2. HTTP JSON → public ATS/board APIs (providers)
Level 3. WebSearch → site: filtered search queries

Default raven discover uses HTTP JSON at scale (ATS reverse scan + board feeds) — not levels 0–1 per company, and not level 3.

Filters

Mon, 01 Jan 0001 00:00:00 +0000

Discovery filters #

Filters apply during raven discover, raven scan-ats, raven scan-boards, and raven query.

CLI flags #

Flag	Description	Example
`--q "keywords"`	Title must match (comma-separated)	`--q "backend, engineer"`
`--not "words"`	Title must not contain	`--not "nurse, sales"`
`--loc Remote,EU`	Location allow-list	`--loc Remote,India`
`--noloc "On-site"`	Location block-list	`--noloc "San Francisco"`
`--since N`	Posted within last N days	`--since 7`
`--ats greenhouse,lever`	Limit ATS platforms	`--ats ashby,workday`
`--max N`	Cap total results	`--max 100`
`--limit N`	Per-provider limit	`--limit 50`

Config vs CLI #

When CLI flags are omitted, Raven merges filters from config/portals.yml:

openjobdata

Mon, 01 Jan 0001 00:00:00 +0000

openjobdata index #

Raven syncs the openjobdata dataset into a local SQLite database for fast offline search.

Sync #

raven sync-jobs
raven sync-jobs --days 3

What happens #

Downloads companies registry from HuggingFace bucket Invicto69/Jobs-Dataset-bucket
Fetches recent daily parquet deltas
Upserts into data/jobs.db
Exports ATS company slug lists to data/cache/ats-companies/

Authentication #

If sync returns HTTP 401:

# .env
HF_TOKEN=hf_xxxxxxxx

Create a read token at huggingface.co/settings/tokens.

Query local index #

raven query --q "software engineer" --since 7
raven query --q "ML engineer" --ats greenhouse,lever --json

Same filter flags as discover: --q, --not, --loc, --since, --ats, --max.

Job discovery on Raven

How discovery works

How job discovery works #

TL;DR #

What “search” means #

Sources & tiers

Job sources #

Tier 1 — Live ATS APIs (--sources ats) #

Scan strategies

Scan strategies #

Four levels (from portals.example.yml) #

Filters

Discovery filters #

CLI flags #

Config vs CLI #

openjobdata

openjobdata index #

Sync #

What happens #

Authentication #

Query local index #

Tier 1 — Live ATS APIs (`--sources ats`) #