Job Discovery #

Raven finds jobs through parallel structured API tiers — live ATS JSON endpoints, public board feeds, and an optional local openjobdata index — then deduplicates and saves results to data/jobs.json.

Not open-web search: Google/WebSearch (search_queries in portals.yml) is documented but not executed by discover today. See How discovery works → and Scan strategies →.

Pages in this section #

How discovery works

How job discovery works #

Deep dive: architecture, data flow, and what Raven does not do.

Module: jobs/discover.mjs · CLI: raven discover

TL;DR #

raven discover does not search Google or scrape arbitrary websites. It runs up to four parallel tiers that pull structured job data from public APIs and feeds, filters client-side, deduplicates by canonical apply URL, and saves to data/jobs.json.

flowchart LR
  A[raven discover] --> B[ATS APIs]
  A --> C[Board feeds]
  A --> D[Local SQLite]
  A --> E[hiring.cafe optional]
  B --> F[merge + dedup]
  C --> F
  D --> F
  E --> F
  F --> G[data/jobs.json]

What “search” means #

Mechanism	How listings are found	Default discover?
ATS reverse scan	Public JSON API per company board	Yes
Board feeds	Aggregator APIs (RemoteOK, Remotive, …)	Yes
openjobdata index	SQL on `data/jobs.db`	Yes (needs sync)
hiring.cafe	POST search API	Opt-in
WebSearch / `search_queries`	Google-style `site:` queries	Not implemented
`scan_method: websearch`	Per-company handoff	Log only

Zero LLM, zero browser by default.

Sources & tiers

Job sources #

Raven orchestrates discovery tiers in parallel via jobs/discover.mjs. Each tier uses public APIs or feeds — not web search engines.

Full architecture: How discovery works · WebSearch status: Scan strategies

Tier 1 — Live ATS APIs (`--sources ats`) #

Reverse-scans public company directories and hits each platform’s zero-auth JSON API.

Platform	Careers URL pattern
Greenhouse	`job-boards.greenhouse.io/{slug}`
Lever	`jobs.lever.co/{slug}`
Ashby	`jobs.ashbyhq.com/{slug}`
Workday	`{tenant}.wd{N}.myworkdayjobs.com/{site}`
Rippling	`ats.rippling.com/{slug}/jobs`
Workable	`apply.workable.com/{slug}`
BambooHR	`{tenant}.bamboohr.com/careers`
SmartRecruiters	`careers.smartrecruiters.com/{slug}`
Recruitee	`{slug}.recruitee.com`
Pinpoint	`{slug}.pinpointhq.com`
Teamtailor	`{slug}.teamtailor.com`
Personio	`{slug}.jobs.personio.de`

Company slug lists come from openjobdata (after raven sync-jobs) or bundled caches in data/cache/ats-companies/.

Scan strategies

Scan strategies #

portals.example.yml documents a 4-level strategy for job discovery beyond default tiers. This page maps each level to what Raven runs automatically today.

Four levels (from portals.example.yml) #

Level 0. local_parser  → custom script per company
Level 1. Playwright    → browser scrape of careers_url
Level 2. HTTP JSON     → public ATS/board APIs (providers)
Level 3. WebSearch     → site: filtered search queries

Default raven discover uses HTTP JSON at scale (ATS reverse scan + board feeds) — not levels 0–1 per company, and not level 3.

Filters

Discovery filters #

Filters apply during raven discover, raven scan-ats, raven scan-boards, and raven query.

CLI flags #

Flag	Description	Example
`--q "keywords"`	Title must match (comma-separated)	`--q "backend, engineer"`
`--not "words"`	Title must not contain	`--not "nurse, sales"`
`--loc Remote,EU`	Location allow-list	`--loc Remote,India`
`--noloc "On-site"`	Location block-list	`--noloc "San Francisco"`
`--since N`	Posted within last N days	`--since 7`
`--ats greenhouse,lever`	Limit ATS platforms	`--ats ashby,workday`
`--max N`	Cap total results	`--max 100`
`--limit N`	Per-provider limit	`--limit 50`

Config vs CLI #

When CLI flags are omitted, Raven merges filters from config/portals.yml:

openjobdata

openjobdata index #

Raven syncs the openjobdata dataset into a local SQLite database for fast offline search.

Sync #

raven sync-jobs
raven sync-jobs --days 3

What happens #

Downloads companies registry from HuggingFace bucket Invicto69/Jobs-Dataset-bucket
Fetches recent daily parquet deltas
Upserts into data/jobs.db
Exports ATS company slug lists to data/cache/ats-companies/

Authentication #

If sync returns HTTP 401:

# .env
HF_TOKEN=hf_xxxxxxxx

Create a read token at huggingface.co/settings/tokens.

Query local index #

raven query --q "software engineer" --since 7
raven query --q "ML engineer" --ats greenhouse,lever --json

Same filter flags as discover: --q, --not, --loc, --since, --ats, --max.

Discovery tiers #

Tier	Flag	Sources	Prep
ATS	`--sources ats`	Greenhouse, Lever, Ashby, Workday, …	None
Boards	`--sources boards`	RemoteOK, Remotive, Arbeitnow, Landing.jobs	None
Index	`--sources index`	Local SQLite from openjobdata	`raven sync-jobs` first

Default runs all three in parallel.

Common commands #

# Full search (uses portals.yml if no --q)
./raven discover

# Explicit filters
./raven discover --q "backend engineer" --loc Remote --since 7

# Fast — boards only
./raven discover --sources boards --q "developer" --max 50

# Query local index only
./raven sync-jobs
./raven query --q "ML engineer" --since 7

Results auto-save to data/jobs.json unless you pass --no-save.

Start with How discovery works → or Sources & tiers →