← All issues

Scraper health check

doneREH-50assignee · Scraper
Created
2026-08-23 02:00:06
Completed
2026-08-23 02:03:29
Comments
1

Description

Daily: Scraper probes all canonical sources (2 methods before declaring dead), posts health table to pipeline tracking issue, escalates critical source loss.

Comments (1)

doneagentScraper2026-08-23 02:03Zrun bb62ef6e

REH-50 Daily Scraper Health Check

Source Status:

  • Up: RSS/Atom, V2EX, YouTube, Bilibili, Web (Jina Reader) — 5 sources
  • Degraded: GitHub (warn, no explicit auth), LinkedIn (MCP not configured), Xueqiu (API 400, needs cookie) — 3 sources
  • Down: Reddit (403 all methods), Twitter/X (no backend), Facebook/Instagram/Xiaohongshu (OFF, no backends) — 5 sources

Probe Results:

  • Hacker News (Algolia): healthy, 20 items
  • Google Trends RSS: healthy XML but off-topic for founder/AI audience this weekend
  • Reddit JSON: 403 across old.json, top.json, .rss methods
  • V2EX: OK
  • Exa/mcporter: returns results for "small team AI tools"
  • Bilibili: search API failed
  • Xueqiu: API 400 (needs login cookie)

Escalation:

  • Reddit 403 investigation needed (network egress review)
  • Twitter/X, Facebook, Instagram, Xiaohongshu: await founder cookie config
  • LinkedIn MCP configuration needed
  • Xueqiu: login cookie configuration needed

Next steps:

  • Queue Reddit egress investigation (REH-25 backlog)
  • Await founder credential setup for social channels
  • Continue daily probing of OK/Tier-1 sources

Workspace files (37)

Every artifact agents produced in the rehanced content workspace. Click any file to read it. Newest first.

PathKindSizeModified
feedback/topic_brief__small-teams-ship-faster-2026-08-22.txttext148 B2026-08-29 18:49
research/archive/2026-08-27/digest-2026-08-27.mdmd6.5 KB2026-08-27 02:34
research/archive/2026-08-27/health-check.mdmd1.3 KB2026-08-27 02:11
research/digest-2026-08-25.mdmd6.4 KB2026-08-25 02:32
drafts/spacex-achievements-post/v1.mdmd3.7 KB2026-08-24 15:29
leads/2026-W36/raw.jsonjson56.3 KB2026-08-24 03:36
handoffs/qualified-leads-reh-53.jsonjson8.5 KB2026-08-24 03:32
research/sources.mdmd3.9 KB2026-08-24 02:32
research/digest-2026-08-24.mdmd7.7 KB2026-08-24 02:31
research/digest-2026-08-24.jsonjson7.8 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/item-49409092.jsonjson6.0 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/item-49402232.jsonjson86.4 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/item-49404380.jsonjson20.6 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/item-49363710.jsonjson23.4 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/item-49331423.jsonjson84.4 KB2026-08-24 02:31
research/archive/2026-08-24/google-trends/us-daily.rssbinary20.2 KB2026-08-24 02:31
research/archive/2026-08-24/hackernews/front-page.jsonjson35.0 KB2026-08-24 02:31
research/digest-2026-08-22.jsonjson4.9 KB2026-08-22 19:36
research/digest-2026-08-22.mdmd6.2 KB2026-08-22 19:36
research/archive/2026-08-22/google-trends/us-daily.rssbinary20.4 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/elevenlabs.jsonjson28.4 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/rust-glancer.jsonjson52.6 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/munder-difflin.jsonjson47.6 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/mcp-roadmap.jsonjson49.1 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/claude-code-effort.jsonjson38.9 KB2026-08-22 19:36
research/archive/2026-08-22/hackernews/front-page.jsonjson39.4 KB2026-08-22 19:36
handoffs/qualified-leads-reh-47.jsonjson7.2 KB2026-08-22 19:27
leads/2026-W35/e2e-qualified.jsonjson7.2 KB2026-08-22 19:27
handoffs/qa-verdict-REH-48.jsonjson4.2 KB2026-08-22 19:26
handoffs/copy-draft-e2e-bad.jsonjson670 B2026-08-22 19:26
leads/2026-W35/e2e-raw.jsonjson4.5 KB2026-08-22 19:20
assets/e2e-2026-08-22/cover.jpgimage45.6 KB2026-08-22 19:20
handoffs/qa-verdict-REH-42.jsonjson1.9 KB2026-08-22 17:54
handoffs/STATUS.mdmd3.4 KB2026-08-22 17:51
handoffs/copy-draft-smt-b7th-engineer.jsonjson3.0 KB2026-08-22 17:49
handoffs/angle-backlog-small-teams-ship-2026-08-22.jsonjson5.7 KB2026-08-22 17:48
handoffs/topic-brief-small-teams-ship-faster-2026-08-22.jsonjson6.0 KB2026-08-22 17:42

Browse all files (full tree) →