---
name: custom-news-site
description: Full AI-powered news digest pipeline — fetch RSS, AI-summarize via NVIDIA NIM, translate titles, generate HTML newspaper with images from sources, deploy to GitHub Pages.
---

# AI-Powered News Digest (valentinnews.sailor.bar)

Builds a daily HTML newspaper at https://valentinnews.sailor.bar with AI-generated Russian summaries, translated foreign headlines, real images from sources, category curation, likes, and currency rates.

## Architecture

Three scripts in `/opt/data/scripts/`, called by a single pipeline entry point:

1. `fetch_all_news.py` — RSS/Atom fetcher
2. `ai_summarize.py` — AI summarization + title translation via NVIDIA NIM
3. `generate_digest_html.py` — HTML newspaper generator
4. `deploy_news_digest.py` — pipeline orchestrator + git deploy

## Pipeline Flow

```
deploy_news_digest.py
  ├── fetch_all_news.py       → news_digest.json
  ├── ai_summarize.py         → summaries.json
  ├── generate_digest_html.py → index.html
  └── git commit + push       → GitHub Pages
```

## Step 1: Fetching News (`fetch_all_news.py`)

- Sources: RIA Novosti, RT, Vzglyad, Lenta.ru, Kommersant, Interfax, Moscow 24, People's Daily (China, 3 feeds), CGTN, Xinhua, ONLINER, BBC Russian, Euronews, arXiv (cs.AI, cs.RO), TechCrunch, NASA, and **9 Habr hubs**: AI, Программирование, Безопасность, DevOps, GameDev, Научпоп, Data Engineering, Nginx, Карьера
- De-duplication by link across all sources
- **Fair per-source distribution** via `distribute_by_source()` — prevents high-volume sources (arXiv: 425 articles) from drowning out smaller sources. Each source gets a base quota (`category_max // num_sources`) then remaining slots are filled round-robin.
- `category_visible` limits how many articles per category appear in curated section
- `category_fetch_limit` limits total per-category before AI summarization
- Category-specific color mapping (Кремль=blue, Китай=gold, etc.)
- Image extraction priority: `<enclosure>` → `media:content`/`media:thumbnail` → `<img>` from HTML description → fallback gradient
- Output: `news_digest.json` with all articles + curated counts + market data
- Language detection: Chinese sources → `needs_translation: True` (for non-Russian sources)
- **Market data also fetched**: currency rates (USD, EUR, CNY, GBP from CBR API), Brent oil price (Yahoo Finance), Google stock GOOGL (Yahoo Finance)

## Step 2: AI Summarization (`ai_summarize.py`)

- Uses **NVIDIA NIM API** (`meta/llama-3.3-70b-instruct`) as primary — **free tier**
- Fallback: OpenRouter (Gemini Flash) if NVIDIA fails
- Processes articles in **batches of 10** to avoid `max_tokens` truncation
- Temperature: 0.1 (deterministic), max_tokens: 8192 per batch
- Prompt: generates `summary` (1-2 sentences in Russian) + `title_ru` (translated title, empty if already Russian)
- Retry logic: 5 attempts with exponential backoff for 429/5xx errors
- Output: `summaries.json` keyed by 8-char MD5 hash of article link
- API key loaded from `/opt/data/.env` (`NVIDIA_API_KEY`) or `/opt/data/config.yaml` (OpenRouter)

## Step 3: HTML Generation (`generate_digest_html.py`)

Options:
- `--summaries` — path to summaries.json (optional, auto-detected from env)
- `--output` — output HTML path (default: index.html)

Features:
- **Dark theme** with CSS custom properties
- **Category sections** with colored headers and per-category visibility
- **Article cards**: image (or gradient placeholder with 📰), translated title as main link (if `title_ru` available) + original title as `.article-title-original` subtitle in muted color, source badge, AI summary with `AI` badge
- **Likes**: localStorage-based toggle per article (🤍 → ❤️)
- **Currency rates**: from CBR API, shown in header
- **Feed section** at bottom for non-curated articles
- Mouse-following glow effect on title

## Step 4: Deploy (`deploy_news_digest.py`)

- Runs the full pipeline sequentially
- Extracts `SUMMARIES_PATH` from ai_summarize.py output
- Verifies HTML is not empty (< 1000 bytes = fail)
- Git commits `index.html` with date message, pushes to `origin master`
- Prints URL on success

## Cron Job

- Schedule: `0 6 * * *` (daily at 06:00)
- Calls `deploy_news_digest.py` directly
- Creates fresh `news_digest.json` → `summaries.json` → `index.html` each run

## API Keys

| Service | Location | Format |
|---------|----------|--------|
| NVIDIA NIM | `/opt/data/.env` | `NVIDIA_API_KEY=nvapi-...` |
| OpenRouter | `/opt/data/config.yaml` | `api_key: sk-or-...` |
| Telegram | `/proc/1/environ` + `.env` | `TELEGRAM_BOT_TOKEN=...` |

## Pitfalls

- **max_tokens too low** → LLM response gets truncated, missing articles. Use 8192+ for batches of 10.
- **NVIDIA API slow** → first request can take 30-60s. Set HTTP timeout to 180s, process timeout to 300s.
- **Last batch timeout** → if the final batch (1-3 small) times out under 300s total, run it independently with a simpler prompt, then merge results back into `summaries.json`.
- **22 → 21 summaries** is normal — same article link in multiple categories produces one hash. `summaries.json` can have fewer entries than curated count.
- **Duplicate articles across categories** → same link in multiple categories produces same hash. Validate by comparing curated count vs unique hashes.
- **HTML description parsing** → some RSS feeds encode `<img>` as escaped HTML in `<description>`. Must unescape and parse with regex or html.parser.
- **Empty summaries** → 5/22 is a sign of truncation. Use batch processing to fix.
- **GitHub Pages delay** → after push, site takes 1-3 minutes to rebuild. Use `curl` to verify after deploy.
- **rate limiting (429)** → NVIDIA NIM may rate-limit. Retry with 5-40s backoff.
- **High-volume sources drown out small ones** → `distribute_by_source()` is essential when a source (arXiv, RSSHub) produces 400+ articles vs a source (Habr) producing 20-30. Without fair distribution, small sources never appear in limited categories. Implement as: `quota = category_max // len(sources_in_category)`, then fill remaining slots round-robin.
- **Habr RSS URLs** → Format: `https://habr.com/ru/rss/hubs/{hub_name}/all/` (e.g., `ai`, `programming`, `gamedev`, `science`, `data_engineering`, `nginx`, `career`, `information_security`, `devops`). They work reliably.
- **Market data needs fallback** → CBR API sometimes returns 0 for some currencies (e.g., CNY). Yahoo Finance queries can fail. Always provide fallback values or graceful degradation.

## Verification

After deploy, check:
1. `curl -sS https://valentinnews.sailor.bar | head -c 100` — should return HTML starting with `<!DOCTYPE html>`
2. `summaries.json` has entries for all curated articles
3. Articles with `needs_translation: True` have `title_ru` set (not empty)
4. Likes persist across page reload (localStorage)
