AI-Powered News Aggregator & Intelligent Trend Clusterer

Overview

A fully automated, multi-channel intelligence pipeline that ingests news streams, applies semantic scoring, and clusters articles into meaningful trend groups — all without relying on cloud-hosted LLM APIs. Every inference step runs locally via Ollama, ensuring full data sovereignty and sub-second latency for scoring tasks.

Technical Architecture

The system is structured around an event-driven backend loop:

1. Ingestion layer — Parallel RSS/API pollers implemented in Python with async I/O, feeding a shared PostgreSQL queue.
2. LLM Orchestration — A structured prompt-chain dispatched to a quantized Llama 3 model hosted on Ollama. Each article is scored across relevance, sentiment, novelty, and market-impact dimensions within a single inference call.
3. Clustering engine — Embeddings generated locally are grouped via DBSCAN with real-time centroid drift tracking to surface emerging trends rather than static categories.
4. Dashboard API — Flask REST endpoints expose scored and clustered data to a lightweight frontend with live WebSocket push updates.

Key Engineering Decisions

• Local LLM over cloud APIs — eliminates per-token cost, API rate limits, and PII data-egress risk.
• Prompt templating system — a reusable prompt DSL makes model instructions version-controlled and A/B-testable.
• Containerised deployment — the entire stack (PostgreSQL, Ollama, Flask API) runs in Docker Compose with health checks and automatic restart policies.

Stack

Python · Flask · PostgreSQL · SQLAlchemy · Ollama (Llama 3) · Prompt Engineering · DBSCAN Clustering · Docker Compose · WebSockets

Outcome

The pipeline processes hundreds of articles per hour with consistent semantic scoring quality, surfacing trend clusters 30–45 minutes ahead of traditional keyword-matching approaches used in comparable systems.