building an event-driven market pipeline with vector search
· 3 min read · case study, data engineering, vector search, nlp
case study of Live Semantic Market · python · fastapi · react
Live Semantic Market turns a stream of financial news into one entry per story, however many outlets publish it.
- Role: Solo project: ingestion, entity extraction, embeddings and clustering, the API and the front end.
- Stack: Python, FastAPI, React, transformer models for zero-shot extraction, dense embeddings, a rolling vector index, asynchronous message queues.
- Result: Reworded copies of the same headline collapse into one event with every source attached. Not benchmarked yet.
- When: 2026.
1. The problem #
When a company reports earnings, dozens of feeds publish nearly the same headline within seconds. Keyword alerts fire on every copy, and exact-string matching misses the rewordings:
"Federal Reserve increases benchmark rate by 25 basis points amidst inflation concerns."
"Fed hikes interest rates 0.25% as price pressures persist."
Same story, but not one word in common:
A federal reserve INCREASES benchmark RATE by 25 BASIS POINTS ...
B FED HIKES interest RATES 0.25% as PRICE PRESSURES persist
exact words shared: 0 of 12 (A), 0 of 9 (B)
same meaning: fed ≈ federal reserve · hikes ≈ increases
0.25% ≈ 25 basis points · price pressures ≈ inflation
Edit distance and BM25 both give them a low match score, even though they're the same story.
2. Approach #
Three stages, connected by queues:
news feeds
│
v
1. ingest + normalize strip markup · timestamps · authors
│
═══ queue ═══
│
v
2. extract (zero-shot) tickers · amounts · sentiment · event type
│
═══ queue ═══ backlog waits here; nothing is dropped
│
v
3. embed + cluster compare with recent events, same ticker
│
v
canonical events ──> FastAPI ──> React
Each new article is either merged into an existing event or starts a new one:
incoming article ──> embedding u
│
v
candidates: same ticker, inside the 6–12 h window
│
├── cos(u, centroid) ≥ threshold ──> attach as a source;
│ update sentiment,
│ confidence, entities
│
└── below threshold ───────────────> new event
similarity = np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))
if similarity >= SIMILARITY_THRESHOLD:
merge_into_event_cluster(incoming_article, cluster_id)
Three decisions mattered more than the model:
Only compare against recent news. A rate hike today and one six months ago are different events, even with identical wording. The index only clusters within a sliding 6-to-12-hour window, so old stories age out:
time ──────────────────────────────────────────────> now [ aged out: never matched ] [ live window: 6–12 h ] o o o o o o oo o o ← newNarrow the search by ticker. A new article is only compared with clusters that share an extracted ticker, which keeps each nearest-neighbor search small as the index grows:
new article mentions AAPL (example) │ ├──> AAPL events searched ├──> MSFT events skipped └──> TSLA events skippedPut a queue between extraction and indexing. When embedding falls behind, articles wait in the queue instead of being dropped.
When an article matches a cluster, it's added as another source, and the event's sentiment, confidence and key entities update in place.
3. Results #
It works end to end: feeds come in, and reworded copies of a story, like the two Fed headlines, become one event that lists every source.
I haven't measured it. I don't have numbers for throughput, latency, how many duplicates it removes, or how often it merges two different stories. I'd rather say that than guess.
4. Next steps #
- Build a labeled set. Tag a few hundred articles by event, then measure duplicate-detection precision and recall and the end-to-end latency.
- Tune
SIMILARITY_THRESHOLDand the window against that set, instead of picking them by eye. - Quantize the vectors to shrink the index.
- Push alerts through webhooks when an event appears or changes.
What I learned #
The hard parts were in the data flow: deciding what counts as "the same story", and keeping a slow stage from losing data. And a pipeline that clearly works on examples still needs an evaluation set before I can say how well it works.