← Projects

StockPulse — Stock Anomaly Detection & AI Analysis Pipeline

  • Solo project · Design to operations
  • Kafka · MSA
  • Kubernetes · ArgoCD
  • TimescaleDB

Overview

When a stock suddenly spikes or plunges during market hours, the first thing an investor wants to know is why. StockPulse watches prices for 68 US and 40 Korean tickers, and when a sharp move appears it first decides whether the move belongs to one stock, a whole sector or the entire market. It then gathers related news, has an LLM explain the cause in Korean and English, and delivers it to a dashboard and Slack.

It is an event-driven design: from ingestion to alerts, seven services are connected by seven Kafka topics (two of them DLQs), with a Next.js dashboard on top. It runs on an on-prem Kubernetes cluster, deployed with ArgoCD GitOps. It is a solo project I designed, built and ran myself.

Architecture

Scroll and the diagram follows the text. Point at or select a node to see its specs.

At a glance

StockPulse takes in external data (prices, news, an LLM), processes it with services that talk over Kafka inside an on-prem Kubernetes cluster, and delivers results to the browser and Slack. Deployment runs through the GitOps flow at the bottom.

services ×8topics ×7Kafka ×3

1How prices come in

For 68 US tickers, stock-collector pulls 1-minute bars from yfinance every 60 seconds during NYSE hours and publishes them to stock.raw.us. When the market closes it sleeps until the next open.

stock.raw.usNYSE calendar

For 40 Korean tickers, kis-bridge receives live trades over the Korea Investment & Securities (KIS) WebSocket, rolls them into 1-minute bars and publishes them to stock.raw.kr. It renews tokens before they expire and reconnects with a 5→60 s backoff.

stock.raw.krKIS H0STCNT0

2Detecting and classifying moves

anomaly-detector reads both topics and keeps only bars from the last 5 minutes whose change or Z-score crosses a threshold. Korea gets a higher threshold because of its ±30% daily price limit.

% OR Z-scorelast 5 min

ETFs act as sector thermometers. Same-direction sector ETFs in three or more sectors, or three or more sectors moving the same way, means MARKET; the mover being a sector ETF itself, or the sector’s ETF or peers moving with it, means SECTOR; otherwise INDIVIDUAL. Results go out on anomaly.detected.

INDIVIDUALSECTORMARKET

3Finding the cause

news-fetcher consumes anomaly.detected and gathers up to four articles per language: English articles from NewsAPI (last three days) and Google News, searched by ticker and sector keywords, and Korean articles from Naver and Google News RSS, searched by sector keywords. It then publishes news.fetched.

anomaly.detectednews.fetched

ai-analyzer writes Korean and English analyses with the Groq LLM, shifting the prompt’s focus to company, sector or macro issues by event type. On a 429 it waits for the advised delay and retries; if failures continue and the circuit breaker opens, messages go to news.fetched.dlq.

llama-3.3-70bcircuit breaker 5×/60snews.fetched.dlq

4Reaching people

api consumes analysis.completed, stores it in TimescaleDB and pushes it straight to open dashboards over the /ws/live WebSocket. Browsers connect through the nginx Ingress.

analysis.completed/ws/live

The notifier filters the same results by sector and event type and sends them to Slack. If it still fails after three attempts, the message moves to analysis.completed.dlq.

3 attemptsanalysis.completed.dlq

Restart behavior differs by stage. Detection, news fetching and notifications use earliest to reprocess anything missed; LLM analysis and api use latest so a backlog doesn’t trigger a burst of LLM costs or re-show the same signals.

earliestlatest

5A model that retrains daily

The ml-trainer CronJobs predict next-day direction for 25 Korean tickers. Each day they score yesterday’s predictions and retrain if 30-day accuracy drops below 52% or the model is older than 7 days. Once a week they tune with Optuna and validate with walk-forward cross-validation.

LGB 0.4 · XGB 0.35 · Cat 0.25Purged CV

6From git push to deploy

A push to main triggers GitHub Actions on a self-hosted runner, which maps changed paths to services, builds only what changed and pushes stock/<svc>:<sha7> to Harbor.

path → service mapimages ×9

Actions commits the new image tags to the manifests ([skip ci] prevents loops), and ArgoCD polls the repo every 60 seconds and syncs the cluster to Git with selfHeal and prune. Images are pulled from Harbor, and secrets live in Git encrypted with Sealed Secrets.

selfHeal · pruneServerSideApply

7Keeping watch — observability

Prometheus scrapes api and anomaly-detector metrics plus Kubernetes and node metrics every 15 seconds, and one Grafana dashboard shows Kafka lag, API p95 latency, detections and ML accuracy.

anomaly_detected_totalml_prediction_accuracy
At a glance1 / 14
External dataOn-prem Kubernetes · namespace stockUsersCI/CD · GitOpsKAFKA ×3yfinanceUS 1-min bars · 60 sKIS OpenAPIKR live tradesNewsAPI · RSSEN · KO newsGroq LLMllama-3.3-70bSlackAnomaly alertsBrowserLive dashboardstock-collectorUS · market hourskis-bridgeticks → 1-min barsnews-fetcherup to 4 per languageai-analyzerKO/EN cause analysisnotifierfilters · retriesanomaly-detector% · Z · ETF classifierapiFastAPI · WebSocketTimescaleDBanomalies · analysesfrontendNext.js 14Ingressnginx · MetalLBml-trainerCronJob ×3PrometheusGrafana · 15 s scrapeSealed Secretsdecrypts secretsGitHubmain branchActionsself-hosted runnerHarborprivate registryArgoCDselfHeal · prune

Try it

US market 1-minute bars are streaming in

Threshold 3.0% or Z 2.0Detections 0

StockPulse’s real ticker list and its detection and classification rules, replayed in the browser. Prices, news and AI analysis text are examples.

Operations log

ArgoCD stuck in OutOfSync forever

Symptom
The StatefulSets stayed OutOfSync after every deploy.
Cause
Fields the cluster fills in, such as StatefulSet volumeClaimTemplates, never matched Git.
Fix
Excluded those fields with ignoreDifferences, along with CronJob status for the same reason. ServerSideApply=true had been turned on in advance so large resources would never hit the 262 KB limit of the last-applied annotation.

With selfHeal and prune enabled, ArgoCD keeps syncing as long as a difference remains. Differences caused by cluster-populated fields never go away no matter how often you sync, so the dashboard was always OutOfSync and real drift was hard to spot. I declared which fields to ignore. Five retries (backing off from 5 s up to 3 min) and PruneLast, which runs pruning last, had been set from the start.

Lesson In GitOps, “in sync with Git” has to leave out fields the cluster owns. I design what gets compared and how it gets applied together.

Three brokers vs. a small node’s memory

Symptom
Without an explicit heap, each broker sized itself from its 2 Gi memory limit, so the three could use up to 6 Gi — an OOMKill risk on 2-core, 8 GB nodes.
Cause
The stateful workloads’ default JVM settings were too large for small nodes, and startup and shutdown times weren’t accounted for.
Fix
Capped broker and ZooKeeper heaps at -Xmx256m, gave the startupProbe up to 630 s to wait for __consumer_offsets loading, and raised terminationGracePeriodSeconds to 90 s for a clean shutdown.

Kafka runs three brokers. Replication factor 3 is set only for the consumer-offsets and transaction-log topics; application topics are auto-created and follow the broker defaults. Alongside memory, I tuned startup and shutdown. A starting broker needs time to read __consumer_offsets, and while it was loading 54 partitions the liveness probe (60 s) failed first and the pod went into CrashLoopBackOff. On the way down, the default 30-second grace period wasn’t enough for Kafka’s controlled shutdown, so pods repeatedly had to be force-deleted. The startupProbe now allows enough time, and on shutdown the pod waits for the controlled shutdown to finish.

Lesson For stateful workloads, the time it takes to come up and go down is part of resource design.

Minute-bar batches outgrew Kafka’s message limit

Symptom
Sending 1-minute bars for 68 US tickers as one message hit the message size limit.
Cause
The collector packed the whole lookup window (5 days by default) into a single batch message.
Fix
Cut the lookup window from 5 days to 1 day in the Kubernetes deployment so the batch fits within the 10 MB message limit that had been set from the start.

Every 60 seconds the collector sends all tickers’ minute bars at once under the key batch. The default five-day window means close to 2,000 bars per ticker, so the batch easily passes 10 MB; the cluster config sends only one day.

Lesson “One batch = one message” runs into limits as data grows. For messages that can grow, I decide the split unit up front.

Design decisions

A different consumer offset strategy per stage

Context
Each pipeline stage wants a different starting point: some must process everything still in the topic, others must not pick up old messages.
Alternatives considered
  • All consumers on earliest
  • All consumers on latest
Why
Consumer groups resume from their committed offsets, so auto.offset.reset only decides where a group starts when it has none — on first deploy or after its offsets expire. Detection, news fetching and notifications use earliest and start from what is still in the topic; LLM analysis and the api use latest, so a backlog at that moment doesn’t trigger a burst of LLM costs or re-show old signals on the dashboard.
Trade-off
Stages on latest skip whatever piled up before they first attached, and skipped signals never reach the database. The dashboard’s “reanalyze past” action only re-runs stored signals from the last 7 days whose analysis is missing or failed (up to 30).

Classifying signals with ETFs as sector thermometers

Context
When one stock jumps, the likely cause, and the news worth looking up, depends on whether the move is company-specific or sector-wide.
Alternatives considered
  • Judge each stock alone and leave cause-finding to the LLM
Why
Each sector has ETFs as thermometers. If same-direction sector ETFs span three or more sectors, or three or more sectors move the same way, it is MARKET; if the mover is the sector ETF itself, or the sector’s ETF or peers move with it, it is SECTOR; otherwise INDIVIDUAL. The event type also shifts the LLM prompt’s focus to company, sector or macro issues.
Trade-off
Classification quality depends on ETF coverage and thresholds, which need tuning per market (higher in Korea because of the ±30% daily price limit).

External API failures: back off, break the circuit, park in a DLQ

Context
External APIs like the LLM and Slack can slow down or return 429 at any time. One message’s failure must not block the whole pipeline.
Alternatives considered
  • Retry a failed message until it succeeds
  • Drop failed messages
Why
ai-analyzer reads the wait time in a 429 response and retries after it (up to three attempts), and after five consecutive failures it stops calling for 60 seconds (circuit breaker). Messages that arrive while the circuit is open go to news.fetched.dlq with the original topic, error and timestamp. Other failures flow on to analysis.completed as an “analysis failed” note and can be re-run with the dashboard’s “reanalyze past” action. The notifier likewise tries Slack up to three times, 5 seconds apart, before sending to analysis.completed.dlq.
Trade-off
There is no tool yet to replay messages from the DLQs.

A daily-bar path that runs without Kafka

Context
If the service works only when Kafka, the collectors and the detector are all up, losing any part of the infrastructure stops everything — and even checking a feature locally needs the full stack.
Alternatives considered
  • Refuse to start without Kafka
Why
Regardless of the Kafka settings, the api runs a daily-bar pipeline (detect → news → analyze → store) itself with APScheduler every hour and right after startup, and when KAFKA_BOOTSTRAP_SERVERS is set it also starts an analysis.completed consumer. So daily-bar results keep accumulating even without Kafka, the collectors or the detector. Both paths use the same detection and analysis modules (core/), so the logic doesn’t drift.
Trade-off
The daily-bar path isn’t real time, and in production it runs every hour alongside the streaming path, calling the LLM for every daily-bar signal not yet analyzed.

GitOps with image-tag commits and ArgoCD

Context
Building nine images — eight services plus ml-trainer — and rolling them out by hand makes it hard to know what is running at which version.
Alternatives considered
  • Deploy straight from CI with kubectl apply
Why
CI builds only changed services, pushes their images to Harbor and commits the new image tags (commit SHAs) to the manifests, with [skip ci] preventing loops. ArgoCD picks up the commit and syncs the cluster to Git with selfHeal and prune, so Git history answers what is deployed.
Trade-off
Tag commits pile up in the history, and ArgoCD needed its own handling — leaving cluster-populated fields out of the comparison and turning on ServerSideApply in advance against the 262 KB annotation limit (see the operations log).

Secrets in Git, too — Sealed Secrets

Context
With every manifest in Git for GitOps, the question is where secrets go.
Alternatives considered
  • Create Secrets by hand with kubectl
  • An external secret store (e.g. Vault)
Why
A manually triggered workflow builds a .env from GitHub Secrets, encrypts six SealedSecrets with kubeseal and commits them. Only the in-cluster controller can decrypt them, so they are safe in Git. The controller itself is managed as an ArgoCD Application.
Trade-off
Losing the controller’s sealing key means recreating every SealedSecret, so backing up that key has to be part of the operating procedures.

Infrastructure & cost

git pushmainActionsself-hosted runnerHarborstock/<svc>:<sha>Tag commit[skip ci]ArgoCDselfHeal · pruneClusternamespace stock

On-prem Kubernetes

  • A single stock namespace runs 12 Deployments, 3 StatefulSets (Kafka ×3, ZooKeeper, TimescaleDB), 2 DaemonSets, 6 CronJobs and 3 Jobs.
  • Traffic enters through one MetalLB (L2) address and an nginx Ingress: /api, /ws, /auth and /metrics go to FastAPI, /grafana and /prometheus to monitoring, everything else to Next.js. WebSocket timeouts are raised to one hour.
  • Storage combines static NFS PVs (Retain) with nfs-subdir-external-provisioner, and heavy workloads (Kafka, the database, the API, ML) are preferentially scheduled on node-tier=heavy nodes.
  • Images live in a private Harbor, and a DaemonSet distributes Harbor’s self-signed certificate to each node’s containerd.

From git push to deploy

GitHub Actions on a self-hosted runner maps changed paths to services and builds only the services that changed. It pushes stock/<svc>:<sha7> to Harbor and commits the new tags to the manifests; ArgoCD polls the repo every 60 seconds, picks up that commit and syncs it to the cluster. Secrets are encrypted with Sealed Secrets before they go into Git.

Retention and cost

  • Two weekly CronJobs handle NFS usage checks (warning at 80%) and Redis AOF rewrites respectively.
  • Kafka retention is capped at 7 days or 15 GiB per partition, whichever comes first.

My contribution

StockPulse is a solo project, so I did all of it: planning, development, infrastructure and operations. Here it is in the order I built it:

Standing up the pipeline on the cluster

  • On an on-prem Kubernetes cluster I set up ArgoCD (GitOps) and CI on a self-hosted runner, then deployed the FastAPI backend, the frontend and the Kafka pipeline.
  • I encrypted secrets with Sealed Secrets so they could live in Git, and configured NFS PVs, MetalLB and Ingress (WebSocket), and Harbor certificate distribution.
  • Kafka, ZooKeeper and TimescaleDB kept crashing over resource and probe problems; memory limits, TCP probes and turning off service links fixed them one by one (incident, incident).

Features and observability

  • I built Korean 1-minute bar ingestion (KIS) and daily-bar lookup, login, a watchlist, real-time portfolio P&L and a responsive layout.
  • I added the LLM analysis report pipeline with a retry when an analysis fails. Detection runs only during market hours, and I cut down requests on weekends and holidays.
  • Prometheus and Grafana (Helm), kube-state-metrics and node-exporter made the cluster and anomaly-detector metrics visible.

Stabilizing and predicting

  • I fixed ArgoCD permanently showing OutOfSync because of StatefulSet volumeClaimTemplates (incident).
  • I switched the LLM from Gemini to Groq (llama-3.3-70b) for its larger free token allowance.
  • I trained a price prediction model with sector ETF features, relative strength and purged walk-forward cross-validation, showed it on the dashboard, and marked anomalies on the chart with what happened after detection (trend held or faded).

Limits & retrospective

Known limitations, and what I would change next time.

  • Minute-bar streaming detection never actually fires. For Korean bars, kis-bridge sends one bar per ticker per message, while the detector looks at one message at a time and skips any ticker with fewer than two bars. For US bars, New York wall-clock times with the time zone stripped are compared against the UTC container clock, so the “last 5 minutes” filter drops them. The signals that piled up came, in effect, from the daily-bar path the api runs every hour. The first fix is to have the detector keep recent bars per ticker as state and to normalize times to UTC.
  • The API is effectively a single instance. WebSocket connections are tracked in process memory, so running several API replicas means some browsers miss alerts. The HPA allows up to three, but it has to stay at one until broadcasts move to Redis pub/sub.
  • CI has no tests. The pipeline builds only what changed, but there is no test stage before the build, so regressions have to be caught manually.
  • Partition counts and replication factors were never set. Topics are left to auto-creation, so adding consumers yields only limited parallelism, and application topics use the default replication factor (1) even with three brokers.
  • The retention policies never reach the real tables. I wrote policies for compression after a day and 30-day price / 90-day analysis retention, plus cleanup CronJobs, but the table names they target differ from the tables the app creates, so nothing is actually compressed or dropped. Re-pointing the policies at the app’s tables comes first.
  • Gaps in observability. ai-analyzer is listed as a scrape target but exposes no metrics endpoint, kafka-exporter isn’t deployed so the Kafka lag panel may be empty, and Redis is deployed but unused.
  • CronJob time zone. The jobs set timeZone: Asia/Seoul but their schedules were written in UTC, so all six training and maintenance CronJobs run nine hours earlier than intended.