← Projects

BARO — Autonomous Taxi Dispatch & Control Platform

  • Apr–Jun 2026
  • Team of 5 · Backend / DevOps
  • AWS ECS · Terraform
  • MQTT · Kafka
  • OpenStack · K3s

Overview

For autonomous taxis to work on real roads, you have to ingest the positions of hundreds of vehicles continuously, dispatch the nearest idle car, and send cars that finish a trip to where demand is high. BARO ties that whole loop — request → dispatch → trip → relocation — into one driverless taxi dispatch and control platform.

Instead of real cars, a Python simulator runs 1,500 vehicles starting from 254 taxi stands in Seoul. The services run on AWS (ECS Fargate plus self-hosted Kafka and Mosquitto), while analytics, observability and long-term storage run on K3s on an on-prem OpenStack cloud. The two sides are connected by an AWS Site-to-Site VPN.

It was the cloud-track capstone project for cohort 3 of Hyundai AutoEver’s Mobility SW School, built by a team of five from April to June 2026. My role was backend and DevOps.

Architecture

Scroll and the diagram follows the text. Point at or select a node to see its specs.

At a glance

BARO splits into rider apps and vehicles, AWS where the services run, and on-prem (OpenStack K3s) for analytics and observability. AWS and on-prem are joined by a Site-to-Site VPN.

ECS Fargate ×7EC2: Kafka · MosquittoIPsec VPN

1How vehicle positions flow

1,500 vehicles publish their position over MQTT (QoS 0) every second (3 seconds of simulated time at 3× speed). The control service receives them through a shared subscription, so each message goes to only one instance even as it scales out.

vehicles/{id}/telemetry$share/control-service

The control console gets positions straight over SSE; everything else goes to the Kafka vehicle-data-topic. Messages are keyed by carId, so each vehicle’s positions stay in order.

4 partitionskey = carId

The dispatch service drops messages older than 10 seconds and keeps only idle and relocating cars in the Valkey GEO index. The on-prem consumer reads the same topic across the VPN and stores it in TimescaleDB.

GEO dispatch:cars:idle:geohypertable vehicle_data

2From one request to a dispatched car

A request passes the ALB and then the gateway. JWTs are issued by the user service and verified only at the gateway. The dispatch service first shows a fare and route from Kakao (the pre-dispatch quote).

X-Authenticated-User-Idquote TTL 10 min

On confirmation it finds 10 candidates within 15 km, skips cars silent for over 300 seconds, and takes a SETNX lock (30 s) so two riders can’t grab the same car. The dispatch is saved in a transaction.

GEOSEARCH 15kmSETNX TTL 30s

The command reaches the car over MQTT (QoS 1) and an ACK comes back. If no ACK arrives within 10 seconds, the next car is dispatched automatically.

vehicles/{id}/commandsACK timeout 10s

The rider app gets live vehicle positions and arrival status (pickup → destination) over SSE.

SSE vehicle-location

3After the trip — relocation

When a car reports arriving at its destination, the control service notifies the relocation service asynchronously, kept separate from the dispatch flow.

ARRIVED(to_dest)202 Accepted

PostGIS scores nearby stands (0.7 · demand − 0.3 · distance) and the car is sent to the best one with RELOCATE. It stays dispatchable while moving, so it can take a new request right away.

ST_DWithin 10→30kmstatus = relocating

4Overnight — learning demand

Every day at 02:00, Airflow pulls the last 24 hours of dispatch data through the internal ALB across the VPN and aggregates demand by stand × day of week × time slot.

gzip CSV exportX-Internal-Api-Key

Stand weights learned with XGBoost are pushed to the relocation service and take effect from the next relocation.

weights 0–1

5Keeping watch — observability

On-prem Prometheus scrapes service, Kafka and EC2 metrics across the VPN, and alerts go to three Slack channels: AWS, on-prem and services.

JMX :9404CloudWatch exporterAlertmanager → Slack
At a glance1 / 13
Clients · vehiclesAWS · ap-northeast-2VPC 10.20.0.0/16On-prem · K3sSITE-TO-SITE VPN · IPSECRider web appReact · SSE clientControl consolebaro-admin · SSEVehicles ×1500asyncio simulatorKakao MobilityRoute & fare APISlack3 alert channelsPublic ALBHTTPS · path routinggatewayJWT · circuit breakeruserAccounts · tokensMosquittoEC2 · MQTTdispatchDispatch · locks · SSEcontrolMQTT ↔ Kafka hubValkeyElastiCache · GEORDS PostgresPostGIS · schemasKafkaKRaft · 4 partitionsrelocationStand scoringInternal ALB/internal · metricsPrometheusGrafana · alert rulesAirflowDaily 02:00 demand modelTimescaleDBTime-series hypertablekafka-consumerConsumes across VPN

Try it

  • Idle—
  • Picking up—
  • On trip—
  • Relocating—

Speed —Dispatched —

A demo where 1,500 cars move on a map and requests, dispatch, trips and relocation play out on their own

BARO’s real dispatch rules (10 nearest candidates within 15 km, 10 s ACK timeout, relocation score), replayed in the browser. Cars follow the OpenStreetMap road network at 60 km/h, and demand weights are examples.

Operations log

When Kafka stalled, the control console stalled too

Symptom
The control console showed only one vehicle, and Mosquitto logs filled with “Broken pipe” errors.
Cause
The Kafka producer’s default wait (max.block.ms, 60 s) blocked the MQTT receive thread, and the broker’s queue overflowed.
Fix
Cut max.block.ms to 500 ms and retries to 0 so a Kafka outage can no longer spill into MQTT ingestion.
Result
Even with Kafka down, MQTT ingestion and the console’s SSE keep working.

The control service fans each MQTT position out to two places: the control console (SSE) and Kafka. When the Kafka broker went down, the producer waited for metadata and blocked in send() for up to 60 seconds — inside the MQTT receive thread. While ingestion stalled, Mosquitto’s queue filled up and the connection broke (“Broken pipe”), leaving just one vehicle on the console.

Kafka only stores positions, so losing a few briefly is acceptable; the console must never freeze. Publishing now fails fast when it takes too long, and the failure log is written at most once every 30 seconds. Failures are counted by the existing metric (baro_control_telemetry_kafka_publish_failed_total).

Lesson A default timeout on a synchronous call can carry a failure into the next system. I set timeouts per path instead of trusting defaults.

Consumer lag hit 2.9M — the culprit was a DB query on the hot path

Symptom
The dispatch service fell behind on positions, and Kafka consumer lag grew to 2.9M.
Cause
For every position message, it looked up the vehicle’s active dispatch in the database.
Fix
Cached the vehicle → dispatch mapping in memory so message handling no longer touches the database.
Result
The per-message query disappeared and the lag cleared.

The dispatch service reads every position from Kafka and, if the car is currently dispatched, pushes it to the rider app over SSE. The problem was that it asked the database “is this car dispatched?” for every message. With 1,000 cars at the time, each reporting once a second, that meant close to 1,000 queries per second; consumption couldn’t keep up with the publish rate, and the lag ballooned to 2.9M.

Now the mapping is registered when the rider app opens its position stream (SSE) and removed when the last connection closes, and message handling reads memory only.

Lesson On a path that runs for every message, even one query is expensive. I move I/O off the hot path first.

At 2,000 vehicles, Mosquitto dropped messages

Symptom
Load tests with more vehicles showed Mosquitto dropping messages past its queue limit, and at 2,000 vehicles the control service’s MQTT connection was cut too.
Cause
Messages beyond max_queued_messages (default 1,000, later 10,000) were being discarded.
Fix
Raised the limit to 10,000; when that overflowed too, switched to unlimited (0), then within minutes rolled back to a 50,000 cap because of the out-of-memory risk.

During load testing, messages past the default limit (1,000) were being dropped, so the limit was raised to 10,000 and vehicle connections were staggered at 0.05 s per 10 cars. When the load went up to 2,000 vehicles, the 10,000 limit overflowed as well and the control service’s MQTT connection was cut. The limit was switched to unlimited (0), but unlimited means broker memory grows without bound whenever a subscriber slows down. On a single t3.micro broker that is an OOM risk, so within minutes it was rolled back to a 50,000 cap.

Lesson A queue limit is a “drop when full” policy. Even when raising it, I set a ceiling memory can actually hold instead of going unlimited.

The VPN tunnel was up, but traffic to AWS vanished

Symptom
The StrongSwan tunnel showed ESTABLISHED, yet on-prem hosts could not reach private endpoints in AWS.
Cause
When StrongSwan (charon) brings up an SA it can add policy routes to Linux routing table 220, which is evaluated before the main table, so traffic for the AWS range left through the Wi-Fi default route instead of the VTI.
Fix
Made the host VPN script remove the conflicting table 220 routes whenever the tunnel state changes.

The AWS VPC and the on-prem OpenStack cloud are joined by a Site-to-Site VPN, and the on-prem end is a StrongSwan VTI tunnel on the OpenStack host. The tunnel looked healthy, but connections to private endpoints in AWS failed. ip route get <AWS private IP> showed the route going out the Wi-Fi interface rather than the VTI, and ip route show table 220 held routes for the AWS range: the routes charon had added were being applied before the VTI routes in the main table.

The host script now cleans up those routes whenever the tunnel state changes. After a reboot, the operating procedure restores br-ex, host routes and iptables first, and only then brings up the tunnel and its VTI routes.

Lesson A tunnel being up is not the same as traffic using it. I check the route and source IP that are actually selected with ip route get first.

Design decisions

Running Mosquitto on EC2 instead of AWS IoT Core

Context
Vehicles first talked to AWS IoT Core (mTLS, X.509 certificates). As the simulated fleet grew, connection rate limits forced a 1.2 s delay per 10 cars, and costs scaled with the number of vehicles.
Alternatives considered
  • Keep AWS IoT Core (the original setup)
Why
The broker’s settings (queue limits, auth, sessions) are fully tunable, and without a connection rate limit, 1,000 cars connect in about 5 seconds at 0.05 s per 10 cars.
Trade-off
Availability, patching and monitoring are on us. Credentials live in Secrets Manager and deployment is automated through SSM to reduce the operational burden.

Moving Kafka from ECS Fargate to EC2 + EBS (single-node KRaft)

Context
Kafka first ran as an ECS Fargate task with data on EFS. The advertised listener address clients connect to was hard to pin down, and the log storage sat on a network file system.
Alternatives considered
  • Keep ECS Fargate + EFS (the original setup)
  • Amazon MSK
Why
A fixed private IP and a Cloud Map name (kafka.baro.internal) pin the advertised listener, and logs sit on EBS gp3. vehicle-data-topic has four partitions to match the dispatch service’s consumer concurrency of four.
Trade-off
With one broker (RF=1), a broker failure stops the position stream. Positions are ephemeral and kept for only an hour, so cost came first.

QoS 0 for positions, QoS 1 for commands — plus shared subscriptions

Context
Positions pour in from every car every second or so and are soon replaced by newer ones, while a single lost dispatch command, arrival event or ACK breaks a dispatch. The control service also had to scale to several instances.
Alternatives considered
  • QoS 1 for every message
  • Every control instance subscribing to all topics
Why
Positions are sent at QoS 0 to stay lightweight; commands, events and ACKs use QoS 1 for at-least-once delivery. The control service uses a shared subscription ($share/control-service/…) and a per-instance clientId, so with several instances each message is handled by only one.
Trade-off
QoS 0 positions are lost while a connection is down. Each one is soon replaced by a newer position, so that loss was accepted.

SSE instead of WebSocket for live updates

Context
The rider app and the control console only need to receive positions and status from the server.
Alternatives considered
  • WebSocket
Why
SSE is enough for one-way push, and because it is plain HTTP it passes through nginx and the Vercel function proxy unchanged. When the connection drops, the rider app reconnects with exponential backoff (1 s up to 10 s) and the control console relies on the browser’s built-in EventSource reconnect.
Trade-off
SSE connections live in server memory, so scaling out splits them across instances (see Limits & retrospective).

Authenticate once at the gateway, block internal paths at two layers

Context
With five services, verifying JWTs in each one would scatter verification logic and key management.
Alternatives considered
  • Verify JWTs in every service
Why
JWTs are verified only at the gateway, and user details travel in X-Authenticated-User-* headers; if a client sends headers with those names, the gateway strips them. The /internal/** APIs called by the on-prem batch jobs (Airflow) are checked with X-Internal-Api-Key, and service-to-service calls travel only over Cloud Map internal addresses. Internal paths are blocked at two layers — the ALB (403) and the gateway (404).
Trade-off
Services trust the headers on the assumption that they sit behind the gateway, so ALB and security group rules must keep internal paths from ever being exposed.

One RDS instance, one schema per service

Context
A database instance per service multiplies cost by the number of services.
Alternatives considered
  • An RDS instance per service
Why
One RDS PostgreSQL instance (db.t4g.micro) holds user, dispatch, relocation and control schemas; the user, dispatch and relocation services each use only their own schema through currentSchema in the connection URL (the control schema exists, but the control service uses no database). A one-off ECS task (db-init) creates the schemas.
Trade-off
The services share one database account, so nothing but convention keeps them out of each other’s schemas, and every service shares database failures and load. Per-service accounts and instances are the next step.

Infrastructure & cost

git pushmainActionsGitHub · path filterBuild · testJDK 21 · GradleECRimage per serviceECS deploypassing CI only

AWS, built with Terraform

  • A single envs/dev environment is managed in Terraform: the VPC (2 AZs, public and private subnets), public and internal ALBs, seven ECS Fargate services, RDS PostgreSQL, ElastiCache (Valkey), EC2 for Kafka and Mosquitto, Cloud Map, Secrets Manager and the Site-to-Site VPN.
  • State lives in S3 with a DynamoDB lock. GitHub Actions runs fmt · validate · plan on pull requests and apply → db-init → edge deploy (SSM) on pushes to main.
  • To prevent accidental teardown, destroy runs only after you type a confirmation string such as destroy-dev.

Deployment pipeline

The server pipeline detects changed paths and builds and tests only those services; if all of them pass CI, it pushes their images to ECR and deploys them to ECS (a single failure blocks the deploy). The rider mobile web app deploys with CodeDeploy blue/green and rolls back automatically on failure.

Keeping costs down

  • One switch, runtime_enabled=false, tears down the NAT gateway, ALBs, ECS, RDS and EC2 while keeping the foundation — the VPC, ECR and secrets. ElastiCache and the Site-to-Site VPN sit outside the switch and stay up.
  • An RDS snapshot is taken automatically before teardown, and the next apply restores the latest one.
  • Kafka’s EC2 went from t3.medium to t3.small, the bastion was removed in favor of SSM-only access, log retention was cut to 7 days, and per-vehicle logs were lowered to DEBUG level to cut CloudWatch costs.

My contribution

I took part in BARO across the board, from planning to deployment and operations, in close discussion with my teammates. What follows is the code I personally contributed: between late April and late June 2026 I merged 128 pull requests across six repositories.

Getting vehicle positions to the screen and to dispatch

  • I reshaped the control service, which receives vehicle positions, into a single MQTT → Kafka path. I removed its early Redis and PostGIS dependencies and added an endpoint that keeps vehicle state in memory and pushes it to the control console over SSE.
  • I fixed MQTT session and inbound-client issues and a 404 on the ALB path (/control), and added trip status and plate numbers to the position messages.
  • I made the Kafka consumers store trip status with the vehicle data, and made each consumer switchable per environment.

Dispatch and relocation

  • I made vehicles that haven’t reported for over 30 seconds drop out of the dispatch candidates (Redis lastSeen), and made an unacknowledged dispatch go to the next car.
  • I built the flow that automatically relocates a vehicle to a high-demand area after a trip. A relocating vehicle can still be dispatched if a request comes in.
  • I split the admin routes in the gateway and made it check admin rights against the role claim in the JWT.

Control console

  • I built the real-time fleet control console: vehicle positions, trip routes and relocating vehicles (purple markers with their destinations) on a map, behind an admin login with JWT.
  • I also fixed the rider app map breaking when the window was resized or the user zoomed.

Infrastructure and delivery (Terraform)

  • In Terraform, I built the Site-to-Site VPN between the baro-dev VPC and the on-prem network, moved Kafka from ECS to EC2 with an attached EBS volume (Cloud Map DNS), and replaced AWS IoT Core with Mosquitto on EC2.
  • I built the ECR build and ECS deploy pipeline for the control service, and a step that deploys the vehicle simulator over SSM after terraform apply.
  • To cut cost, I added an automatic RDS snapshot before teardown with automatic restore on the next apply, moved Kafka’s EC2 to t3.small, removed the bastion (SSM only) and lowered CloudWatch log costs. Kafka’s EBS volume is protected from accidental deletion.

Fixes while scaling to 1,500 vehicles

  • Removing a per-message DB lookup from the hot path cleared a 2.9M consumer lag (incident).
  • When Kafka stalled, the producer’s wait blocked MQTT intake too; I cut that wait short (incident).
  • Mosquitto’s queue overflowed and dropped messages; broker queue settings and staggered simulator connections (0.05 s every 10 vehicles) stopped it (incident).

Limits & retrospective

Known limitations, and what I would change next time.

  • In-memory state blocks horizontal scaling. The control service’s vehicle state, the dispatch service’s pending-ACK list and SSE connections all live in JVM memory, so adding instances splits the state. Moving this state to a shared store such as Redis, or to pub/sub, is the next step.
  • Single points of failure remain. Kafka runs on one broker (RF=1), there is one NAT gateway, and RDS is single-AZ. That was a cost-first choice; production would need three brokers, a NAT per AZ and Multi-AZ RDS.
  • Service-to-service trust rests on the network alone. Calls between services travel only over Cloud Map internal addresses but carry no authentication, and every service shares a single RDS master account. Service-to-service authentication and per-service database accounts are the next step.