Production-Ready Asynchronous Object Detection API

📌 From Local Script to Production System

In computer vision engineering, moving a deep learning model from a local testing script (model.predict()) into a production cloud environment introduces severe compute bottlenecks. Deep learning inference is extremely CPU/GPU intensive. When multiple clients send high-resolution image payloads concurrently to a standard synchronous API, the server threads freeze, resulting in socket timeouts, thread/RAM exhaustion, and container crashes.

To resolve this, I engineered a Production-Ready Asynchronous Object Detection API using FastAPI and YOLOv8. By leveraging an asynchronous queueing pattern, the API decouples long-running model inference into non-blocking sequential background tasks, protecting cloud hardware from resource starvation while maintaining high service availability.

🏛️ Non-Blocking Background Queue Pattern

The application decouples client requests from heavy model execution using a non-blocking background queue and an asynchronous state polling pattern. Each component has a single, clear responsibility.

🧹 Production Hardening

def fastapi_queue_cleaner():
    """Background loop: wakes every 60s, purges tasks older than 5 minutes."""
    while True:
        current_time = time.time()
        expired_tasks = [
            task_id for task_id, task_info in list(tasks_db.items())
            if current_time - task_info.get("updated_at", 0) > 300  # 5 minutes
        ]
        for task_id in expired_tasks:
            del tasks_db[task_id]
        time.sleep(60)

threading.Thread(target=fastapi_queue_cleaner, daemon=True).start()

nparr = np.frombuffer(file_bytes, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

if img is None:
    raise ValueError("Failed to decode image: corrupted or invalid format")

🐳 Docker & Cloud Deployment

A key production challenge was packaging the API with all low-level graphics dependencies, specifically solving the classic libGL.so.1 missing library error that occurs when running OpenCV on a headless Linux Docker container.

The solution is a custom multi-stage Dockerfile that installs system-level graphics bindings before the Python layer, ensuring cv2.imshow and matrix rendering work correctly inside the container without a display server. The resulting image is deployed to Hugging Face Spaces as a Docker space, exposing a persistent public API endpoint.

# Build the image
docker build -t yolo-queue-api .

# Run locally
docker run -d -p 7860:7860 --name yolo-api yolo-queue-api
# → API live at http://localhost:7860/docs

📂 Codebase Overview

🛠️ Key Tooling

💡 Engineering Decisions That Matter

This project demonstrates engineering judgment, not just model execution, every design decision was driven by a real production constraint.