Video Transcoding Engine
Distributed FFmpeg → HLS pipeline with autoscaling workers and live streaming.
Role · Backend engineer — transcoding pipeline, queue design, worker scaling and live-stream orchestration.
- 9
- Job queues
- 4
- Encode outputs
- Hetzner
- Worker cloud
- 4
- Live-stream hooks
Overview
The backend of an in-house video platform. It ingests chunked uploads to object storage, transcodes them to adaptive HLS on a fleet of Celery workers that autoscale on Hetzner Cloud, and exposes gallery, drive, analytics, comments and live-stream APIs to the admin console and the learning platform.
Architecture
API tier
Django REST Framework viewsets for media, folders, tags, analytics, live streams, workers and webhooks.
Queue tier
Celery on Redis. Jobs are routed by file size and priority; locks prevent duplicate encodes when workers recycle.
Worker fleet
Containerised FFmpeg workers on Hetzner Cloud, scaled by a scheduled task from server snapshots.
Storage & delivery
Raw uploads and HLS renditions in Cloudflare R2, served through a CDN.
Live
Stream-key authentication and event callbacks from the streaming server; streams map back to media records.
What it does
Ingest
- Chunked uploads landing as raw objects in Cloudflare R2
- Media, encode-profile, encoding, folder and tag models
- Thumbnails and media comments
Transcoding
- FFmpeg to adaptive HLS with configurable encode profiles
- Thumbnail and GIF preview generation
- WebVTT chapter generation
Distributed workers
- Nine queues: small / medium / large files × default / priority / batch lanes
- Redis distributed locks with expired-lock cleanup
- Queue dispatcher, encode cancellation and periodic worker health checks
Autoscaling
- A scheduled task provisions workers from cloud snapshots while backlog exists
- Workers are retired when the queue drains
Live streaming
- Stream-key pool and live-stream lifecycle
- Authentication, event, mapping and release hooks for the streaming server
Library & analytics
- Gallery and drive with folders, tags and gallery-wide search
- System logs and notifications
- Player analytics collection and reporting
- Outbound encode-complete webhooks to subscribed consumers
Access
- JWT login and signup with a custom user model
- Roles for admins, standard users, uploaders, API clients and faculty
Engineering decisions
Right-sizing work
Routing by file size × priority keeps a multi-gigabyte lecture from blocking a batch of short clips, while priority lanes serve urgent re-encodes.
Exactly-once encodes
Distributed locks with TTL and a cleanup task guard against duplicate encodes when workers are recycled mid-job.
Elastic cost
Workers exist only while backlog exists, so compute is paid for during encode bursts rather than around the clock.