External transcoding microservice via Kafka

The Challenge

Resolve scalability and performance issues with an extensive library of image and video files, while still supporting metadata extraction and transcoding to multiple formats.

Our Solution

Kept the existing DAM ingestion workflow seamless while redesigning transcoding as an external microservice — storing assets to AWS S3 via a custom S3 Bulk Importer and sending binaries to a transcoding cluster over Kafka.

Impacts

• Lower processing costs
• Improved DAM performance
• Ability to render non-native file formats