How to Scale FFmpeg Processing Without Hitting CPU Limits

There's a moment every developer who works with video eventually hits. The prototype works fine. You upload a file, FFmpeg chews through it, and a minute later you've got a compressed MP4. Then you ship it.

And the first time twenty users upload at once, your server goes from "totally fine" to "unresponsive, dropping requests, fan spinning like a jet engine." You check top. There are twenty ffmpeg processes, each one pinned at 100% of a core, fighting each other for memory bandwidth. Your API queue backs up. Health checks start failing. Somebody on the team says the words "we should probably move this to a bigger instance," and now you're paying $400 a month for a machine that still falls over during a busy Tuesday.

This is the classic FFmpeg scaling problem, and it's not really a code problem. It's a resource-model problem. FFmpeg is a CPU-hungry, memory-hungry, occasionally disk-hungry command-line tool that was designed for one job at a time on your laptop. Bolting it onto a web server and hoping for the best works until it doesn't.

The good news: scaling FFmpeg processing without hitting CPU limits is a solved problem, and there are several paths to solving it. Some are cheap and manual. Some involve architecture work. Some involve handing the whole mess to a managed API and never thinking about vCPU allocation again. This article walks through all of them, with the actual numbers and trade-offs you need to make a decision.

Let's start with why this is hard in the first place.

Why FFmpeg Hits a Wall (And Why It's Usually CPU, Not Code)

FFmpeg is one of the most impressive pieces of software in the open-source world. It supports hundreds of codecs, hundreds of filters, and can handle almost any container format you throw at it. It's also, by design, single-mindedly focused on doing one encode as fast as it can.

When you run something like:

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac output.mp4

You're asking FFmpeg to decode every frame, run it through a software H.264 encoder, and mux the result. The default libx264 encoder will use as many threads as your CPU has cores, roughly speaking. On a 4-core VPS, that one job consumes all four cores. Run two jobs at once and now they're splitting those cores, which means each one takes twice as long and the total throughput doesn't improve much.

This is the core issue. Video encoding is not like a typical web request. A web request that takes 30 milliseconds can be handled by thousands of concurrent connections on a modest machine, because most of that time is spent waiting on I/O. An FFmpeg job that takes 30 seconds is spending nearly all of that time burning CPU cycles doing math on pixels. There's no waiting to overlap. Adding concurrency just divides the same fixed pool of compute.

A few things make it worse:

• Memory bandwidth matters as much as core count. Encoders like x264 and x265 move enormous amounts of data through RAM. On a shared VPS with noisy neighbors, you can lose 30–40% of your expected throughput and never figure out why.
• Filters are single-threaded in places. A complex filter_complex graph with overlays, scaling, and audio mixing can become a serial bottleneck that pinches the whole pipeline.
• Disk I/O can choke things too. If you're reading from and writing to the same spinning volume, or streaming to an S3 bucket over a slow connection, FFmpeg will stall waiting for bytes.
• Memory spikes are real. A 4K decode with a big filter graph can easily want 2–4 GB of RAM for a single process. Ten of those on a 16 GB box and you're into swap, which is worse than just failing.

So when people say "FFmpeg is hitting CPU limits," what they usually mean is: the machine I put in front of it can't run enough jobs simultaneously to keep up with demand, and adding more jobs makes each one slower without increasing total throughput.

That's the wall. Now let's talk about measuring it properly, because you can't scale what you don't measure.

The Real Cost of Video Processing: Understanding Your CPU Math

Before you pick a strategy, you need to know what a "unit" of work costs you. There are two numbers that matter, and confusing them will lead you to make bad decisions.

Wall-Clock Time vs CPU Seconds

Wall-clock time is how long the job takes from beginning to end. If a 10-minute 1080p video takes 90 seconds to transcode, wall-clock time is 90 seconds.

CPU seconds (sometimes called vCPU-seconds) is wall-clock time multiplied by the average number of cores the process was using. If that same job averaged 3.8 cores of usage across those 90 seconds, you've spent about 342 CPU seconds.

Why does this matter? Because most cloud pricing is based on CPU seconds, not wall-clock time. And because it tells you whether your job is parallelizing well. If a job takes 90 seconds on a 4-core machine and 85 seconds on a 16-core machine, it's not parallelizing — you're wasting money on cores the encoder can't use.

You can get a rough read on this by running a job while watching htop. If FFmpeg shows up as one process at 100% of one core, it's single-threaded (common for some filters and codecs). If it shows 700%, you're using seven cores.

Why Concurrency Multiplies the Problem

Here's a simple table that shows why "just run more jobs" stops working.

Notice the pattern. Going from 1 concurrent job to 8 improves your throughput by about 33%, while making every individual job six times slower. Latency for the end user collapses. Your queue's expected wait time goes up, not down, once you factor in that jobs started later have to wait behind slower ones.

For a while, it's a reasonable trade — you're squeezing marginal gains out of the hardware. Then you hit the point where adding a ninth job literally makes the system slower because of context-switching, cache thrashing, and memory pressure. That's the practical ceiling.

The way out isn't to keep adding jobs to the same box. It's to either make each job cheaper, or to move jobs somewhere with more boxes.

Five Scaling Strategies and What Each One Actually Costs You

Let's go through the realistic options, roughly in order of effort.

1. Vertical Scaling (Bigger Machine)

This is the move everyone makes first. Bump the instance from 4 vCPUs to 16, get 4x the throughput, done.

The upside: Zero code changes. Might work for a while.

The downside: Pricing scales linearly or worse, you're paying for capacity during idle hours, and you still have a hard ceiling. A single 64-core machine is expensive and rare in most cloud regions. You also haven't solved failure — if that box dies mid-encode, every job on it dies with it.

Vertical scaling is a stopgap. It buys you time, but it's not a strategy.

2. Queue Plus Worker Pool

This is the standard, sane starting architecture. Instead of running FFmpeg inside your web request handler, you do this:

1. Client uploads a file (or provides a URL).
2. Your API creates a job record and pushes it onto a queue (SQS, Redis, RabbitMQ, whatever).
3. A separate pool of worker processes pulls jobs off the queue and runs FFmpeg.
4. When a job finishes, the worker updates the job record and uploads the output.

The web tier and the encoding tier are now decoupled. If the queue backs up, your API stays responsive. You can size each tier independently.

The catch: You now have a fixed worker pool that's either too small (backlog grows under load) or too large (you're paying for idle machines overnight). Autoscaling the worker pool helps, but FFmpeg workers are slow to start — cold-starting a container that needs to pull a 2 GB image and warm up FFmpeg takes 30–60 seconds, which is a long time when a burst of jobs lands all at once.

3. Autoscaling Containers (Kubernetes, ECS, Cloud Run)

You take the worker pool idea and let the platform handle the sizing. During a spike, the cluster spins up more workers. During quiet hours, it scales back down.

The upside: Reasonable cost efficiency. Reasonable resilience.

The downside: Operational complexity. You need to configure autoscaling policies based on the right metric (queue depth, not CPU utilization — because FFmpeg workers are always at 100% CPU while working, which makes CPU-based scaling useless). You need health checks that actually understand what a busy-but-healthy FFmpeg worker looks like. You need to handle graceful shutdowns so a scaling-down event doesn't kill a running encode.

Also, the underlying nodes still have a finite number of cores. If you're running a shared cluster, your video jobs will fight with your other workloads for CPU.

4. Spot/Preemptible Instances

Video encoding is a great fit for spot instances — it's interruptible, embarrassingly parallel work. You can often get 60–80% discounts.

The upside: Massive cost savings at scale.

The downside: You have to build for interruption. That means checkpointing, retries, and idempotent job processing. It also means handling the case where ten workers all get reclaimed at once because AWS decided to take back a whole availability zone.

Doable. Not trivial.

5. Offloading to a Managed FFmpeg API

You stop running FFmpeg at all. Instead, you send a JSON payload to an API describing what you want done — input URLs, the FFmpeg arguments — and the service runs it for you and hands back the output.

This is the model FFmpeGo uses, and it's the one that scales most cleanly, because the scaling isn't your problem anymore. The provider has a fleet of workers, a scheduler, and a billing model tied to actual compute consumed.

The upside: No servers to manage. No autoscaling policies to tune. No cluster to babysit. If 500 jobs land at once, you don't have a spike — you have a bigger bill, which is much easier to reason about.

The downside: You're trusting a third party with your media pipeline. For some teams that's a dealbreaker. For most, it's a huge win.

Which approach is right depends on your volume, your tolerance for infrastructure work, and how much of your product is "video processing" versus "everything else."

I'd argue if video is not your core product — you're a SaaS app that needs thumbnails generated, or a mobile app that needs user uploads compressed — offloading is almost always the right answer. If video processing is your product, you might want the control that comes with running it yourself.

Cutting CPU Usage Before You Scale Anything

Here's the thing nobody wants to hear: most teams try to scale before they've optimized. Before you spend another dollar on compute, make sure you're not wasting cycles.

Pick the Right Encoder Preset

The -preset flag controls the speed/quality tradeoff. For libx264:

(These are rough, from memory and experience — your mileage will vary by content.)

What this table tells you: moving from medium to fast cuts your encode time in half and costs you about 5% in file size. That is a fantastic trade in most applications. Moving from fast to slow costs you 4x the CPU for maybe 8% smaller files. Almost never worth it.

For a huge number of use cases — user-generated content, social clips, internal review copies — veryfast or faster is more than good enough. Start there and only move slower if someone actually complains about file size.

Hardware Acceleration: When It Helps and When It Doesn't

GPU encoding (NVENC, QuickSync, VAAPI, VideoToolbox) is dramatically faster than software encoding. A job that takes 60 seconds on CPU can take 6 seconds on a modern GPU.

But there's a tradeoff. Hardware encoders produce worse quality at the same bitrate. A good software x264 encode at 2 Mbps might look noticeably better than an NVENC encode at 2 Mbps.

The practical result: hardware encoding is great when you have compute headroom but not bitrate headroom, or when you're doing real-time or near-real-time work where CPU can't keep up. It's not great when you're trying to squeeze a small file out of a large source and quality matters.

Also: GPU instances cost more per hour, and they're harder to autoscale. If you're going to spend GPU money, do the math first — a lot of teams "optimize" with GPUs and end up with a higher bill.

Trim the Filter Graph

filter_complex graphs can be surprisingly expensive. Some common footguns:

• Upscaling. Scaling 720p up to 1080p is pure cost for no quality gain. Don't do it.
• Chained scale operations. Scaling twice in a row wastes cycles. Consolidate into one scale call.
• Expensive scalers. The default bicubic is a good balance; lanczos is sharper but ~2x slower. neighbor is fastest but blocky.
• Overlays on every frame. An overlay filter at 4K input is heavy. If you can do the overlay on a smaller frame and then scale up, it's much cheaper.
• Repeated format conversions. Every format filter in your chain is a real cost. Keep the pixel format consistent.

A common pattern is to write a filter graph that works and never look at it again. Take an hour, profile it, and see if you can cut 30% of the work.

Avoid Re-Encoding When Stream Copy Would Do

If you just need to remux (change container without re-encoding), use -c copy:

ffmpeg -i input.mkv -c copy output.mp4

This is typically 100x faster than a re-encode. If your API is currently re-encoding files just to change container format, you're paying 100x more than you need to.

Similarly, if you're only touching one stream (say, adding a subtitle track), you can stream-copy the video and audio and only transcode what's necessary.

Use Fast Seeking and Smart Input Handling

Placement of -ss matters:

ffmpeg -ss 00:05:00 -i input.mp4 -t 10 ... # fast, seeks before reading
ffmpeg -i input.mp4 -ss 00:05:00 -t 10 ... # slow, decodes everything before the point

The first version uses the container's index and jumps straight to the keyframe near 5 minutes. The second version decodes 5 minutes of video just to throw it away. Same output, wildly different cost.

Also, if your source is on S3, use presigned URLs and let FFmpeg stream via HTTP rather than downloading to local disk first. Half the I/O, half the disk cost.

Break Long Jobs Into Chunks

For long files (say, a 2-hour lecture), you can split the input into segments and encode each one in parallel, then stitch the results. This works well when you don't need single-pass rate control across the whole file.

The tradeoff: chunk boundaries may show quality discontinuities if your encoder's rate control is doing something adaptive. Segmenting into keyframe-aligned chunks (using -force_key_frames during segmentation) mitigates this.

Chunking is a real optimization for very long files or when latency matters more than total cost.

Architecture Patterns That Survive Traffic Spikes

Once you've optimized the individual jobs, the architecture is about smoothing demand.

The Queue Is Non-Negotiable

Whatever else you do, put a queue between your API and your FFmpeg workers. This is the single highest-leverage architectural decision you can make. Without a queue, your system's failure mode under load is "everything falls over at once." With a queue, the failure mode is "wait times go up." One is recoverable. The other isn't.

Prioritize Jobs

If you have a mix of latency-sensitive work (user-facing thumbnails, small previews) and batch work (large archival re-encodes), route them to separate queues. Low-priority jobs should never block high-priority ones.

Fail Fast on Bad Inputs

FFmpeg jobs can hang. A malformed input file, a stuck network fetch, a filter graph in an infinite loop — all of these happen. Always set a timeout on the FFmpeg process (the -t flag for duration, or a wall-clock timeout enforced by your worker). A stuck job holding a worker slot indefinitely is worse than a failed job.

Output to Object Storage, Not Local Disk

Don't write output files to a worker's local disk and then serve them. Write them to S3 (or GCS, or R2) and hand back URLs. This decouples workers from the delivery path, makes cleanup trivial, and means a worker crash doesn't lose completed work.

Design for Idempotency

Every job should have an ID, and processing a job with the same ID twice should produce the same output. This makes retries safe. It's essential if you're using spot instances or a serverless platform where jobs can be retried.

Designing for Failure: Retries, Partial Jobs, and Orphaned Processes

Failure handling is where scaling systems go from "works in demo" to "works under load." A few patterns that matter.

Retry With Backoff

Transient failures (network blip, temporarily saturated downstream) should be retried. Permanent failures (bad input, unsupported codec) shouldn't. Distinguish them by exit code and stderr output — FFmpeg is actually pretty verbose about what went wrong.

Use exponential backoff with jitter. Three retries is usually plenty.

Watch for Zombie Processes

If your worker wrapper crashes or times out but doesn't kill the FFmpeg subprocess, you get zombies that eat CPU for no reason. Always kill the process group, not just the parent. kill -TERM -- -PID is your friend.

Monitor the Right Metrics

Things worth alerting on:

• Queue depth — the leading indicator of a backlog.
• Job success rate — a spike in failures often precedes a capacity problem.
• Per-job CPU seconds — if this drifts up over time, something in your pipeline got more expensive (a codec change, a new filter, bigger inputs).
• P95 latency end-to-end — the number users actually feel.

Graphs of CPU utilization on your worker nodes are basically useless for FFmpeg workloads, because the answer is always "yes, it's pinned." Watch the queue.

The Hidden Costs Nobody Budgets For

If you're comparing "build it ourselves" against "use a service," make sure you're comparing apples to apples. The raw compute cost of an EC2 instance is only one line item. Here's what else creeps in.

Idle capacity. Your workers sit idle overnight and on weekends. You still pay for them. A service that bills per compute second doesn't charge you for idle.

Sunk engineering time. Every hour someone on your team spends tuning an autoscaler, debugging a stuck job, or chasing down why one codec is failing is an hour not spent on your product. For a typical team, this adds up to a noticeable fraction of a full-time engineer.

Failed jobs. Traditional infrastructure bills you for CPU seconds whether the job succeeds or not. If you have an integration bug that causes 5% of your jobs to fail after 30 seconds of processing, you're paying for that 5%. Some services — FFmpeGo, notably — only bill for successful jobs (2xx responses). That's an underrated cost saving.

Overflow and burst. If traffic is spiky, you need enough headroom for the peak, which means paying for capacity that's idle most of the time. This is often the largest single hidden cost.

Egress. Video files are large. Transferring them out of AWS can be surprisingly expensive. Check your egress column carefully.

Hard cap enforcement. A runaway batch job with no monthly cap can produce a genuinely terrifying invoice. Whatever you build, put a kill switch on it.

When a Managed FFmpeg API Is the Right Answer

Okay, let's talk about the service-based approach honestly, including where it's not the right fit.

It's a good fit when:

• Video processing is a feature of your product, not the product itself.
• Your volume is spiky and hard to predict.
• You don't have (and don't want) a dedicated infrastructure engineer.
• You need arbitrary FFmpeg commands — the full flexibility of the command line — rather than a handful of preset operations.
• You want to stop paying for failed jobs.
• You want the cost to be tied directly to what you actually consume.

It's less of a good fit when:

• Your volume is enormous and steady, and the economics of owning hardware beat out any managed service.
• You have hard data residency requirements that a third party can't meet.
• You need sub-second latency, which no remote API can provide.
• You enjoy running infrastructure. That's a real reason, and if it's you, own it.

For the vast majority of teams in the middle of that list, though, a managed API ends up being cheaper once you count everything.

How the Workflow Actually Looks

With a service like FFmpeGo, the mental model is similar to running FFmpeg locally. You send a request describing the input and the arguments you want run, and you get back a result.

Conceptually, a request says: "Here's an input URL. Here are the FFmpeg arguments I'd pass on my own machine. Please run them and give me the output."

Because the service supports arbitrary commands, you're not limited to a menu of preset operations. That means complex filter_complex graphs, multi-input jobs (mixing an audio track with a video, compositing overlays, drawing text), and any codec or container FFmpeg supports.

Billing is based on compute seconds — actual wall-clock execution time of the FFmpeg process — which is a fair unit. If your command is efficient, you pay less. If it's doing something fancy, you pay for what you use.

The two details that matter most operationally:

1. You only pay for successful jobs. Jobs that fail don't count. That changes the calculus for testing and edge-case handling — you can be aggressive about retries without worrying about the cost.
2. Hard monthly caps. You set a ceiling and the service won't exceed it. No "I forgot to check the console and now finance is mad" surprises.

A Practical Migration Walkthrough

If you're currently running FFmpeg yourself and want to move to an API-driven model, here's the sequence I'd suggest.

Step 1: Inventory your commands. Go through your codebase and collect every FFmpeg invocation. You'll probably be surprised at how many variants have accumulated. Group them into a handful of job types.

Step 2: Test each command standalone. Before you integrate anything, run each command against representative inputs and measure the wall-clock time and quality. This is your baseline.

Step 3: Rewrite invocations as API requests. For each command, translate the local ffmpeg ... call into an API request with the input URL and the same arguments. Verify the output is byte-identical (or close enough).

Step 4: Move the job lifecycle to your side. Instead of a worker that runs FFmpeg, your worker now makes an API call and waits for a result. It should have a retry policy and a timeout. Because the API call is cheap to retry, you can be more aggressive about retries than you would be with a local job.

Step 5: Kill your old workers. Once the new path is stable, remove the old infrastructure. If you're on spot instances, cancel them. If you have a Kubernetes namespace dedicated to encoding, decommission it. Enjoy the smaller cloud bill.

Step 6: Set up caps and alerts. Configure a monthly limit. Set up alerting on job failures and on spend.

The whole thing can usually be done in a few days for a team with a mature-enough codebase.

Common Mistakes That Blow Up Your CPU Budget

Some of these are FFmpeg-specific. Some are general scaling failures. All of them show up repeatedly.

1. Running FFmpeg inside the request handler. Always the first mistake. Never run a 30-second job inside a web request. Queue it.

2. Using the default preset without checking. medium is the default. It's almost never the right choice for production workloads.

3. Upscaling by accident. If someone uploads a 480p video and your pipeline outputs 1080p, you're paying 5x more per frame for a worse-looking result.

4. Re-encoding when -c copy would work. Remuxing with stream copy is free compared to re-encoding.

5. Not setting timeouts. A stuck job that runs forever is much worse than a failed job that retries.

6. Scaling based on CPU utilization. FFmpeg workers are always at 100%. Scale on queue depth.

7. Letting one noisy job starve everything else. A 4-hour 4K re-encode and a 10-second thumbnail generation shouldn't share a queue.

8. Ignoring memory. A machine with plenty of CPU but not enough RAM will swap and become useless. Size both.

9. Forgetting about disk. Streaming output to a full disk is a bad way to find out you're out of space.

10. No cap on monthly spend. This one has ended careers.

A Pre-Scaling Checklist

Before you add capacity, run through this.

• I know the wall-clock time and CPU seconds for a representative job.
• I'm using the fastest preset that produces acceptable quality.
• My filter graph has been profiled; no redundant scaling or format conversions.
• I use stream copy wherever possible.
• Fast seeking (-ss before -i) is used for trim operations.
• Jobs run in a queue, not in the request handler.
• There's a timeout on every FFmpeg process.
• The queue is the thing I scale on, not CPU.
• Priorities or separate queues exist for latency-sensitive work.
• Output goes to object storage, not local disk.
• I have a monthly cost cap somewhere.
• I've measured what "one more unit of traffic" costs me in compute.

If you can check all of those boxes and you still have capacity problems, now it's time to add compute — or move to a service that handles that for you.

FAQ

Do I really need a GPU for FFmpeg?

For most workloads, no. A GPU is worth it if you're doing real-time encoding, running at very large scale, or dealing with huge volumes of short clips where the fixed cost of hardware encoding amortizes well. For typical "compress this user upload" workloads, software x264 or x265 on a decent CPU is fine and often produces better quality per bit.

What's the difference between vCPU and CPU seconds?

vCPU is a unit of capacity (how much processing power you have available). CPU seconds is a unit of work (how much you've used). If a job runs for 10 seconds and averages 4 vCPUs of usage, that's 40 CPU seconds. Most cloud billing is in some form of CPU seconds.

Is it faster to run more concurrent FFmpeg jobs or to run one at a time?

For total throughput, running more jobs concurrently (up to the number of available cores) is usually faster overall, even though each individual job is slower. But it hurts per-job latency. If your users are waiting for the output, favor fewer concurrent jobs. If it's a batch pipeline, favor more.

Why does my FFmpeg job use less than 100% of one core?

Either the job is I/O-bound (waiting on disk or network), or something in the pipeline is single-threaded and bottlenecking the rest. Common culprits: certain filters, some audio codecs, and reading from a slow source. Check top and watch for I/O wait percentage.

How do I know if I should build or buy?

Rough rule: if video processing is a core part of your product and you have dedicated infrastructure engineering, build. If it's a supporting feature and infrastructure is a cost center, buy. The break even point is usually somewhere around a few hundred to a few thousand hours of encoding per month, but it varies heavily with how efficient your pipeline is.

What's a reasonable target for per-job cost?

It depends enormously on resolution and length. As a rough benchmark, modern CPUs can encode 1080p at something like 2–5x realtime with a fast preset, so a 10-minute 1080p video might cost 2–5 minutes of wall clock. At typical cloud pricing, that's a few cents. If you're paying dollars per job, something is wrong.

Can FFmpeg run in a serverless function like AWS Lambda?

Yes, but the constraints are real. Lambda caps at 15 minutes per invocation, has limited ephemeral storage (currently up to 10 GB), and cold starts can be slow. It works for short jobs. For long or complex jobs, you'll want a container-based service or an offloaded API.

Final Thoughts

Scaling FFmpeg without hitting CPU limits comes down to three things: make each job cheaper, decouple job execution from your request path, and put a system in front of it that can absorb bursts.

Do those three things and you can push an enormous amount of media through without ever worrying about your instance count again. Skip any one of them, and you'll spend the next quarter babysitting a server that's always one upload spike away from falling over.

If you'd rather not build and maintain that system yourself, FFmpeGo exists specifically for this. You send your input URL and the FFmpeg arguments you'd use locally, and it runs the job for you on infrastructure you never have to touch. Arguably the most useful part: you only pay for successful jobs, and you can set a hard monthly cap so you never get a surprise bill.

The next time your video pipeline starts struggling, take the checklist above and start working through it. And if it turns out the right answer for your team is to stop running FFmpeg at all, now you know a good place to look.