After migrating from gm to sharp we noticed a strange behaviour of our image processing servers. They memory consumption seemed inadequate and constantly growing.
At first we suspected a memory lead in our JS code, but after some research it turned out that all this memory was consumed by either sharp or libvips. And it's not, strictly speaking, a leak, but rather a very strange and greedy memory allocation.

Using node.js 14 partially negates the issue, but it's still there.

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?

Yes, sharp@0.27.2

What are the steps to reproduce?

Start processing some large images concurrently and memory consumption will grow very rapidly.

At some point memory consumption will reach it's limit and will stay there. But increasing the size of processed images will result in further memory allocation.

What is the expected behaviour?

• sharp should limit it's memory usage with some sane threshold
• sharp should free allocated memory whet it's no longer needed

Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?

Here is a simple code example to illustrate strange memory consumption behaviour:

const sharp = require('sharp');

const filePath = '4ce2474a893e1e02e5f342808fe618a7.jpg';
const outputFormat = 'raw'; // raw format leaks memory faster
const parallelTasks = 10;
const totalTicks = 100;

const bytesToMb = (bytes) => (bytes / Math.pow(1024, 2)).toFixed(2);

// disable internal cache to increase mamory leaking speed
sharp.cache(false);

let pendingCount = 0;
let ticksCount = 0;
let errorsCount = 0;

const tick = async (trials = 0) => {
	pendingCount += 1;

	try {
		await sharp(filePath).toFormat(outputFormat).toBuffer();
	} catch (err) {
		console.log(err);
		errorsCount += 1;
	} finally {
		pendingCount -= 1;
		ticksCount += 1;

		const {rss, heapTotal} = process.memoryUsage();

		process.stdout.clearLine();
		process.stdout.cursorTo(0);
		process.stdout.write(
			`${ticksCount} ticks, ${errorsCount} errors, heap ${bytesToMb(heapTotal)}MB, rss ${bytesToMb(rss)}MB`
		);

		if (ticksCount + pendingCount < totalTicks) {
			setTimeout(tick, 0);
		}
	}
}

Array(parallelTasks).fill().map(()=>{
  tick();
});

Here is a result of running this script on my system using node.js v12.18.1:

↪ node test-memory.js
100 ticks, 0 errors, heap 8.48MB, rss 2105.86MB

You could use any outputFormat. I used raw because it leaks memory faster than compressed formats.

At first I thought that the issue is related to .toBuffer internal buffering, but using files or streams will result in exactly the same memory consumption behaviour.

Here is an example of running this script with jpeg format and an infinite total ticks threshold:

↪ node test-memory.js
1606 ticks, 0 errors, heap 5.48MB, rss 1406.85MB

Memory consumption stops somewhere around this value, so it's not an actual leak.

Doing the same with raw format results in node.js being killed by OOM killer after consuming all available memory.

Are you able to provide a sample image that helps explain the problem?

Any large image will do, here is the one I used in my tests:

What is the output of running npx envinfo --binaries --system?

  System:
    OS: Linux 5.8 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    Memory: 11.44 GB / 15.35 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 12.18.1 - ~/.nvm/versions/node/v12.18.1/bin/node
    Yarn: 1.22.4 - ~/.nvm/versions/node/v12.18.1/bin/yarn
    npm: 6.14.5 - ~/.nvm/versions/node/v12.18.1/bin/npm

But the same behaviour could be reproduced using node docker container.

I reproduced aforementioned issue using node:8, node:10 and node:12 containers.

With node:14 script output seems fine, but memory consumption is still high. And memory is still not released.
But at least with node:14 memory consumption lo longer goes through the roof.