Memory Snapshot

Modal can save the state of your Function’s memory right after initialization and restore it directly later, skipping initialization work.

These “memory snapshots” can dramatically improve cold start performance for Modal Functions.

During initialization, your code might read many files from the file system, which is quite expensive. For example, the torch package is hundreds of MiB and requires over 20,000 file operations to load! Such Functions typically start several times faster with memory snapshots enabled.

How do I use memory snapshots? 

You can enable memory snapshots for your Function with the enable_memory_snapshot=True parameter:

@app.function(enable_memory_snapshot=True)
def my_func():
    print("hello")

Then deploy the App with modal deploy. Memory snapshots are created only for deployed Apps.

When using classes decorated with @cls, @modal.enter() hooks are not included in the snapshot by default. Add snap=True to include them:

@app.cls(enable_memory_snapshot=True)
class MyCls:
    @modal.enter(snap=True)
    def load(self):
        ...

How do I use GPU memory snapshots? 

Pass the additional option experimental_options={"enable_gpu_snapshot": True} to your Function to enable GPU snapshotting.

@app.function(
    gpu="a10",
    enable_memory_snapshot=True,
    experimental_options={"enable_gpu_snapshot": True},
)
def my_gpu_func():
    print("hello CUDA")

Otherwise, memory snapshots only capture CPU memory. GPU memory is not available during snapshots (though CUDA drivers are).

Refer to the code sample here for a more complete example.

What are the limitations of GPU memory snapshots? 

GPU memory snapshots are in alpha. We’ve seen that they can massively reduce cold boot time but we are still exploring their limitations. Try it for yourself and let us know how it goes!

When are snapshots updated? 

Redeploying your Function with new configuration (e.g. a new GPU type) or new code will cause previous snapshots to become obsolete. Subsequent invocations to the new Function version will automatically create new snapshots with the new configuration and code.

Changes to Modal Volumes do not cause snapshots to update. Deleting files in a Volume used during restore will cause restore failures.

I haven’t changed my Function. Why do I still see snapshots being created sometimes? 

Modal recaptures snapshots to keep up with the platform’s latest runtime and security changes.

Additionally, you may observe your Function being memory snapshot multiple times during its first few invocations. This happens because memory snapshots are specific to the underlying worker type that created them (e.g. low-level processor details), and Modal Functions run across a handful of worker types.

Snapshots may add a small amount of latency to Function initialization.

CPU-only Functions need around 6 snapshots for full coverage, and Functions targeting a specific GPU (e.g. A100) need 2-3.

How do snapshots handle randomness? 

If your application depends on uniqueness of state, you must evaluate your Function code and verify that it is resilient to snapshotting operations. For example, if a variable is randomly initialized and snapshotted, that variable will be identical after every restore, possibly breaking uniqueness expectations of the proceeding Function code.