Modal logo
Guide Introduction to Modal Getting started Functions Functions Invocation methods Function lookups Container lifecycle management Parametrized Functions Dynamic configuration Sandboxes Sandboxes Running commands Networking and security File access Snapshots Pricing and resources Sidecars BetaSecret injection BetaEndpoints Endpoints Shared Endpoints Dedicated Endpoints Endpoint candidates Endpoint integrations Training Quickstart Tinker-compatible API BetaMulti-node training Images Defining Images Using existing container images Named images Fast pull from registry GPUs and other resources GPU acceleration Using CUDA on Modal Configuring CPU, memory, and disk Scaling out Scaling out Input concurrency Batch processing Job queues Dynamic batching Multi-node Clusters Deployment Apps, Functions, and entrypoints Managing deployments Continuous deployment Scheduled Functions HTTP Applications Servers Sticky Sessions Web Functions Streaming endpoints Web Function URLs Request timeouts Proxy Tokens Networking Tunnels Proxies BetaCluster networking Data sharing and storage Passing local data Volumes Storing model weights Cloud bucket mounts Dicts Queues Dataset ingestion Secrets and environment variables Secrets Environment variables Performance Cold start performance Memory Snapshots High-performance LLM inference Reliability and robustness Failures and retries Preemption Timeouts GPU health Troubleshooting Security and privacy Security and privacy Data residency Customer-supplied encryption keys AlphaAudit logs Integrations Using OIDC to authenticate with external services Connecting Modal to your Datadog account Connecting Modal to your OpenTelemetry provider Okta SSO Microsoft Entra SSO Custom SAML SSO SCIM Integration Slack notifications BetaWorkspace & account settings Workspaces Environments Service users User groups Role-Based Access Control (RBAC) Billing Network egress billing Budgets Other topics Feature maturity Notebooks Restricted Functions JavaScript/Go SDKs BetaModal 1.0 migration guide File and project structure Developing and debugging Jupyter notebooks Asynchronous API usage Global variables Region selection GPU Metrics

Endpoint candidates

A Dedicated Endpoint serves traffic from its production deployment. Create candidates to test changes to your model or weights without changing production. Each candidate has its own URL and containers.

Create a candidate 

Open your Endpoint in the dashboard, then open the Deployment dropdown and choose Create candidate.

An Endpoint can have up to five candidates at once. Choose a model from the Modal Library with the same task type as production: text generation or embeddings. To use custom weights, select Custom weights, then provide a Hugging Face repository and revision or a Modal Volume and path. The weights must be compatible with the selected base model; see Serve custom weights.

Changes to the source weights do not update an existing candidate. Promotion uses the same weights you tested.

Candidates start with the serving recipe’s autoscaling defaults, not production’s settings.

Test a candidate 

Select the candidate in the Deployment dropdown and wait for provisioning to finish. Copy its URL from the dashboard to send test requests. Candidates use production’s model name and authentication settings.

Mirror production traffic 

Traffic mirroring copies production requests to a candidate. Clients receive production’s responses; candidate responses are discarded.

Select a provisioned candidate, click Mirror traffic, and choose the percentage of traffic to copy. You can mirror to multiple candidates, each with its own percentage. To disable it, open the mirroring control and click Stop mirroring.

Mirrored requests can scale up the candidate and incur compute charges.

Promote a candidate 

Select the candidate and click Promote. Promotion starts a new production rollout using the candidate’s model, weights, and serving recipe. The production URL and request model name stay the same.

Production keeps its minimum, maximum, and buffer container settings and its scale-up and scale-down windows. Candidate autoscaling changes do not carry over to production.

Promotion removes the candidate when the rollout starts and stops its mirrored traffic. Its URL is no longer available; other candidates are unchanged.

To change back after a successful promotion, create and promote a new candidate using the previous model or weights.

Delete a candidate 

Select the candidate and click Delete candidate. Deletion stops its containers and any mirrored traffic to it. Production and other candidates are unaffected.

Stopping the Endpoint stops all its candidates.

Pricing 

You pay for the compute resources each candidate’s containers use. A candidate scaled to zero does not incur compute charges.