Skip to content
AI & Tools

LLM Inference Providers Compared: Groq, Together AI, DeepInfra, Replicate, Modal and Baseten

AIDToolStack EditorialNovember 7, 20266 min read
LLM InferenceGroqTogether AIModalPricing
Share:Twitter

You've prototyped on a frontier model API, the bill is starting to matter, and someone suggests "just run an open model somewhere cheaper." Then you open six tabs (Groq, Together AI, DeepInfra, Replicate, Modal, Baseten) and realise they aren't selling the same thing. Some sell tokens. Some sell GPU time. Some sell a place to run your own container that happens to have a GPU in it.

This post sorts them by what you're actually paying for, with prices as of November 2026 from each provider's own pricing page. I don't quote speed benchmarks unless the provider publishes the number itself, and where I do, it's their number, not a test.

Three things you can buy

The three ways inference providers bill: per token on a shared model, per GPU-hour on a dedicated endpoint, or per second for your own container
The three ways inference providers bill: per token on a shared model, per GPU-hour on a dedicated endpoint, or per second for your own container
  1. Tokens on a shared model. The provider runs a popular open model and bills per million input and output tokens. You get no idle cost and no control over hardware. Groq, Together AI serverless, DeepInfra and Baseten's Model APIs all do this.
  2. A dedicated endpoint. The provider runs a model on GPUs reserved for you and bills per GPU-hour or minute, with or without traffic. Together, DeepInfra and Baseten offer this.
  3. Your own code on rented GPUs. You bring a container or a Python function, and the platform bills by the second while it runs. Modal is built around this, and Replicate does it for private models.

Most teams should start at 1 and move right only when a number forces them to: steady high volume, a fine-tuned model nobody hosts, or a latency target the shared tier can't meet.

The token sellers

These are the easiest switch from OpenAI, because they speak the same API. Each one documents an OpenAI-compatible base URL:

from openai import OpenAI
import os

# Groq:       https://api.groq.com/openai/v1
# Together:   https://api.together.ai/v1
# DeepInfra:  https://api.deepinfra.com/v1/openai
client = OpenAI(
    base_url="https://api.together.ai/v1",
    api_key=os.environ["TOGETHER_API_KEY"],
)

Sources: Groq, Together, DeepInfra. "Compatible" has limits. Groq's page, for example, says logprobs, logit_bias and top_logprobs return a 400 error and that n must be 1, so test the parameters you rely on.

To compare like with like, here's one open model that several providers list, OpenAI's gpt-oss-120b, per million tokens as of November 2026:

ProviderInputOutputSource
Groq$0.15$0.60models page
Together AI$0.15$0.60pricing
Baseten Model APIs$0.10$0.50pricing

DeepInfra's pricing page didn't list that model when I checked, but its Llama-3.3-70B-Instruct-Turbo was $0.10 in and $0.32 out, and Qwen3-32B $0.08 in and $0.28 out.

What separates them beyond price:

  • Groq builds its own inference hardware (groq.com describes its LPX working alongside Nvidia GPUs), and its models page lists its own speed figures: 500 tokens per second for gpt-oss-120b and 1,000 for gpt-oss-20b. Those are Groq's numbers. The catalogue is short, and some models (like llama-3.3-70b-versatile) show "Contact Sales" in place of a price. On ownership: Groq announced on December 24, 2025 a non-exclusive licensing deal with Nvidia. Its founder and president moved to Nvidia, Simon Edwards became CEO, and Groq said it stays independent and that GroqCloud "will continue to operate without interruption." There's a free plan with lower rate limits.
  • Together AI has the broadest menu: serverless tokens, dedicated endpoints, GPU clusters and fine-tuning (for example $0.34 per million training tokens for supervised fine-tuning of Llama 3.1 8B, with a $4 job minimum, per its pricing page).
  • DeepInfra has service tiers that multiply the base price: Priority at 1.5× for faster time to first token, Flex at 0.8× for cheaper and slower. It also caps accounts at 200 concurrent requests unless you talk to sales, and you add a card or prepay before using it.

Our Groq vs Together AI and Together AI vs DeepInfra comparisons go further on each pair.

The GPU-time sellers

Once you're paying for GPU time, the question becomes: what do I pay when nobody is calling? Here's one H100 across providers, as listed on each pricing page in November 2026. They aren't identical products, so read the "what you get" column.

ProviderH100 priceBilling unitWhat you get
DeepInfra$2.90/hourminute granularityDedicated GPU for a custom LLM (pricing)
Modal$0.001097/second (about $3.95/hour)per secondYour function in a container; scales to zero (pricing)
Together AI$5.49/hourlisted per GPU-hourSingle-tenant dedicated endpoint (pricing)
Replicate$0.001525/second ($5.49/hour)per secondPrivate model on dedicated hardware (pricing)
Baseten$0.10833/minute ($6.50/hour)per minuteDedicated deployment (pricing)

Together also sells raw H100s in GPU clusters at $3.99/hour on demand, which isn't the same product as its managed endpoint.

The idle-cost rules are where real bills diverge:

  • Replicate says private models bill for every period the instance is online, including setup, active processing and idle time while it waits for requests. Fast-booting fine-tunes are the exception and bill only while active. Replicate has also changed hands: it announced on November 17, 2025 that it is joining Cloudflare, saying "The API isn't changing" and that it continues as a distinct brand.
  • Modal stops charging once an app scales to zero containers, but by default a container stays alive for 60 seconds after its last input, and you pay for that. Plans add a monthly fee: Starter is $0 with $30 of credits, Team is $250 with $100 of credits. Since September 1, 2026, Starter's shared endpoint tokens bill from the first request rather than drawing on the credit.
  • Baseten lists the same per-minute GPU rates for dedicated inference and training.

Here's what Modal's model looks like in code, from its guide:

import modal

app = modal.App("example-inference")
image = modal.Image.debian_slim().uv_pip_install("transformers[torch]")

@app.function(gpu="h100", image=image)
def chat(prompt: str | None = None) -> list[dict]:
    ...

You run it with modal run path/to/file.py. No Dockerfile, no YAML. That's the appeal, and it's also why Modal is a better fit for batch jobs and custom pipelines than for "just give me an endpoint." See Modal vs Baseten and Hugging Face vs Replicate for more.

A decision rule

  • Prototype or spiky traffic on a popular open model: a token seller. Pick on price for your model and on the API parameters you need.
  • Latency is the product (voice, live agents): try Groq first, and measure on your own prompts rather than trusting anyone's chart, including theirs.
  • Steady, high volume on one model: do the maths on a dedicated endpoint. Divide the hourly price by the tokens you'd actually push through it in an hour; if that beats the per-token rate with room to spare, switch.
  • Fine-tuned or unusual models, or pipelines with pre- and post-processing: Modal, Replicate private models or Baseten dedicated.
  • Data must stay on your machine: none of the above. See running local LLMs for development.

Before any of that, count your tokens. Our token counter gives you the input side of the estimate, and our Groq vs OpenAI comparison shows what changes when you leave a frontier API.

AE

Written by AIDToolStack Editorial

Researched from official documentation, specifications and vendor pages, listed under Sources below. Last checked November 2026.

Sources

  1. console.groq.com/docs/openai
  2. docs.together.ai/docs/openai-api-compatibility
  3. docs.deepinfra.com/chat/overview
  4. console.groq.com/docs/models
  5. www.together.ai/pricing
  6. www.baseten.co/pricing/
  7. deepinfra.com/pricing
  8. groq.com/pricing
  9. groq.com/newsroom/groq-and-nvidia-enter-non-exclusive-inference-technology-licensing-agreement-to-accelerate-ai-inference-at-global-scale
  10. console.groq.com/docs/rate-limits
  11. modal.com/pricing
  12. replicate.com/pricing
  13. replicate.com/blog/replicate-cloudflare
  14. modal.com/docs/guide

Related Posts

Explore on AIDToolStack

Discover tools and servers featured in our directory.

MCP Servers

Trigger.dev Jobs MCP ServerManage Trigger.dev background jobs from your AI coding assistant. Monitor job runs, trigger tasks manually, inspect execution logs, manage schedules, and debug failed runs. The server provides operational visibility into your background job infrastructure — see which tasks are running, queued, or failed, and take action. Your AI can help diagnose job failures from execution logs, retry failed tasks, and optimize concurrency configurations for better throughput.Tailscale Network MCP ServerManage your Tailscale mesh VPN network from your AI coding assistant. View connected devices, manage ACLs, configure exit nodes, check device status, and manage DNS settings. The server provides visibility into your Tailscale network — see which machines are online, troubleshoot connectivity issues, manage network access policies, and configure routes. Useful for developers managing distributed infrastructure where Tailscale provides secure networking between services.Portainer Docker MCP ServerManage Docker environments through Portainer's API from your AI coding assistant. Control containers, images, networks, and volumes across multiple Docker hosts and Kubernetes clusters. The server provides tools for container lifecycle management, image pulling and building, stack deployment, and resource monitoring. Portainer's multi-environment management makes this server particularly valuable for teams managing containers across development, staging, and production environments.

Reviews

Sign in to leave a review.

No reviews yet. Be the first!

Comments

Sign in to leave a comment.

No comments yet. Be the first!

Sponsor Our Newsletter

Reach thousands of developers who are actively evaluating AI tools, MCP servers, and dev infrastructure. Our weekly newsletter goes to engaged technical decision-makers.

Growing subscriber baseHigh open rates

All sponsored content is clearly labeled per our editorial policy.