Skip to main content

AI providers overview

Toby uses a large language model for chat, summarization, organization, and background schedules. You configure one or more AI providers in Toby.app → Settings → AI, then pick a provider and model per persona.

Toby supports five AI providers:

ProviderBest for
OpenAI (direct)A single OpenAI API key and familiar model ids (gpt-5-mini, gpt-4.1, …)
Vercel AI GatewayOne API key that routes to OpenAI, Anthropic, Google, Amazon, and more via provider/model slugs
ChutesOpen-source TEE-backed models (DeepSeek, Qwen, GLM, Kimi) via Chutes' OpenAI-compatible endpoint
OpenRouterHundreds of models from many vendors through a single OpenRouter API key
OllamaRun open-source models locally on your machine (no API key needed)

You only need one provider configured to start chatting. New installs should start with Vercel AI Gateway (guided setup on the Dashboard checklist). Use OpenAI direct for a single-vendor key, Chutes or OpenRouter for open-source catalogs, or Ollama for fully local inference.

Quick setup​

Recommended: Dashboard onboarding → Configure AI provider → Connect opens a chooser for guided setup:

  • Vercel AI Gateway (recommended) — multi-model chat, free credits, web search & transcription catalogs
  • OpenRouter — hundreds of models through one key

Or open Toby.app → Settings:

  1. Open AI and enter credentials for at least one provider (Vercel AI Gateway, OpenAI, Chutes, OpenRouter, or an Ollama base URL).
  2. Open Personas, choose AI Provider and Model (the guided Vercel flow configures the built-in Toby persona for you).

Credentials are stored in ~/.toby/credentials.json on your Mac (encrypted at rest; see Configuration overview).

Choosing a provider​

If you want…Consider
Fastest full Toby setup (chat + web search + multi-vendor models)Vercel AI Gateway (recommended)
The simplest path with OpenAI models onlyOpenAI (direct)
Open-source TEE-backed models with privacy guaranteesChutes
The widest selection of models from many vendorsOpenRouter
Fully local, private inference with no API costsOllama
The same model on either pathGateway slug openai/gpt-5-mini ≈ direct id gpt-5-mini

Model ids differ by provider:

  • OpenAI (direct) — bare ids, e.g. gpt-5-mini
  • Vercel AI Gateway — provider/model slugs, e.g. anthropic/claude-sonnet-4.6
  • Chutes — slash-delimited ids, e.g. deepseek-ai/DeepSeek-V3.2-TEE
  • OpenRouter — slash-delimited ids, e.g. openai/gpt-5.6-luna
  • Ollama — local model names, e.g. llama3.2

After you add credentials, persona and Settings model menus load an up-to-date list from that provider. Chutes and OpenRouter expose a public model catalog, so their model lists are fetched live even before you configure an API key. Until a provider is configured or its catalog is reachable, Toby shows a short curated default list.

Context window display​

In chat, Toby may show context usage as ctx N% when it knows the selected model family’s context window. The estimate supports common OpenAI and gateway-routed model families, including Anthropic, Gemini, Nova, Llama, Mistral, DeepSeek, Grok, GLM, and Kimi models.

Context compaction​

When a long chat approaches the model’s context limit, Toby can automatically compact the model’s history so the next turn still fits. It prefers cheap reclaim first: clamp runaway blobs, drop superseded re-reads of the same resource, then blank older tool dumps while keeping recent results. Your messages stay intact; the assistant can re-fetch tools if needed. The visible chat transcript is not rewritten. Compaction may briefly reduce prompt-cache reuse for that session after a rewrite.

Auxiliary models​

Pretreatment, planning, and other lightweight LLM steps use a smaller auxiliary model tied to the active persona’s provider:

  • OpenAI direct: gpt-4.1-nano by default
  • Vercel AI Gateway: openai/gpt-4.1-nano by default
  • Chutes: deepseek-ai/DeepSeek-V3.2-TEE by default
  • OpenRouter: openai/gpt-5.6-luna by default
  • Ollama: llama3.2 by default

Override with TOBY_PRETREAT_MODEL (bare id for OpenAI, or a full provider-specific model id). Set TOBY_PRETREAT_DELTA=0 to disable follow-up delta pretreatment.

Default integration providers​

Separate from AI: in Toby.app → Settings → Default Providers, you pick which connected integration Toby prefers per category (email, calendar, tasks, and so on). That is unrelated to which LLM vendor you use. See Default providers.