36 How API access works with ellmer

Calling LLMs programmatically via an API is a powerful step up from chat interfaces. First, it gives you control: chat software like ChatGPT or Copilot manages your interaction for you, quietly adding context (potentially from your chat history), so it’s hard to know exactly how the model is being prompted. With direct API access, you control exactly what goes to the LLM. Second, accessing it from R or Python code lets you automate calls — which is how we’ll build agents and literature-review tools later in this section.

36.1 What is an Application Programming Interface?

An API lets different software programs talk to each other. Large language models need serious compute to run, so rather than hosting one yourself (which is possible, but not what we’ll cover here) we’ll send prompts over the internet to a Provider — a server run by a company like OpenAI or Anthropic, or an aggregator like OpenRouter — and get the response back.

How an API call works:

Your code sends a request to the API endpoint (a web address).
The request includes your prompt, model choice, and parameters.
The provider’s service processes the request.
It sends back a structured response with the model’s output.
Your code uses that response however you like.

36.2 Your first API call

Model names change often. We’ll set them once here and reuse them for the rest of the section, so there’s only one place to update when a model is retired:

cheap_model <- "anthropic/claude-3.5-haiku"   # small, fast, inexpensive
strong_model <- "anthropic/claude-sonnet-4"   # more capable, more expensive

library(ellmer)

chat <- chat_openrouter(
    system_prompt = "You are a helpful assistant.",
    model = cheap_model,
    api_args = list(max_tokens = 50)
)
chat$chat("Ecologists like to eat ")

import requests, json, os
from dotenv import load_dotenv
load_dotenv()

api_key = os.getenv("OPENROUTER_API_KEY")
response = requests.post(
  url="https://openrouter.ai/api/v1/chat/completions",
  headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
  data=json.dumps({
    "model": "anthropic/claude-3.5-haiku",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Ecologists like to eat "}
    ]
  })
)
data = response.json()
data['choices'][0]['message']['content']

Notice the model doesn’t just complete your sentence — it answers as an assistant instead. That’s the system prompt at work: a stronger directive than your user prompt, which sets the overall behaviour of the chat. In most chat interfaces (like Copilot) you never see the system prompt, the provider sets it for you.

It is generally more effective to tell the LLM what to do rather than what not to do — same as with people.

Try overriding the default assistant behaviour with your own system prompt:

chat <- chat_openrouter(
    system_prompt = "Complete the sentences the user provides you. Continue from where the user left off. Provide one answer only. Don't provide any explanation, don't reiterate the text the user provides",
    model = cheap_model,
    api_args = list(max_tokens = 50)
)
chat$chat("Ecologists like to eat ")

36.3 Temperature and top_k

temperature controls the randomness of token predictions — lower (near 0) is more deterministic, higher is more creative and unpredictable. The valid range depends on the provider: Anthropic models accept 0–1, while some others go to 2, so check before you push it high. top_k restricts the model to choosing among a smaller set of likely next tokens; top_k = 1 gives the most predictable output.

chat_temp <- chat_openrouter(
    system_prompt = "Complete the sentences the user provides you. Continue from where the user left off. Provide one answer only.",
    model = cheap_model,
    api_args = list(max_tokens = 50, temperature = 0, top_k = 1)
)
chat_temp$chat("Marine ecologists like to eat ")

Try it again with temperature = 1 and no top_k (or a very high one). Low temperature/top_k gives you consistent, “safe” completions; higher values give more varied, sometimes less coherent ones.

36.4 Navigating OpenRouter

The Models page lists everything available — click a model to copy its id string for the model = argument. A single model may be served by multiple providers; OpenRouter picks one automatically, but its docs explain how to pin a specific provider (useful if you have data-security or ethical reasons to avoid particular ones). The docs page has the full parameter reference, usually with Python examples — an assistant can translate these to R for you if needed.

36.5 Comparing model complexity

anthropic/claude-3.5-haiku is a smaller and cheaper model than anthropic/claude-sonnet-4 — haiku is much cheaper (roughly 80c per million input tokens vs $3 for sonnet at time of writing) but less capable on nuanced tasks.

36.6 Understanding context windows

The context window is the amount of text (input plus output) a model can consider at once — typically 100-200K tokens, though some models now offer 1M+. We’ll come back to this once we start building longer prompts and agents.

36.7 API errors

Occasionally you’ll get an HTTP error instead of a response. A few common ones from OpenRouter:

400 — something’s wrong in your request.
401 — invalid API key.
402 — you’re out of credit.
408 — timeout, your request may be too large or your connection too slow.
429 — you’re being rate-limited (calling too much, too fast).

Full list of error codes here.

Challenge

Run the same completion prompt three times with temperature = 0, top_k = 1, then three times with temperature = 1 and no top_k. Compare how much the outputs vary between runs at each setting.