Definition
A chat website is one way to use an LLM. A few lines of Python let you use the same model inside your own app: grading 500 essays overnight, summarising every complaint as it arrives or answering students in a school portal.
An LLM API is a web service that lets programs send a request to a hosted model and receive its reply. A typical request includes the model name, an optional system prompt, a list of messages with roles (user and assistant), and settings such as maximum output tokens and temperature. The response contains the generated text, the reason generation stopped and token usage for billing. Providers offer official Python libraries that handle authentication with a secret API key, retries and streaming of partial output.
In simple words
Using an API is like ordering food through a delivery app's kitchen counter: you send a clear order slip (the request), the kitchen cooks (the model generates), and a parcel comes back with the food and the bill (the reply and token usage).
Overview
- An LLM API lets a program send messages to a model and receive replies over the internet.
- Requests include the model name, a list of messages and settings such as temperature and max_tokens.
- Official Python SDKs wrap the HTTP calls, for example client.messages.create(...).
- API keys must be kept secret, in environment variables, never in code or notebooks shared publicly.
- Responses report token usage, which determines cost.
- Robust code handles timeouts, rate limits and errors with retries.
Why it matters
APIs turn LLMs into building blocks for real products. Knowing how requests, roles, limits, errors and costs work lets you build reliable tools, and lets you switch providers or models when needed.
Key terms
- API key
- A secret string that identifies your account; anyone with it can spend your credit.
- Messages and roles
- The conversation as a list of turns marked user or assistant, plus a system prompt.
- Max tokens
- The upper limit on how many tokens the reply may contain.
- Streaming
- Receiving the reply piece by piece as it is generated, so users see text sooner.
- Rate limit
- The maximum number of requests or tokens allowed per minute.
- Exponential backoff
- Retrying after a failure with waits that double each time: 1, 2, 4, 8 seconds.
Types
LLM API calls come in several types, each suited to a different kind of application.
1. Single-turn completion calls
Sending one prompt and receiving one reply.
- How it works: The message list contains a system instruction and one user message; the reply text is read from the response object.
- Example: Summarising each of 500 customer reviews in one sentence.
- When to use it: Use it for independent tasks such as classification, extraction and summarisation.
2. Multi-turn chat calls
Sending the whole conversation history with each request.
- How it works: The API is stateless, so the program appends each user and assistant message to a list and resends it, trimming or summarising when it grows long.
- Example: A tutoring chatbot remembering earlier questions within a session.
- When to use it: Use it for conversations, keeping an eye on context length and cost.
3. Streaming responses
Receiving the reply token by token as it is generated.
- How it works: The SDK yields text chunks over a stream, so the interface can show words immediately.
- Example: A chat app showing the answer appear word by word like typing.
- When to use it: Use it for interactive apps where waiting for the full reply would feel slow.
4. Tool-use and structured calls
Calls that define tools or a response schema.
- How it works: The program describes functions; when the model asks to use one, the program runs it and sends the result back in a follow-up call.
- Example: An assistant looking up live train status through a railway API before answering.
- When to use it: Use it to give models accurate, current information and safe ways to act.
5. Batch and asynchronous calls
Sending many requests together, often at lower cost, or concurrently from asynchronous code.
- How it works: Batch APIs process large jobs within hours at a discount; asyncio sends many requests in parallel within rate limits.
- Example: Classifying 100,000 support tickets overnight through a batch job.
- When to use it: Use them for large offline workloads where an immediate answer is not needed.
Explained in detail
Anatomy of a request and response
With an official Python library, a call looks roughly like this: create a client (it reads the API key from an environment variable), then call a create method with the model name, max_tokens, an optional system string and messages=[{"role": "user", "content": "Summarise this complaint..."}]. The response object holds the reply text, a stop reason, such as end of turn or hitting the token limit, and usage counts for input and output tokens. Exact parameter names differ slightly between providers, so always check the current documentation, and pin the model version you tested so that behaviour does not change unexpectedly.
| Part | Purpose |
|---|---|
| model | Which model to use; newer and larger models cost more |
| system | Standing instructions for role, rules and format |
| messages | The conversation so far, alternating user and assistant |
| max_tokens | Caps the length and cost of the reply |
| temperature | Controls randomness |
| usage (in response) | Input and output tokens used, for billing and monitoring |
Writing robust code
Real applications must handle failure. Networks drop, servers are briefly overloaded and rate limits are hit when many requests arrive at once. Wrap calls in error handling, retry only errors that are temporary (rate limits, timeouts, server errors) using exponential backoff with a little random jitter, and give up after a few attempts with a clear message to the user. Do not retry errors caused by a bad request, such as an invalid parameter, because they will fail again. Log each request's token usage and latency so you can spot cost spikes. For long replies, stream the output so users see text appear immediately. For large offline jobs, many providers offer batch APIs at a lower price that return results within hours.
Keeping keys and data safe
- Store the API key in an environment variable or secret manager, never in code, notebooks you share or a public GitHub repository.
- Call the API from your server, not directly from a web page, where anyone could read the key.
- Set monthly spending limits in the provider's console, and rotate a key immediately if it leaks.
- Read the provider's data-use terms before sending personal or confidential information.
How it works
- Create an account, generate an API key and store it as an environment variable.
- Install the provider's Python library and make a first small request.
- Add a system prompt, settings and a max-token limit suited to the task.
- Handle errors with retries and backoff, and log usage and latency.
- Test on your evaluation set, then deploy behind your own server.
In code
import json
import random
import time
class FakeLLMClient:
"""Stands in for a real provider's client so this example runs offline."""
def __init__(self):
self.calls = 0
def create(self, model, max_tokens, system, messages):
self.calls += 1
if self.calls <= 2: # the first two calls hit a rate limit
raise RuntimeError("429 rate limit")
text = "Category: water. Summary: no tap water in Ward 12 for 3 days."
return {"content": text, "stop_reason": "end_turn",
"usage": {"input_tokens": 42 + len(system.split()), "output_tokens": len(text.split())}}
def call_with_retry(client, max_attempts=4, **request):
for attempt in range(max_attempts):
try:
return client.create(**request)
except RuntimeError as err:
if "429" not in str(err) and "503" not in str(err):
raise # do not retry bad requests
wait = (2 ** attempt) * 0.01 + random.random() * 0.005 # tiny waits for the demo
print(f"attempt {attempt + 1} failed ({err}); retrying in {wait:.3f}s")
time.sleep(wait)
raise RuntimeError("gave up after retries")
request = {
"model": "example-model",
"max_tokens": 100,
"system": "You triage municipal complaints. Reply with a category and a one-line summary.",
"messages": [{"role": "user", "content": "Ward 12 mein teen din se nal mein paani nahi aaya"}],
}
print("Request:", json.dumps(request, ensure_ascii=False)[:90], "...")
reply = call_with_retry(FakeLLMClient(), **request)
print("Reply:", reply["content"])
u = reply["usage"]
print(f"Tokens in/out: {u['input_tokens']}/{u['output_tokens']} cost at $3/$15 per million: ${(u['input_tokens'] * 3 + u['output_tokens'] * 15) / 1e6:.6f}")Output
Request: {"model": "example-model", "max_tokens": 100, "system": "You triage municipal complaints. ...
attempt 1 failed (429 rate limit); retrying in 0.014s
attempt 2 failed (429 rate limit); retrying in 0.023s
Reply: Category: water. Summary: no tap water in Ward 12 for 3 days.
Tokens in/out: 54/12 cost at $3/$15 per million: $0.000342Try it yourself
Grading 500 essays overnight
A teacher uses an LLM API to give first-draft feedback on 500 English essays.
Retry or not?
Should the code retry after each error?
Network timeout
503 server overloaded
429 rate limit exceeded
Invalid parameter: max_tokens must be positive
401 invalid API key
Common mistakes
- Putting the API key in code or in a web page.
- Retrying every error immediately in a tight loop.
- Not setting max_tokens or spending limits.
Check yourself
Summary
- An LLM API takes a model name, system prompt, messages and settings, and returns text plus usage.
- Handle temporary errors with backoff, log tokens and latency, and stream long replies.
- Protect keys on a server, set spending limits and check data-use terms.
Finished reading?
Sources: Anthropic API documentation: Messages · OpenAI API documentation: text generation