I9 · Generative AI and Large Language Models in Practice

Using an LLM API from Python

An LLM API is a web service that lets programs send a request to a hosted model and receive its reply. A typical request includes the model name, an optional system prompt, a list of messages with roles (user and assistant), and settings such as maximum output tokens and temperature. The response contains the generated text, the reason generation stopped and token usage for billing. Providers offer official Python libraries that handle authentication with a secret API key, retries and streaming of partial output.

15 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A chat website is one way to use an LLM. A few lines of Python let you use the same model inside your own app: grading 500 essays overnight, summarising every complaint as it arrives or answering students in a school portal.

An LLM API is a web service that lets programs send a request to a hosted model and receive its reply. A typical request includes the model name, an optional system prompt, a list of messages with roles (user and assistant), and settings such as maximum output tokens and temperature. The response contains the generated text, the reason generation stopped and token usage for billing. Providers offer official Python libraries that handle authentication with a secret API key, retries and streaming of partial output.

In simple words

Using an API is like ordering food through a delivery app's kitchen counter: you send a clear order slip (the request), the kitchen cooks (the model generates), and a parcel comes back with the food and the bill (the reply and token usage).

Overview

  • An LLM API lets a program send messages to a model and receive replies over the internet.
  • Requests include the model name, a list of messages and settings such as temperature and max_tokens.
  • Official Python SDKs wrap the HTTP calls, for example client.messages.create(...).
  • API keys must be kept secret, in environment variables, never in code or notebooks shared publicly.
  • Responses report token usage, which determines cost.
  • Robust code handles timeouts, rate limits and errors with retries.

Why it matters

APIs turn LLMs into building blocks for real products. Knowing how requests, roles, limits, errors and costs work lets you build reliable tools, and lets you switch providers or models when needed.

Key terms

API key
A secret string that identifies your account; anyone with it can spend your credit.
Messages and roles
The conversation as a list of turns marked user or assistant, plus a system prompt.
Max tokens
The upper limit on how many tokens the reply may contain.
Streaming
Receiving the reply piece by piece as it is generated, so users see text sooner.
Rate limit
The maximum number of requests or tokens allowed per minute.
Exponential backoff
Retrying after a failure with waits that double each time: 1, 2, 4, 8 seconds.

Types

LLM API calls come in several types, each suited to a different kind of application.

1. Single-turn completion calls

Sending one prompt and receiving one reply.

  • How it works: The message list contains a system instruction and one user message; the reply text is read from the response object.
  • Example: Summarising each of 500 customer reviews in one sentence.
  • When to use it: Use it for independent tasks such as classification, extraction and summarisation.

2. Multi-turn chat calls

Sending the whole conversation history with each request.

  • How it works: The API is stateless, so the program appends each user and assistant message to a list and resends it, trimming or summarising when it grows long.
  • Example: A tutoring chatbot remembering earlier questions within a session.
  • When to use it: Use it for conversations, keeping an eye on context length and cost.

3. Streaming responses

Receiving the reply token by token as it is generated.

  • How it works: The SDK yields text chunks over a stream, so the interface can show words immediately.
  • Example: A chat app showing the answer appear word by word like typing.
  • When to use it: Use it for interactive apps where waiting for the full reply would feel slow.

4. Tool-use and structured calls

Calls that define tools or a response schema.

  • How it works: The program describes functions; when the model asks to use one, the program runs it and sends the result back in a follow-up call.
  • Example: An assistant looking up live train status through a railway API before answering.
  • When to use it: Use it to give models accurate, current information and safe ways to act.

5. Batch and asynchronous calls

Sending many requests together, often at lower cost, or concurrently from asynchronous code.

  • How it works: Batch APIs process large jobs within hours at a discount; asyncio sends many requests in parallel within rate limits.
  • Example: Classifying 100,000 support tickets overnight through a batch job.
  • When to use it: Use them for large offline workloads where an immediate answer is not needed.

Explained in detail

Anatomy of a request and response

With an official Python library, a call looks roughly like this: create a client (it reads the API key from an environment variable), then call a create method with the model name, max_tokens, an optional system string and messages=[{"role": "user", "content": "Summarise this complaint..."}]. The response object holds the reply text, a stop reason, such as end of turn or hitting the token limit, and usage counts for input and output tokens. Exact parameter names differ slightly between providers, so always check the current documentation, and pin the model version you tested so that behaviour does not change unexpectedly.

PartPurpose
modelWhich model to use; newer and larger models cost more
systemStanding instructions for role, rules and format
messagesThe conversation so far, alternating user and assistant
max_tokensCaps the length and cost of the reply
temperatureControls randomness
usage (in response)Input and output tokens used, for billing and monitoring

Writing robust code

Real applications must handle failure. Networks drop, servers are briefly overloaded and rate limits are hit when many requests arrive at once. Wrap calls in error handling, retry only errors that are temporary (rate limits, timeouts, server errors) using exponential backoff with a little random jitter, and give up after a few attempts with a clear message to the user. Do not retry errors caused by a bad request, such as an invalid parameter, because they will fail again. Log each request's token usage and latency so you can spot cost spikes. For long replies, stream the output so users see text appear immediately. For large offline jobs, many providers offer batch APIs at a lower price that return results within hours.

Keeping keys and data safe

  • Store the API key in an environment variable or secret manager, never in code, notebooks you share or a public GitHub repository.
  • Call the API from your server, not directly from a web page, where anyone could read the key.
  • Set monthly spending limits in the provider's console, and rotate a key immediately if it leaks.
  • Read the provider's data-use terms before sending personal or confidential information.

How it works

  1. Create an account, generate an API key and store it as an environment variable.
  2. Install the provider's Python library and make a first small request.
  3. Add a system prompt, settings and a max-token limit suited to the task.
  4. Handle errors with retries and backoff, and log usage and latency.
  5. Test on your evaluation set, then deploy behind your own server.

In code

api_client.py
import json
import random
import time

class FakeLLMClient:
    """Stands in for a real provider's client so this example runs offline."""
    def __init__(self):
        self.calls = 0
    def create(self, model, max_tokens, system, messages):
        self.calls += 1
        if self.calls <= 2:                              # the first two calls hit a rate limit
            raise RuntimeError("429 rate limit")
        text = "Category: water. Summary: no tap water in Ward 12 for 3 days."
        return {"content": text, "stop_reason": "end_turn",
                "usage": {"input_tokens": 42 + len(system.split()), "output_tokens": len(text.split())}}

def call_with_retry(client, max_attempts=4, **request):
    for attempt in range(max_attempts):
        try:
            return client.create(**request)
        except RuntimeError as err:
            if "429" not in str(err) and "503" not in str(err):
                raise                                     # do not retry bad requests
            wait = (2 ** attempt) * 0.01 + random.random() * 0.005   # tiny waits for the demo
            print(f"attempt {attempt + 1} failed ({err}); retrying in {wait:.3f}s")
            time.sleep(wait)
    raise RuntimeError("gave up after retries")

request = {
    "model": "example-model",
    "max_tokens": 100,
    "system": "You triage municipal complaints. Reply with a category and a one-line summary.",
    "messages": [{"role": "user", "content": "Ward 12 mein teen din se nal mein paani nahi aaya"}],
}
print("Request:", json.dumps(request, ensure_ascii=False)[:90], "...")
reply = call_with_retry(FakeLLMClient(), **request)
print("Reply:", reply["content"])
u = reply["usage"]
print(f"Tokens in/out: {u['input_tokens']}/{u['output_tokens']}  cost at $3/$15 per million: ${(u['input_tokens'] * 3 + u['output_tokens'] * 15) / 1e6:.6f}")

Output

Request: {"model": "example-model", "max_tokens": 100, "system": "You triage municipal complaints.  ...
attempt 1 failed (429 rate limit); retrying in 0.014s
attempt 2 failed (429 rate limit); retrying in 0.023s
Reply: Category: water. Summary: no tap water in Ward 12 for 3 days.
Tokens in/out: 54/12  cost at $3/$15 per million: $0.000342

Try it yourself

Grading 500 essays overnight

A teacher uses an LLM API to give first-draft feedback on 500 English essays.

Retry or not?

Should the code retry after each error?

  • Network timeout

  • 503 server overloaded

  • 429 rate limit exceeded

  • Invalid parameter: max_tokens must be positive

  • 401 invalid API key

Common mistakes

  • Putting the API key in code or in a web page.
  • Retrying every error immediately in a tight loop.
  • Not setting max_tokens or spending limits.

Check yourself

Question 1

1. Where should an API key be stored?

Question 2

2. What does exponential backoff do?

Question 3

3. Why stream responses in a chat app?

Summary

  • An LLM API takes a model name, system prompt, messages and settings, and returns text plus usage.
  • Handle temporary errors with backoff, log tokens and latency, and stream long replies.
  • Protect keys on a server, set spending limits and check data-use terms.

Finished reading?

Sources: Anthropic API documentation: Messages · OpenAI API documentation: text generation