Improve workshop instructions and instrumentation examples

exercises/0-foundations/01-setup-braintrust-cli.md

@@ -1,28 +1,77 @@
-# 0.1 Set up the Braintrust CLI [CLI]
+# 0.1 Set up the local environment [CLI]
 
-## Task
+Before you instrument the app, give the Python SDK and the Braintrust CLI the
+credentials they need. The API key determines which Braintrust org receives
+your traces. The project is selected later in code.
 
-1. Install the `bt` CLI. See the
-   [CLI quickstart](https://www.braintrust.dev/docs/reference/cli/quickstart) for
-   the install command for your platform.
-2. Run setup:
-   First, `bt setup` will run through initializing the CLI. Authenticate with your Braintrust org and create a project for this course. It will also optionally ask to setup agent skills. It's highly recommended to set these up to allow coding agents to use Braintrust better.
-   ```bash
-   bt setup -i
-   ```
+## Step 1: Install the project dependencies
 
-3. Explore a couple of commands, for example:
+From the repository root, run:
 
-   ```bash
-   bt status
-   bt projects list
-   ```
+```bash
+uv sync
+```
 
-4. Ask a coding agent to use the CLI to fetch the latest logs from a Braintrust
-   project.
+This creates a local virtual environment and installs the app, Braintrust SDK,
+and CLI dependencies used in the exercises.
 
+## Step 2: Create your local environment file
 
-## Troubleshooting
-- If install isn't successful, ensure there aren't any network or VPN blockers preventing the install.
-- If `bt` command is unrecognized after install, ensure that it was added to your $PATH. This will vary by environment. Follow the instructions in the CLI after running install.
-- If authentication is not working, ensure you are provisioned access to Braintrust for your organization.
+Create a private `.env` file from the template:
+
+```bash
+cp example.env .env
+```
+
+Open `.env` and replace the placeholders. Quote an org name that contains a
+space.
+
+```dotenv
+BRAINTRUST_API_KEY=your-api-key
+BRAINTRUST_ORG_NAME="your-org-name"
+BASE_URL=https://gateway.braintrust.dev
+```
+
+`BRAINTRUST_API_KEY` authenticates both the SDK and the CLI. The SDK uses it to
+write traces, datasets, and evals. `BRAINTRUST_ORG_NAME` is sent to the
+Braintrust Gateway when the app makes a model call. Keep `.env` private. Do
+not put a real key in `example.env` or commit `.env`.
+
+## Step 3: Confirm that the CLI can reach your org
+
+Install the `bt` CLI if it is not already available. The
+[CLI quickstart](https://www.braintrust.dev/docs/reference/cli/quickstart)
+has the installation command for your platform.
+
+Then run this from the repository root:
+
+```bash
+bt projects list --env-file .env
+```
+
+You should see the projects in your Braintrust org. This confirms that the key
+in `.env` works. `bt setup` is not needed for this workshop because it manages
+separate CLI profiles and does not fill in the Python app's `.env` file.
+
+## Step 4: Create the course project
+
+Create a dedicated project for the workshop:
+
+```bash
+bt projects create learn-bt --env-file .env
+```
+
+If the project already exists, keep using it. In Exercise 1.1,
+`braintrust.init_logger(project="learn-bt")` sends the app's traces to this
+project.
+
+## Check your setup
+
+At this point, you should have all of the following:
+
+- A local `.venv` created by `uv sync`.
+- A private `.env` with a valid API key and org name.
+- A `learn-bt` project in your Braintrust org.
+
+If `bt projects list` returns an authentication error, create a new API key in
+Braintrust, replace the value in `.env`, and run the command again.

exercises/1-instrumentation/01-instrument-tracing.md

@@ -1,69 +1,246 @@
 # 1.1 Instrument the agent with Braintrust tracing
 
-The agent in `agent/` has no tracing yet. Add it three ways, seeding a few traces after
-each so you can see how the spans come across.
+The agent in `agent/` has no tracing yet. You will add it in four small stages
+and seed traces after each one. This makes it clear what each change adds.
 
-Seed traces with:
+`scripts.seed` is a traffic generator, not tracing code. For each run, it
+selects an account or opportunity fixture, asks an LLM to write a realistic
+account-executive request, and passes that request to `run_agent()`.
+
+## Step 1: Log model spans
+
+Open `agent/agent.py`. First, add the Braintrust imports alongside the existing
+third-party imports:
+
+```python
+import braintrust
+from braintrust import wrap_openai
+```
+
+Then initialize a logger immediately after `load_dotenv()`:
+
+```python
+load_dotenv()
+braintrust.init_logger(project="learn-bt")
+```
+
+The API key in `.env` selects your org. `project="learn-bt"` selects the
+project within that org where these spans are written.
+
+Finally, update both branches of `get_client()` so each `OpenAI` client is
+wrapped. Change this pattern:
+
+```python
+return OpenAI(
+    # existing arguments
+)
+```
+
+To this pattern:
+
+```python
+return wrap_openai(
+    OpenAI(
+        # keep the existing arguments unchanged
+    )
+)
+```
+
+`wrap_openai()` observes every `chat.completions.create()` call made through
+that client. You do not need to add logging around individual model calls.
+
+Run five seeded requests:
+
+```bash
+uv run python -m scripts.seed --count 5
+```
+
+Open **learn-bt**, then **Logs**, in Braintrust. Open one recent row. You
+should see an `llm` span named **Chat Completion** with model, token, latency,
+and cost information. At this stage, each LLM span is its own trace. You will
+give the whole agent run a root span in Step 3.
+
+![Standalone Chat Completion LLM span in Braintrust Logs](assets/01-log-model-spans.png)
+
+## Step 2: Observe automatic attachment capture
+
+Keep the code unchanged. Run the seed command again, this time with an
+attachment on every request:
 
 ```bash
-uv run python -m scripts.seed --count <int>
+uv run python -m scripts.seed --count 5 --attachment-ratio 1.0
+```
+
+The seeder asks an LLM to write a customer message, then alternates between a
+PDF and PNG version of that message. `run_agent()` sends the file to the model.
+Because the client is wrapped, Braintrust stores it as an `Attachment` on the
+nested `llm` span.
+
+In **Logs**, open a recent **Chat Completion** span. Its input should contain a
+previewable PDF or image attachment. The attachment represents forwarded customer context.
+
+![Chat Completion span showing an automatically captured PDF attachment](assets/02-automatic-attachment-capture.png)
+
+## Step 3: Add trace structure
+
+The wrapped client records model calls, but it does not know which model calls,
+tools, and helper functions belong to one agent run. Add `@traced` decorators
+to create that hierarchy.
+
+### Make `run_agent()` the root span
+
+In `agent/agent.py`, extend the Braintrust import:
+
+```python
+from braintrust import traced, wrap_openai
 ```
 
-## Task
+Then add this decorator directly above `run_agent()`:
+
+```python
+@traced(type="task", name="agent_run")
+def run_agent(...):
+```
 
-### 1. Instrument with the CLI and a coding agent
+One call to `run_agent()` is now one root `agent_run` span.
 
-Once agent skills are setup via the `bt setup` command from the previous exercise, we can use our coding agent directly to instrument this agent. 
+### Mark the agent tools
 
-Ask your agent of choice to instrument the Sales Assistant agent with Braintrust tracing.
+Open `agent/tools.py`. Add this import:
 
-Then seed some traces and confirm they appear in your project:
+```python
+from braintrust import traced
+```
+
+Add `@traced(type="tool")` directly above each of these functions:
+
+```python
+@traced(type="tool")
+def lookup_customer(...):
+
+@traced(type="tool")
+def get_opportunity(...):
+
+@traced(type="tool")
+def search_knowledge_base(...):
+
+@traced(type="tool")
+def draft_email(...):
+
+@traced(type="tool")
+def update_crm_record(...):
+```
+
+The type makes the purpose of each span clear in the trace UI.
+
+### Trace the fixture helpers
+
+Open `agent/fixtures.py`. Add this import:
+
+```python
+from braintrust import traced
+```
+
+Then add the basic decorator above both helper functions:
+
+```python
+@traced
+def find_accounts(...):
+
+@traced
+def search_docs(...):
+```
+
+For now, let the decorator capture each helper's normal input and output. You
+will take control of `search_docs()` data in the next step.
+
+Run the seed command again:
 
 ```bash
 uv run python -m scripts.seed --count 5
 ```
 
-Did it work? Open the logs (`bt view logs` or the Braintrust UI) and inspect a trace. 
-If the coding agent instrumented correctly, you should see the agent run as the root span, 
-with model and tool calls nested underneath, without having written any per-call tracing code yourself.
+In **Logs**, open one new trace. It should have an `agent_run` task root, with
+LLM, tool, and fixture spans nested underneath.
 
-Now, undo the coding agent's changes (discard changes from git history). We will now explore instrumenting tracing by hand, to understand how things work under the hood. With every exercise in this course, however, feel free to use a coding agent to solve the task. 
+- `agent_run` is the task root for one complete agent run.
+- `Chat Completion` span is nested LLM work.
+- `draft_email`, `update_crm_record`, and `lookup_customer`
+  appear as tool spans.
+- `find_accounts` or `search_docs` appears as a helper span
 
-### 2. Provider wrapping and traced functions
+![Agent run root span with nested LLM and tool spans](assets/03-agent-run-trace-structure.png)
 
-The Braintrust SDK provides wrapper integrations for most of the common agent frameworks
-and model providers. Our agent calls the OpenAI client directly, so wrap that client with
-`wrap_openai()` where it is built in `agent/agent.py`. You also need somewhere for the
-spans to go, so call `init_logger()` at the top level of the module.
+## Step 4: Log useful custom data
 
-Seed some traces and look at how they appear. Every model call the agent makes is now an
-`llm` span, with metrics automatically parsed.
+`@traced` captures function arguments and return values automatically. Use
+`current_span().log()` when you need an extra field for filtering or a smaller,
+safer representation of large data.
 
-The wrapper also captures attachments. Seed with `--attachment-ratio 1.0` and any file
-sent to the model, such as a PDF or image, is logged as an `Attachment` and previews in
-the trace, again with no extra code.
+### Mark runs that include attachments
 
-What this doesn't give us is the trace structure. An agent run is several different steps, and logically we want to associate these all with a single trace. A trace should be a unique agent run. We can trace all of the intermediate functions and tool calls via the `@traced` decorator. This decorator automatically captures input and output of the decorated function as a span, and nests the span in its proper trace heirarchy. Pass `type` and `name` parameters to the decorator to control how the span appears:
+In `agent/agent.py`, change the Braintrust import to include `current_span`:
 
-- `run_agent` in `agent/agent.py` is the root span of a run. Give it `type="task"` and
-  `name="agent_run"`.
-- each tool function in `agent/tools.py` gets `type="tool"`.
-- the business logic functions (`find_accounts()` and `search_docs()`) in `agent/fixtures.py`
+```python
+from braintrust import current_span, traced, wrap_openai
+```
+
+Inside `run_agent()`, find the existing `input_files` list. Immediately after
+it, add:
+
+```python
+current_span().log(metadata={"has_attachments": bool(attachments)})
+```
+
+This puts one boolean on the root `agent_run` span. You can later filter for
+file-bearing runs without opening each nested LLM span.
 
-Seed again. Spans nest by execution, so each trace should now be one `agent_run` root span
-with the model and tool calls underneath it, in the order the agent made them.
+### Trim the knowledge-base search output
 
-### 3. Log a custom span
+`search_docs()` returns full document bodies. The agent needs those bodies, but
+the trace only needs enough information to explain which documents matched.
+
+In `agent/fixtures.py`, change the import and decorator:
+
+```python
+from braintrust import current_span, traced
+
+@traced(notrace_io=True)
+def search_docs(query: str) -> list[dict]:
+```
+
+`notrace_io=True` turns off the decorator's automatic input and output capture
+for this function only. Just before `return hits`, add:
+
+```python
+current_span().log(
+    input={"query": query},
+    output={"num_hits": len(hits), "doc_ids": [d["id"] for d in hits]},
+    metadata={"matched_terms": terms},
+)
+```
+
+This preserves the query, matching terms, hit count, and document IDs without
+writing entire knowledge-base documents to the trace.
+
+Run the final seed command:
+
+```bash
+uv run python -m scripts.seed --count 5 --attachment-ratio 1.0
+```
 
-`@traced` auto captures a function's arguments and return value. Often This is helpful for most scenarios, but sometimes we want to control what gets logged more granularly, or add additional metdata to the sapn.
-an extra field, or less than the full input and output. We can achieve this via the `current_span().log()` method. This method manaully logs additional, arbitrary data on the currently active span. This gets merged with anything else that's already captured on the span.
+In a trace confirm both of the following:
 
-We want to be able to easily denote agent runs that work with attachments. In `run_agent()` log a metadata field called `has_attachments : bool` that is `True` if the agent run is working with attachments.
+1. The root `agent_run` span has `metadata.has_attachments`. It is `true` for runs with an attachment and `false` otherwise.
 
-We can also prevent `@traced` from auto capturing the input and output.
-`search_docs` currently returns the full matching documents, including their entire bodies, which is more than you want on the span. Pass `notrace_io=True` to its decorator and call `current_span().log()` to log a trimmed output instead, such as the number of hits and the matched document ids, along with the matched search terms as metadata. Seed some more traces and compare how `search_docs` now logs against the automatic capture on `find_accounts`.
+2. For one of the  `search_docs` child spans beneath, you should see:
 
+- Input with only the `query`.
+- Output with `num_hits` and `doc_ids`.
+- Metadata with `matched_terms`.
+- No full document `body`.
 
+![Custom search_docs span showing the trimmed query and result summary](assets/04-custom-search-docs-span.png)
 
 ## Solution