📄️ 🧪 What Are Evals?
As language models and agentic systems become more powerful, evaluating their behavior becomes increasingly critical.
📄️ 🧵 Introducing Steel Thread
Steel Thread is a lightweight, extensible framework for evaluating LLM agents — designed to help teams measure quality, catch regressions, and improve performance with minimal friction.
📄️ 🚀 Getting Started with Steel Thread
Steel Thread lets you evaluate your agents — both during development and in production — using real data, real metrics, and minimal boilerplate.
📄️ 📊 Evals
Evals are static datasets designed to be run multiple times against your agent.
📄️ 🆕 Adding Evals
💻 Through the UI
📄️ 🌊 Streams
Streams are a way to sample real plans and plan runs from your account allowing you to monitor the performance of your agents in production.
📄️ 🧮 Evaluators
Evaluators are responsible for the calculation of metrics.
📄️ 📈 Visualize Eval Results
Results from both Streams and Evals are pushed to the Portia UI for visualization. This allows you to quickly see trends over time and to drill into why metrics may have changed.