diff --git a/docs.json b/docs.json
index 2607d64..5473d85 100644
--- a/docs.json
+++ b/docs.json
@@ -98,7 +98,7 @@
"user-events/custom"
]
},
- "features/triggers",
+ "features/automations",
"features/embedded-analytics"
]
},
@@ -465,9 +465,9 @@
"group": "Traces",
"pages": [
"api-reference/traces/overview",
- "api-reference/traces/get-trace-details",
+ "api-reference/traces/get-trace",
"api-reference/traces/get-thread-details",
- "api-reference/traces/search-traces",
+ "api-reference/traces/search",
"api-reference/traces/create-public-trace-path",
"api-reference/traces/delete-public-trace-path"
]
@@ -504,9 +504,9 @@
]
},
{
- "group": "Triggers",
+ "group": "Automations",
"pages": [
- "api-reference/triggers/create-slack-trigger"
+ "api-reference/automations/create-slack-automation"
]
},
{
@@ -614,6 +614,11 @@
"tagId": "GTM-KJ4S6Z9C"
}
},
+ "analytics": {
+ "posthog": {
+ "apiKey": "phc_oOlj3H19T2JlGbFXmrGrjSLbDPDNyPKYdIFaTdrkXOY"
+ }
+ },
"redirects": [
{
"source": "/ci/cache/busted/1773936454",
diff --git a/introduction.mdx b/introduction.mdx
index 3ea3780..3cc27a0 100644
--- a/introduction.mdx
+++ b/introduction.mdx
@@ -1,6 +1,6 @@
---
title: "LangWatch: The Complete LLMOps Platform"
-description: "Ship AI agents 8x faster with comprehensive observability, evaluation, and prompt optimization. Open-source platform, with over 2.5k stars on GitHub."
+description: "Ship AI agents 8x faster with comprehensive observability, evaluation, and prompt optimization. Open-source platform, with over 3k stars on GitHub."
sidebarTitle: Introduction
keywords: langwatch, llm, ai, observability, evaluation, prompt optimization, llmops, open-source, github
---
diff --git a/llms-full.txt b/llms-full.txt
index b343200..8aebd68 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -4,7 +4,7 @@
---
title: "LangWatch: The Complete LLMOps Platform"
-description: "Ship AI agents 8x faster with comprehensive observability, evaluation, and prompt optimization. Open-source platform, with over 2.5k stars on GitHub."
+description: "Ship AI agents 8x faster with comprehensive observability, evaluation, and prompt optimization. Open-source platform, with over 3k stars on GitHub."
sidebarTitle: Introduction
keywords: langwatch, llm, ai, observability, evaluation, prompt optimization, llmops, open-source, github
---
@@ -17,61 +17,29 @@ keywords: langwatch, llm, ai, observability, evaluation, prompt optimization, ll
/>
-## Quick Start
-Ready to start taking control of your LLM application quality? Quick start with observability or agent simulations right away:
+## Quick Start
-
-
-
-
+
+ **Starting from scratch?** Use [Better Agents](/better-agents/overview) to scaffold a new agent project.
+ **Want to install skills?** Browse the [Skills Directory](/skills/directory).
+
+
## What is LangWatch?
LangWatch is the **open-source** LLMOps platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications. All platform features are natively integrated to accelerate the development workflow.
@@ -80,22 +48,14 @@ Building AI applications is hard. Developers spend weeks debugging issues, optim
LangWatch provides the missing operations platform for AI applications. Every LLM call, tool usage, and user interaction is automatically tracked with detailed traces, spans, and metadata. See the full conversation flow, identify bottlenecks, and understand exactly how your AI applications behave in production.
+## What LangWatch Does
-## For Every Role
-
-LangWatch serves different needs across your organization, providing value to every team member working with AI applications.
-
-### For Developers
-
-Debug faster with detailed traces that show exactly what happened in each LLM call. Build datasets from production data, run batch evaluations, and continuously improve your AI applications with comprehensive debugging tools and performance insights.
-
-### For Domain Experts
-
-Easily sift through conversations, see topics being discussed, and annotate messages for improvement in a collaborative manner with the development team. Provide feedback on AI outputs and help guide quality improvements through intuitive interfaces.
-
-### For Business Teams
-
-Track conversation metrics, user analytics, and cost tracking with custom dashboards and reporting. Monitor AI application performance, understand user behavior, and make data-driven decisions about your AI investments.
+
+
+
+
+
+
## Where to Start?
@@ -423,6 +383,7 @@ Use LangWatch to effortlessly integrate with popular AI model providers
No-code agent builders and tools
+
@@ -455,6 +416,10 @@ title: Quick Start
mode: "wide"
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch helps you understand every user interaction (**Thread**), each individual AI task (**Trace**), and all the underlying steps (**Span**) involved. We've made getting started super smooth.
Let's get cracking.
@@ -869,6 +834,10 @@ icon: python
keywords: langchain, instrumentation, callback, langwatch, python, tracing
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch integrates with Langchain to provide detailed observability into your chains, agents, LLM calls, and tool usage.
## Installation
@@ -951,6 +920,10 @@ description: Instrument Agno agents with LangWatch’s Python SDK to send traces
keywords: agno, openinference, langwatch, python, tracing, observability
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch integrates with Agno through OpenInference instrumentation to capture traces from your Agno agents automatically.
## Installation
@@ -2929,6 +2902,10 @@ icon: python
keywords: langgraph, instrumentation, callback, langwatch, python, tracing
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch integrates with LangGraph to provide detailed observability into your graph-based agents, LLM calls, and tool usage.
## Installation
@@ -3773,6 +3750,10 @@ icon: python
keywords: openai, instrumentation, autotrack, langwatch, python
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch integrates with OpenAI to automatically capture detailed information about your LLM calls.
## Installation
@@ -5601,6 +5582,10 @@ icon: square-js
keywords: langchain, instrumentation, callback, langwatch, typescript, tracing
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
LangWatch integrates with Langchain to provide detailed observability into your chains, agents, LLM calls, and tool usage.
## Installation
@@ -5670,6 +5655,10 @@ sidebarTitle: Vercel AI SDK
keywords: vercel ai sdk, langwatch, tracing, observability, vercel, ai, sdk
---
+
+ **Quick setup?** Instead of following these steps manually, [copy a prompt](/skills/code-prompts#instrument-my-code) into your coding agent and it will set this up for you automatically.
+
+
+
+
+
+Hit save, and that's it! Now every time a new message matches the filter, the automation will be fired and the dataset will be populated with the new row.
-```python
-import langwatch
+---
-result = langwatch.experiment.run("your-evaluation-slug")
-result.print_summary()
-```
+# FILE: ./datasets/overview.mdx
- ### TypeScript
+---
+title: Datasets
+sidebarTitle: Overview
+description: Create and manage datasets in LangWatch to build evaluation sets for LLMs and structured AI agent testing.
+---
-```typescript
-import { LangWatch } from "langwatch";
+
+
+## Adding data
- STATUS=$(echo $STATUS_RESPONSE | jq -r '.status')
- PROGRESS=$(echo $STATUS_RESPONSE | jq -r '.progress')
- TOTAL=$(echo $STATUS_RESPONSE | jq -r '.total')
+There are a couple ways to add data to a dataset;
- echo "Progress: $PROGRESS/$TOTAL"
+- **Manually**: You can add data on a per message basis.
+- **From traces**: You can fill the dataset by selecting a group of messages already captured.
+- **CSV Upload**: You can fill the dataset by uploading a CSV file.
+- **Continuously populate**: You can continuously populate the dataset with data traced from production.
- if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
- break
- fi
+### Manually
- sleep 2
-done
+To add data manually, click the "Add to Dataset" button on the messages page after selecting a message. You will then be able to choose the dataset type and preview the data that will be added.
-# Show summary and exit
-echo $STATUS_RESPONSE | jq '.summary'
+
+
+
-if [ "$STATUS" = "failed" ]; then
- exit 1
-fi
-```
+### From traces
+To add data by selecting a group, simply click the "Add to Dataset" button after choosing the desired messages in the table view. You'll then be able to select the type of dataset you wish to add to and preview the data that will be included.
+
+
+
-
-
-
-
-Hit save, and that's it! Now every time a new message matches the filter, the trigger will be fired and the dataset will be populated with the new row.
-
----
-
-# FILE: ./datasets/overview.mdx
-
----
-title: Datasets
-sidebarTitle: Overview
-description: Create and manage datasets in LangWatch to build evaluation sets for LLMs and structured AI agent testing.
----
-
-## Create datasets
-
-LangWatch allows you to create and manage datasets, with a built-in excel-like interface for collaborating with your team.
-
-* Import datasets in any format you want, manage columns and data types
-* Keep populating the dataset with data traced from production
-* Create new datasets from scratch with AI assistance
-* Generate synthetic data from documents
-* Import, export and manage versions
-
-### Usage
-
-To create a dataset, simply go to the datasets page and click the "Upload or Create Dataset" button. You will be able to select the type of dataset you want as well as the columns you want to include.
-
-
-
+ **Let your agent set this up.** [Copy the evaluations prompt](/skills/code-prompts#set-up-evaluations) into your coding agent to get started automatically.
+
+
Online evaluation lets you continuously score your LLM's production traffic. Unlike [experiments](/evaluations/experiments/overview) which test before deployment, online evaluation monitors your live application to catch quality issues, detect regressions, and ensure safety.
+
+
+In this example, a virtual try-on pipeline is evaluated with three image columns:
+- **original** → mapped to `contexts` (the person's photo)
+- **request** → mapped to `input` (the clothing item)
+- **generated** → mapped to `output` (the try-on result)
+
+The LLM-as-a-Judge prompt instructs the model to evaluate all three images and score the quality of the generated output.
+
+### Step 3: Run and Iterate
+
+Click the **play button** to run the evaluator. The model receives all images as vision content and returns structured results (score, pass/fail, or category) with detailed reasoning.
+
+Use this workflow to **iterate on your evaluator prompt** until you have reliable evaluation criteria, then save it for reuse across experiments and CI/CD pipelines.
+
+## Custom Workflow Evaluators for Complex Logic
+
+For more advanced evaluation pipelines, you can create a **Custom Workflow Evaluator** in the Evaluators page. This gives you a visual workflow builder where you can chain multiple LLM nodes, add image variables to prompts, and build multi-step evaluation logic.
+
+
+ 
+
+
+In the workflow builder:
+1. Add **image-typed variables** to your prompt node inputs
+2. Use `{{ "{{variable_name}}" }}` syntax to reference images in the prompt template
+3. Map dataset columns to the image variables in the entry node
+4. The workflow handles multipart content assembly automatically
+
+This is useful when you need to split evaluation into multiple steps, use different models for different aspects, or combine vision evaluation with text-based checks.
+
+## Evaluating Images via SDK
+
+For programmatic evaluation from notebooks or CI/CD, use the Python or TypeScript SDK with a [saved evaluator](/evaluations/evaluators/saved-evaluators).
+
+### Using a Saved Evaluator
+
+After iterating on your evaluator in the UI, save it and call it from code:
+
+
-
+{/* TODO: Add screenshot of saved evaluator creation UI */}
### Via the Evaluators Page
@@ -23715,7 +23664,7 @@ LangWatch is composed of a few core components:
- **LangWatch App**: The main application and user interface for the platform.
- **LangEvals**: The evaluation service for LangWatch built-in evaluators.
- **LangWatch NLP**: The service the runs workflows and custom built evaluators in the platform.
-- **LangWatch Workers**: The background workers running inside the app, processing traces, triggering alerts, real-time evaluations and so on.
+- **LangWatch Workers**: Background workers that process traces, trigger alerts, run real-time evaluations and so on. By default, workers are deployed as a separate pod for independent scaling.
- **Redis**: A redis queue that coordinates and add resiliency for the traces to be processed.
- **Elasticsearch/OpenSearch**: The main database for the platform, storing traces and evaluation results.
- **PostgreSQL**: The core database for the platform, storing user accounts, teams, and other metadata.
@@ -23906,7 +23855,7 @@ The primary LangWatch application that provides the web interface and core funct
- API endpoints for trace collection and management
- User authentication and session management
- Dashboard and analytics interface
-- Background workers for processing traces and evaluations
+- Background workers for processing traces and evaluations (deployed as a separate pod in Helm by default)
**Key Features:**
- Trace collection and storage
@@ -24110,8 +24059,16 @@ app:
secretKeyRef:
name: langwatch-secrets
key: metricsApiKey
- sentry:
- enabled: false
+
+workers:
+ replicaCount: 2
+ resources:
+ requests:
+ cpu: 500m
+ memory: 2Gi
+ limits:
+ cpu: 2000m
+ memory: 4Gi
ingress:
enabled: true
@@ -24188,8 +24145,6 @@ app:
telemetry:
metrics:
enabled: false
- sentry:
- enabled: false
resources:
requests:
cpu: 250m
@@ -24283,6 +24238,16 @@ app:
name: langwatch-secrets
key: metricsApiKey
+workers:
+ replicaCount: 2
+ resources:
+ requests:
+ cpu: 500m
+ memory: 2Gi
+ limits:
+ cpu: 2000m
+ memory: 4Gi
+
ingress:
enabled: true
className: nginx
@@ -24546,7 +24511,7 @@ If you want to use SSO with our Cloud version, contact our team with the require
| Configuration | Value |
| ------------------ | ------------------------------------------------------------------------------------------- |
-| Required Variables | `AZURE_AD_CLIENT_ID`