Add Rails engine for remote eval server

Appraisals

@@ -99,3 +99,12 @@ appraise "rails" do
   gem "activesupport", "~> 8.0"
   gem "railties", "~> 8.0"
 end
+
+# Rails engine testing for the eval server engine
+appraise "rails-server" do
+  gem "actionpack", "~> 8.0"
+  gem "railties", "~> 8.0"
+  gem "activesupport", "~> 8.0"
+  gem "rack", "~> 3.0"
+  gem "rack-test", "~> 2.1"
+end

README.md

@@ -392,7 +392,11 @@ See [trace_scoring.rb](./examples/eval/trace_scoring.rb) for a full example.
 
 ### Dev Server
 
-Run evaluations from the Braintrust web UI against code in your own application. Define evaluators, pass them to the dev server, and start serving:
+Run evaluations from the Braintrust web UI against code in your own application.
+
+#### Run as a Rack app
+
+Define evaluators, pass them to the dev server, and start serving:
 
 ```ruby
 # eval_server.ru
@@ -418,10 +422,21 @@ run Braintrust::Server::Rack.app(
 )
 ```
 
+Add your Rack server to your Gemfile:
+
+```ruby
+gem "rack"
+gem "puma" # recommended
+```
+
+Then start the server:
+
 ```bash
 bundle exec rackup eval_server.ru -p 8300 -o 0.0.0.0
 ```
 
+See example: [server/eval.ru](./examples/server/eval.ru)
+
 **Custom evaluators**
 
 Evaluators can also be defined as subclasses:
@@ -438,6 +453,51 @@ class FoodClassifier < Braintrust::Eval::Evaluator
 end
 ```
 
+#### Run as a Rails engine
+
+Use the Rails engine when your evaluators live inside an existing Rails app and you want to mount the Braintrust eval server into that application.
+
+Define each evaluator in its own file, for example under `app/evaluators/`:
+
+```ruby
+# app/evaluators/food_classifier.rb
+class FoodClassifier < Braintrust::Eval::Evaluator
+  def task
+    ->(input:) { classify(input) }
+  end
+
+  def scorers
+    [Braintrust::Scorer.new("exact_match") { |expected:, output:| output == expected ? 1.0 : 0.0 }]
+  end
+end
+```
+
+Then generate the Braintrust initializer:
+
+```bash
+bin/rails generate braintrust:eval_server
+```
+
+```ruby
+# config/routes.rb
+Rails.application.routes.draw do
+  mount Braintrust::Contrib::Rails::Engine, at: "/braintrust"
+end
+```
+
+The generator writes `config/initializers/braintrust_server.rb`, where you can review or customize the slug-to-evaluator mapping it discovers from `app/evaluators/**/*.rb` and `evaluators/**/*.rb`.
+
+See example: [contrib/rails/eval.rb](./examples/contrib/rails/eval.rb)
+
+**Developing locally**
+
+If you want to skip authentication on incoming eval requests while developing locally:
+
+- **For Rack**: Pass `auth: :none` to `Braintrust::Server::Rack.app(...)`
+- **For Rails**: Set `config.auth = :none` in `config/initializers/braintrust_server.rb`
+
+*NOTE: Setting `:none` disables authentication on incoming requests into your server; executing evals requires a `BRAINTRUST_API_KEY` to fetch resources.*
+
 **Supported web servers**
 
 The dev server requires the `rack` gem and a Rack-compatible web server.
@@ -449,14 +509,7 @@ The dev server requires the `rack` gem and a Rack-compatible web server.
 | [Passenger](https://www.phusionpassenger.com/) | 6.x               |                                      |
 | [WEBrick](https://github.com/ruby/webrick)     | Not supported     | Does not support server-sent events. |
 
-Add your chosen server to your Gemfile:
-
-```ruby
-gem "rack"
-gem "puma" # recommended
-```
-
-See example: [server/eval.ru](./examples/server/eval.ru)
+See examples: [server/eval.ru](./examples/server/eval.ru), 
 
 ## Documentation
 

examples/README.md

@@ -33,6 +33,7 @@ BRAINTRUST_DEBUG=true ruby examples/login/login_basic.rb
 ### Dev Server Examples
 
 - **`server/eval.ru`**: Set up a dev server for remote evals — define evaluators (subclass or inline) and serve them via a Rack app. Start with: `bundle exec appraisal server rackup examples/server/eval.ru -p 8300 -o 0.0.0.0`
+- **`contrib/rails/eval.rb`**: Mount the dev server as a Rails engine, define evaluator classes under `app/evaluators/`, and generate `config/initializers/braintrust_server.rb` with `bin/rails generate braintrust:server`
 
 ## Coming Soon
 

examples/contrib/rails/eval.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+# Braintrust Rails Engine — mount example
+#
+# This file shows one conventional setup for the Braintrust eval server in Rails:
+#   1. Define evaluator classes under app/evaluators/
+#   2. Generate the initializer with:
+#        bin/rails generate braintrust:server
+#   3. Mount the engine in config/routes.rb
+#
+# Requirements:
+#   gem 'actionpack', '~> 8.0'
+#   gem 'railties', '~> 8.0'
+#   gem 'activesupport', '~> 8.0'
+
+# ---------------------------------------------------------------------------
+# app/evaluators/my_classifier.rb
+# ---------------------------------------------------------------------------
+
+class MyClassifier < Braintrust::Eval::Evaluator
+  def task
+    ->(input:) { classify(input) }
+  end
+
+  def scorers
+    [Braintrust::Scorer.new("accuracy") { |expected:, output:| (output == expected) ? 1.0 : 0.0 }]
+  end
+end
+
+# ---------------------------------------------------------------------------
+# config/initializers/braintrust_server.rb
+# ---------------------------------------------------------------------------
+
+# Generated by: bin/rails generate braintrust:server
+#
+# require "braintrust/contrib/rails/server"
+#
+# Braintrust::Contrib::Rails::Server::Engine.configure do |config|
+#   config.evaluators = {
+#     "my-classifier" => MyClassifier.new
+#   }
+#
+#   # Default is :clerk_token. Use :none only for local development without
+#   # incoming request authentication. Outgoing Braintrust API calls still need
+#   # normal Braintrust credentials.
+#   config.auth = :clerk_token
+# end
+
+# ---------------------------------------------------------------------------
+# config/routes.rb
+# ---------------------------------------------------------------------------
+
+# Rails.application.routes.draw do
+#   mount Braintrust::Contrib::Rails::Server::Engine, at: "/braintrust"
+# end
+
+puts "Braintrust Rails Engine example — see comments for usage"

gemfiles/rails_server.gemfile

@@ -0,0 +1,14 @@
+# This file was generated by Appraisal
+
+source "https://rubygems.org"
+
+gem "minitest-reporters", "~> 1.6"
+gem "minitest-stub-const", "~> 0.6"
+gem "climate_control", "~> 1.2"
+gem "actionpack", "~> 8.0"
+gem "railties", "~> 8.0"
+gem "activesupport", "~> 8.0"
+gem "rack", "~> 3.0"
+gem "rack-test", "~> 2.1"
+
+gemspec path: "../"

lib/braintrust/contrib/rails/server.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+begin
+  require "action_controller"
+  require "rails/engine"
+rescue LoadError
+  raise LoadError,
+    "Rails (actionpack + railties) is required for the Braintrust Rails server engine. " \
+    "Add `gem 'rails'` or `gem 'actionpack'` and `gem 'railties'` to your Gemfile."
+end
+
+require "json"
+require_relative "../../eval"
+require_relative "../../server/sse"
+require_relative "../../server/auth/no_auth"
+require_relative "../../server/auth/clerk_token"
+require_relative "../../server/middleware/cors"
+require_relative "../../server/services/list_service"
+require_relative "../../server/services/eval_service"
+require_relative "server/engine"

lib/braintrust/contrib/rails/server/application_controller.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Braintrust
+  module Contrib
+    module Rails
+      module Server
+        class ApplicationController < ActionController::API
+          before_action :authenticate!
+
+          private
+
+          def authenticate!
+            auth_result = Engine.auth_strategy.authenticate(request.env)
+            unless auth_result
+              render json: {"error" => "Unauthorized"}, status: :unauthorized
+              return
+            end
+
+            request.env["braintrust.auth"] = auth_result
+            @braintrust_auth = auth_result
+          end
+
+          def parse_json_body
+            body = request.body.read
+            return nil if body.nil? || body.empty?
+            JSON.parse(body)
+          rescue JSON::ParserError
+            nil
+          end
+        end
+      end
+    end
+  end
+end

lib/braintrust/contrib/rails/server/engine.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+module Braintrust
+  module Contrib
+    module Rails
+      module Server
+        class Engine < ::Rails::Engine
+          isolate_namespace Braintrust::Contrib::Rails::Server
+
+          config.evaluators = {}
+          config.auth = :clerk_token
+
+          # Register the engine's routes file so Rails loads it during initialization.
+          paths["config/routes.rb"] << File.expand_path("routes.rb", __dir__)
+
+          initializer "braintrust.server.cors" do |app|
+            app.middleware.use Braintrust::Server::Middleware::Cors
+          end
+
+          # Class-level helpers that read from engine config.
+
+          def self.evaluators
+            config.evaluators
+          end
+
+          def self.auth_strategy
+            resolve_auth(config.auth)
+          end
+
+          def self.list_service
+            Braintrust::Server::Services::List.new(-> { config.evaluators })
+          end
+
+          # Long-lived so the state cache persists across requests.
+          def self.eval_service
+            @eval_service ||= Braintrust::Server::Services::Eval.new(-> { config.evaluators })
+          end
+
+          # Support the explicit `|config|` style used by this integration while
+          # still delegating zero-arity DSL blocks to Rails' native implementation.
+          def self.configure(&block)
+            return super if block&.arity == 0
+            yield config if block
+          end
+
+          def self.resolve_auth(auth)
+            case auth
+            when :none
+              Braintrust::Server::Auth::NoAuth.new
+            when :clerk_token
+              Braintrust::Server::Auth::ClerkToken.new
+            when Symbol, String
+              raise ArgumentError, "Unknown auth strategy #{auth.inspect}. Expected :none, :clerk_token, or an auth object."
+            else
+              auth
+            end
+          end
+          private_class_method :resolve_auth
+
+          generators do
+            require "braintrust/contrib/rails/server/generator"
+          end
+        end
+      end
+    end
+  end
+end
+
+require_relative "application_controller"
+require_relative "health_controller"
+require_relative "list_controller"
+require_relative "eval_controller"

lib/braintrust/contrib/rails/server/eval_controller.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module Braintrust
+  module Contrib
+    module Rails
+      module Server
+        class EvalController < ApplicationController
+          include ActionController::Live
+
+          def create
+            body = parse_json_body
+            unless body
+              render json: {"error" => "Invalid JSON body"}, status: :bad_request
+              return
+            end
+
+            result = Engine.eval_service.validate(body)
+            if result[:error]
+              render json: {"error" => result[:error]}, status: result[:status]
+              return
+            end
+
+            response.headers["Content-Type"] = "text/event-stream"
+            response.headers["Cache-Control"] = "no-cache"
+            response.headers["Connection"] = "keep-alive"
+
+            sse = Braintrust::Server::SSEWriter.new { |chunk| response.stream.write(chunk) }
+            Engine.eval_service.stream(result, auth: @braintrust_auth, sse: sse)
+          ensure
+            response.stream.close
+          end
+        end
+      end
+    end
+  end
+end