.NET: Support for Foundry Observability and Evaluations

[rogerbarreto]

Part of #3440

Support for Foundry Observability and Evaluations. Review existing samples and verify everything is working.

[rogerbarreto]

Subplan: #3675 - Support for Foundry Observability and Evaluations

Issue: #3675
Parent Issue: #3440
Scope: .NET only (parity with Python samples)
Deliverable: Sample files demonstrating Observability and Evaluation features

---

Objective

Create .NET samples demonstrating Foundry Observability and Evaluation capabilities, achieving parity with existing Python samples. This includes verifying existing observability (Step07) works correctly and adding new evaluation samples.

---

Current State Analysis

Existing .NET Samples

Feature	.NET Status	Sample
Observability/Telemetry	✅ Exists	FoundryAgents_Step07_Observability

Missing from .NET (Exists in Python)

Feature	Python Sample	Status
Red Teaming Evaluation	evaluation/red_teaming/red_team_agent_sample.py	❌ Missing
Self Reflection Evaluation	evaluation/self_reflection/self_reflection.py	❌ Missing

---

Workplan

Phase 1: Verify Existing Observability Sample

• Task 1.1: Review existing FoundryAgents_Step07_Observability sample
  • Verify it builds and runs successfully
  • Check telemetry data is being generated
  • Document current capabilities
• Task 1.2: Compare with Python observability patterns
  • Identify any gaps in functionality
  • Check for Azure Monitor integration
• Task 1.3: Update Step07 if needed
  • Ensure parity with Python approach
  • Add any missing telemetry types (if applicable)

Phase 2: Red Teaming Evaluation Sample

• Task 2.1: Analyze Python red teaming sample structure
  • Review red_team_agent_sample.py
  • Review README.md for setup requirements
  • Understand Azure AI Evaluation SDK dependencies
• Task 2.2: Research .NET equivalents
  • Check for Azure.AI.Evaluation NuGet package
  • Identify available .NET evaluation APIs
• Task 2.3: Create evaluation sample directory structure

  dotnet/samples/GettingStarted/Evaluation/
  ├── Evaluation_Step01_RedTeaming/
  │   ├── Evaluation_Step01_RedTeaming.csproj
  │   ├── Program.cs
  │   └── README.md
  

• Task 2.4: Implement Red Teaming sample
  • Configure red team evaluator
  • Set up target callback (FoundryAgent integration)
  • Run evaluation against attack prompts
  • Output evaluation results

Phase 3: Self Reflection Evaluation Sample

• Task 3.1: Analyze Python self-reflection sample
  • Review self_reflection.py
  • Review resources/ folder contents
  • Understand evaluation criteria used
• Task 3.2: Create Self Reflection sample

  dotnet/samples/GettingStarted/Evaluation/
  └── Evaluation_Step02_SelfReflection/
      ├── Evaluation_Step02_SelfReflection.csproj
      ├── Program.cs
      ├── Resources/
      │   └── (evaluation data files)
      └── README.md
  

• Task 3.3: Implement Self Reflection sample
  • Create evaluation agent
  • Configure reflection criteria
  • Run self-assessment
  • Generate quality scores

Phase 4: Documentation & Verification

• Task 4.1: Create Evaluation/README.md
  • Overview of evaluation capabilities
  • Setup instructions (Azure resources needed)
  • Links to Azure AI Evaluation documentation
• Task 4.2: Update main FoundryAgents/README.md
  • Add links to evaluation samples
  • Document observability integration
• Task 4.3: Verify all samples
  • Build verification
  • Run with test data
  • Verify output format

---

Python Sample References

Red Teaming

File	Purpose	GitHub Link
red_team_agent_sample.py	Main sample code	Link
README.md	Setup instructions	Link
.env.example	Environment variables	Link

Self Reflection

File	Purpose	GitHub Link
self_reflection.py	Main sample code	Link
README.md	Setup instructions	Link
resources/	Evaluation data	Link

---

Key Concepts from Python Samples

Red Teaming (from Python)

# Pattern from red_team_agent_sample.py
# Uses Azure AI Evaluation SDK to test agent safety

from azure.ai.evaluation import RedTeamEvaluator

evaluator = RedTeamEvaluator(...)
result = await evaluator.evaluate(
    target_callback=agent_callback,  # FoundryAgent as target
    attack_prompts=[...],
    risk_categories=[...]
)

Self Reflection (from Python)

# Pattern from self_reflection.py
# Agent evaluates its own responses for quality

evaluation_agent = create_evaluation_agent(...)
results = await evaluation_agent.evaluate(
    responses=agent_responses,
    criteria=["accuracy", "completeness", "relevance"]
)

---

Dependencies & Prerequisites

NuGet Packages to Research

• Azure.AI.Evaluation - Core evaluation SDK
• Azure.AI.Projects - Foundry project integration
• Any additional packages from Python requirements.txt

Azure Resources Required

• Azure AI Foundry project
• Azure OpenAI deployment (for evaluation models)
• Azure AI Evaluation resource (if separate)

---

Open Questions

1. Azure SDK Parity: Does Azure.AI.Evaluation .NET SDK exist with same capabilities as Python?
2. Existing Step07: What specific telemetry does it capture? OpenTelemetry or Azure-specific?
3. Red Team Categories: Which risk categories should the sample cover?
4. Self Reflection Criteria: What evaluation criteria are most useful to demonstrate?

---

Success Criteria

• Existing Step07 Observability verified working
• Red Teaming sample created with full documentation
• Self Reflection sample created with full documentation
• All samples build with dotnet build
• README files provide clear setup instructions
• Samples demonstrate practical evaluation scenarios