Detect agentic engine secrets in imported custom engine steps with exemptions for agentic frameworks

[Copilot]

Implementation Complete: Detect Agentic CLI Invocation in Imported Steps

Summary

Successfully implemented validation to detect when imported custom engine steps use agentic engine secrets (like COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, CODEX_API_KEY, etc.). These secrets should only be used within the secure firewall environment.

• Strict mode: Returns error when agentic secrets detected in custom steps
• Non-strict mode: Emits warning when agentic secrets detected in custom steps
• Exemption: Custom agentic engines (like OpenCode) are exempt from this check
• Dynamic validation: Secret names are programmatically retrieved from all registered engines via the engine registry
• Infrastructure filtering: Non-agentic infrastructure secrets (MCP_GATEWAY_API_KEY, GITHUB_TOKEN, etc.) are ignored

Detection Strategy

The validation uses the engine registry to programmatically iterate through all registered engines:

• Calls GetGlobalEngineRegistry().GetAllEngines() to get all engines
• For each engine (except custom), calls GetRequiredSecretNames()
• Filters out infrastructure secrets using isAgenticEngineSecret() helper:
  • MCP_GATEWAY_API_KEY - MCP gateway infrastructure
  • GITHUB_MCP_SERVER_TOKEN - GitHub MCP server access
  • GH_AW_GITHUB_TOKEN - GitHub API access
  • GITHUB_TOKEN - GitHub Actions default token
• Maps agentic secrets to their engine display names

Benefits:

• Automatically includes any new engines added to the registry
• Single source of truth (engine instances define their secrets)
• No manual maintenance needed when engines change
• Fully programmatic and maintainable
• Clear separation between agentic and infrastructure secrets

Exemptions: Custom engines that are actually agentic framework wrappers (like opencode-ai) are exempt from this validation since they legitimately need these secrets.

Implementation Checklist

• Understand current import and validation flow
• Create validation function to detect engine secrets in custom steps
• Add validation to compiler orchestrator
• Add warning for non-strict mode with warning count
• Write comprehensive tests (all passing)
• Fix linting issues
• Add exemption for custom agentic engines (OpenCode, etc.)
• Refactor to use GetRequiredSecretNames() from engine instances
• Refactor to use engine registry for programmatic iteration
• Document infrastructure secret filtering
• Validation working correctly with exemptions

Files Modified

• pkg/workflow/imported_steps_validation.go - Dynamic validation using engine registry with documented filtering
• pkg/workflow/imported_steps_validation_test.go - Comprehensive test coverage including OpenCode exemption
• pkg/workflow/compiler_orchestrator_engine.go - Integration into compilation flow
• .github/workflows/smoke-opencode.md - Restored to strict mode (now exempt)

Validation Results

The implementation successfully:

• Detects unsafe usage of agentic secrets in custom engine steps
• Programmatically retrieves secret names from all registered engines (fully maintainable)
• Properly filters infrastructure secrets that are safe to use
• Exempts legitimate custom agentic engines like OpenCode
• Works correctly in both strict and non-strict modes
• Automatically adapts to new engines added to the registry

Custom agent used: technical-doc-writer
AI technical documentation writer for GitHub Actions library using GitHub Docs voice

Original prompt

When importing steps in the action job, the compiler should detect invocation of the agentic cli. The steps run outside the firewall and are unsafe.

In strict mode, these steps should trigger an error. In non strict mode, a warning.

To detect this scenario, detect if the secret of the agentic engine are used like (${{ secrets.COPILOT_GITHUB_TOKEN }}). These secrets are not allowed in steps in strict mode.

Custom agent used: technical-doc-writer
AI technical documentation writer for GitHub Actions library using GitHub Docs voice

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[pelikhan]

@copilot ignore this check for custom agentic engines like open code

[Copilot]

@copilot ignore this check for custom agentic engines like open code

Added exemption for custom agentic engines like OpenCode. The validation now detects when a custom engine is actually an agentic framework wrapper (by looking for opencode-ai, opencode, etc.) and skips the check in those cases.

Changes in commit ea8dfa2:

• Added isCustomAgenticEngine() function to detect agentic framework wrappers
• Restored smoke-opencode.md to strict: true (now exempt from validation)
• Added test coverage for OpenCode exemption

The smoke-opencode workflow now compiles successfully in strict mode.

[Copilot]

Pull request overview

This pull request implements validation to detect when imported custom engine steps use agentic engine secrets that should only be accessible within the secure firewall environment. The implementation dynamically queries engine instances for their required secrets and validates custom engine steps against this list, with exemptions for custom agentic engines like OpenCode.

Changes:

• Added imported_steps_validation.go with dynamic secret detection using GetRequiredSecretNames() from engine instances
• Integrated validation into compiler_orchestrator_engine.go to run during workflow compilation
• Added comprehensive test coverage in imported_steps_validation_test.go including OpenCode exemption scenarios

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
pkg/workflow/imported_steps_validation.go	Core validation logic that dynamically builds agentic secret map, detects secret usage in custom steps, and provides exemptions for agentic frameworks
pkg/workflow/imported_steps_validation_test.go	Comprehensive test suite covering Copilot, Claude, Codex secrets, safe secrets, multiple secrets, non-custom engines, and OpenCode exemption
pkg/workflow/compiler_orchestrator_engine.go	Integration point that calls validation during engine setup with proper strict mode handling

Comments suppressed due to low confidence (4)

pkg/workflow/imported_steps_validation_test.go:396

• The test suite is missing coverage for several edge cases:

1. A custom engine that uses agentic secrets but isn't detected by isCustomAgenticEngine (e.g., a different agentic framework not in the exemption list)
2. A custom engine with no steps (empty steps array)
3. A workflow with multiple imports where some have violations and some don't
4. Error handling when convertStepToYAML fails
5. Secrets with lowercase letters (the regex only matches [A-Z_][A-Z0-9_]*)
6. Complex fallback expressions like secrets.A || secrets.B || secrets.C

Adding tests for these cases would improve confidence in the validation logic.

func TestValidateImportedStepsNoAgenticSecrets_Copilot(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755), "Failed to create workflows directory")

	// Create an import file with custom engine using COPILOT_GITHUB_TOKEN
	importContent := `---
engine:
  id: custom
  steps:
    - name: Call Copilot CLI
      run: |
        gh copilot suggest "How do I use Git?"
      env:
        GH_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
---

# Shared Custom Engine
This shared config uses Copilot CLI with the agentic secret.
`
	importFile := filepath.Join(workflowsDir, "shared", "copilot-custom-engine.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow that imports this
	mainContent := `---
name: Test Workflow
on: push
imports:
  - shared/copilot-custom-engine.md
---

# Test Workflow
This workflow imports a custom engine with agentic secrets.
`
	mainFile := filepath.Join(workflowsDir, "test-copilot-secret.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode - should fail
	t.Run("strict mode error", func(t *testing.T) {
		compiler := NewCompiler()
		compiler.SetStrictMode(true)
		err := compiler.CompileWorkflow(mainFile)

		require.Error(t, err, "Expected error in strict mode")
		if err != nil {
			assert.Contains(t, err.Error(), "strict mode", "Error should mention strict mode")
			assert.Contains(t, err.Error(), "COPILOT_GITHUB_TOKEN", "Error should mention the secret name")
			assert.Contains(t, err.Error(), "Copilot engine", "Error should mention the engine")
			assert.Contains(t, err.Error(), "custom engine steps", "Error should mention custom engine steps")
		}
	})

	// Test in non-strict mode - should succeed with warning
	t.Run("non-strict mode warning", func(t *testing.T) {
		// Update main file to explicitly disable strict mode
		mainContentNonStrict := `---
name: Test Workflow
on: push
strict: false
imports:
  - shared/copilot-custom-engine.md
---

# Test Workflow
This workflow imports a custom engine with agentic secrets.
`
		mainFileNonStrict := filepath.Join(workflowsDir, "test-copilot-secret-nonstrict.md")
		require.NoError(t, os.WriteFile(mainFileNonStrict, []byte(mainContentNonStrict), 0644))

		compiler := NewCompiler()
		err := compiler.CompileWorkflow(mainFileNonStrict)

		require.NoError(t, err, "Should not error in non-strict mode")
		assert.Positive(t, compiler.GetWarningCount(), "Should have warnings")
	})
}

func TestValidateImportedStepsNoAgenticSecrets_Anthropic(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create an import file with custom engine using ANTHROPIC_API_KEY
	importContent := `---
engine:
  id: custom
  steps:
    - name: Use Claude API
      uses: actions/claude-action@v1
      with:
        api-key: ${{ secrets.ANTHROPIC_API_KEY }}
---

# Shared Custom Engine
`
	importFile := filepath.Join(workflowsDir, "shared", "claude-custom-engine.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow
	mainContent := `---
name: Test Workflow
on: push
imports:
  - shared/claude-custom-engine.md
---

# Test
`
	mainFile := filepath.Join(workflowsDir, "test-anthropic-secret.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	require.Error(t, err, "Expected error in strict mode")
	if err != nil {
		assert.Contains(t, err.Error(), "ANTHROPIC_API_KEY")
		assert.Contains(t, err.Error(), "Claude engine")
	}
}

func TestValidateImportedStepsNoAgenticSecrets_Codex(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create an import file with custom engine using both CODEX_API_KEY and OPENAI_API_KEY
	importContent := `---
engine:
  id: custom
  steps:
    - name: Use OpenAI API
      run: |
        curl -X POST https://api.openai.com/v1/completions \
          -H "Authorization: Bearer $OPENAI_API_KEY"
      env:
        OPENAI_API_KEY: ${{ secrets.CODEX_API_KEY || secrets.OPENAI_API_KEY }}
---

# Shared Custom Engine
`
	importFile := filepath.Join(workflowsDir, "shared", "codex-custom-engine.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow
	mainContent := `---
name: Test Workflow
on: push
imports:
  - shared/codex-custom-engine.md
---

# Test
`
	mainFile := filepath.Join(workflowsDir, "test-codex-secret.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	require.Error(t, err, "Expected error in strict mode")
	if err != nil {
		// Should detect both secrets
		errMsg := err.Error()
		hasCodex := strings.Contains(errMsg, "CODEX_API_KEY")
		hasOpenAI := strings.Contains(errMsg, "OPENAI_API_KEY")
		assert.True(t, hasCodex || hasOpenAI, "Should mention at least one of the Codex secrets")
		assert.Contains(t, errMsg, "Codex engine")
	}
}

func TestValidateImportedStepsNoAgenticSecrets_SafeSecrets(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create an import file with custom engine using non-agentic secrets (should be fine)
	importContent := `---
engine:
  id: custom
  steps:
    - name: Use Custom API
      run: |
        curl -X POST https://api.example.com/v1/data \
          -H "Authorization: Bearer $MY_CUSTOM_TOKEN"
      env:
        MY_CUSTOM_TOKEN: ${{ secrets.MY_CUSTOM_API_KEY }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
---

# Shared Custom Engine
This uses safe, non-agentic secrets.
`
	importFile := filepath.Join(workflowsDir, "shared", "safe-custom-engine.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow
	mainContent := `---
name: Test Workflow
on: push
strict: false
permissions:
  contents: read
  issues: read
  pull-requests: read
imports:
  - shared/safe-custom-engine.md
---

# Test
`
	mainFile := filepath.Join(workflowsDir, "test-safe-secrets.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode - should succeed
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	assert.NoError(t, err, "Should not error when using safe secrets")
	// Note: We may have warning for experimental feature (custom engine), but not for our secret validation
}

func TestValidateImportedStepsNoAgenticSecrets_NonCustomEngine(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create workflow with non-custom engine (copilot) - should not validate
	mainContent := `---
name: Test Workflow
on: push
engine: copilot
strict: false
permissions:
  contents: read
  issues: read
  pull-requests: read
---

# Test
This uses the standard copilot engine, not custom.
`
	mainFile := filepath.Join(workflowsDir, "test-copilot-engine.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode - should succeed (validation doesn't apply)
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	assert.NoError(t, err, "Should not error for non-custom engines")
}

func TestValidateImportedStepsNoAgenticSecrets_MultipleSecrets(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create an import file with custom engine using multiple agentic secrets
	importContent := `---
engine:
  id: custom
  steps:
    - name: Use Multiple AI APIs
      run: |
        echo "Using Copilot"
        gh copilot suggest "help"
      env:
        GH_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }}
    - name: Use Claude
      run: |
        echo "Using Claude"
      env:
        ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
---

# Shared Custom Engine
`
	importFile := filepath.Join(workflowsDir, "shared", "multi-secret-engine.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow
	mainContent := `---
name: Test Workflow
on: push
imports:
  - shared/multi-secret-engine.md
---

# Test
`
	mainFile := filepath.Join(workflowsDir, "test-multi-secrets.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	require.Error(t, err, "Expected error in strict mode")
	if err != nil {
		errMsg := err.Error()
		// Should mention both secrets
		assert.Contains(t, errMsg, "COPILOT_GITHUB_TOKEN")
		assert.Contains(t, errMsg, "ANTHROPIC_API_KEY")
		// Should mention both engines
		hasCopilot := strings.Contains(errMsg, "Copilot engine")
		hasClaude := strings.Contains(errMsg, "Claude engine")
		assert.True(t, hasCopilot || hasClaude, "Should mention the engines")
	}
}

func TestValidateImportedStepsNoAgenticSecrets_OpenCodeExemption(t *testing.T) {
	tmpDir := testutil.TempDir(t, "test-*")
	workflowsDir := filepath.Join(tmpDir, constants.GetWorkflowDir())
	require.NoError(t, os.MkdirAll(workflowsDir, 0755))

	// Create an import file with OpenCode custom engine using agentic secrets
	importContent := `---
engine:
  id: custom
  env:
    GH_AW_AGENT_VERSION: "0.15.13"
  steps:
    - name: Install OpenCode
      run: |
        npm install -g "opencode-ai@${GH_AW_AGENT_VERSION}"
      env:
        GH_AW_AGENT_VERSION: ${{ env.GH_AW_AGENT_VERSION }}
    - name: Run OpenCode
      run: |
        opencode run "test prompt"
      env:
        ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
---

# OpenCode Engine
This is a custom agentic engine wrapper.
`
	importFile := filepath.Join(workflowsDir, "shared", "opencode.md")
	require.NoError(t, os.MkdirAll(filepath.Dir(importFile), 0755))
	require.NoError(t, os.WriteFile(importFile, []byte(importContent), 0644))

	// Create main workflow with strict mode
	mainContent := `---
name: Test OpenCode Workflow
on: push
strict: true
permissions:
  contents: read
  issues: read
  pull-requests: read
imports:
  - shared/opencode.md
---

# Test
This workflow uses OpenCode which is a custom agentic engine.
`
	mainFile := filepath.Join(workflowsDir, "test-opencode.md")
	require.NoError(t, os.WriteFile(mainFile, []byte(mainContent), 0644))

	// Test in strict mode - should succeed because OpenCode is exempt
	compiler := NewCompiler()
	compiler.SetStrictMode(true)
	err := compiler.CompileWorkflow(mainFile)

	assert.NoError(t, err, "Should not error for OpenCode custom engine even in strict mode")
}

pkg/workflow/imported_steps_validation.go:211

• The error message on line 205-211 repeats the secretsList three times - once in the description, once in "These secrets are for", and once in "Remove references to". This repetition could be confusing if there are many secrets. Consider restructuring the message to be clearer, for example: "These secrets ({secretsList}) are for {enginesDisplay} and should only be used within the secure firewall environment. Custom engine steps run outside the firewall and bypass security isolation."

	errorMsg := fmt.Sprintf(
		"custom engine steps use agentic engine secrets (%s) which are not allowed. "+
			"These secrets are for %s and should only be used within the secure firewall environment. "+
			"Custom engine steps run outside the firewall and bypass security isolation. "+
			"Remove references to %s from your custom engine steps. "+
			"See: https://github.github.com/gh-aw/reference/engines/",
		secretsList, enginesDisplay, secretsList,

pkg/workflow/imported_steps_validation.go:83

• The buildAgenticEngineSecretsMap function creates empty engine instances without providing WorkflowData, which is then passed to GetRequiredSecretNames. While this works for getting the base agentic secrets, it means the map is built with an empty WorkflowData. This could be problematic if the secret names depend on workflow configuration. However, for the current use case of detecting only core agentic secrets (COPILOT_GITHUB_TOKEN, ANTHROPIC_API_KEY, etc.), this approach works but may need adjustment if engine secret requirements become more dynamic in the future. Consider adding a comment explaining this design decision.

// buildAgenticEngineSecretsMap dynamically builds a map of agentic engine secret names
// by querying each registered engine for its required secrets
func buildAgenticEngineSecretsMap() map[string]string {
	secretsMap := make(map[string]string)

	// Create a minimal WorkflowData for querying engine secrets
	// (we don't need MCP servers or other config for base secret names)
	workflowData := &WorkflowData{}

	// Get secrets from Copilot engine
	copilotEngine := NewCopilotEngine()
	for _, secret := range copilotEngine.GetRequiredSecretNames(workflowData) {
		// Skip non-agentic secrets (like MCP_GATEWAY_API_KEY, GITHUB_MCP_SERVER_TOKEN)
		if secret == "COPILOT_GITHUB_TOKEN" {
			secretsMap[secret] = "Copilot engine"
		}
	}

	// Get secrets from Claude engine
	claudeEngine := NewClaudeEngine()
	for _, secret := range claudeEngine.GetRequiredSecretNames(workflowData) {
		// Skip non-agentic secrets
		if secret == "ANTHROPIC_API_KEY" || secret == "CLAUDE_CODE_OAUTH_TOKEN" {
			secretsMap[secret] = "Claude engine"
		}
	}

	// Get secrets from Codex engine
	codexEngine := NewCodexEngine()
	for _, secret := range codexEngine.GetRequiredSecretNames(workflowData) {
		// Skip non-agentic secrets
		if secret == "CODEX_API_KEY" || secret == "OPENAI_API_KEY" {
			secretsMap[secret] = "Codex engine"
		}
	}

	return secretsMap
}

pkg/workflow/imported_steps_validation_test.go:335

• The test for multiple secrets (lines 329-335) uses OR logic to check if either engine is mentioned, which is less strict than AND logic. Since the error message should mention both engines when both secrets are detected, the test should verify that both "Copilot engine" and "Claude engine" are present in the error message. Using OR logic (hasCopilot || hasClaude) means the test would pass even if only one engine is mentioned, which doesn't fully validate the error message construction.

		// Should mention both secrets
		assert.Contains(t, errMsg, "COPILOT_GITHUB_TOKEN")
		assert.Contains(t, errMsg, "ANTHROPIC_API_KEY")
		// Should mention both engines
		hasCopilot := strings.Contains(errMsg, "Copilot engine")
		hasClaude := strings.Contains(errMsg, "Claude engine")
		assert.True(t, hasCopilot || hasClaude, "Should mention the engines")

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.