feat: comprehensive pipeline failure mode test coverage#128
Draft
nextlevelshit wants to merge 7 commits intomainfrom
Draft
feat: comprehensive pipeline failure mode test coverage#128nextlevelshit wants to merge 7 commits intomainfrom
nextlevelshit wants to merge 7 commits intomainfrom
Conversation
Define comprehensive test coverage for all pipeline failure scenarios: - Contract schema mismatch and validation failures - Step timeout handling - Missing artifact detection - Permission denial enforcement - Workspace corruption recovery - Non-zero adapter exit codes - Contract validator false-positive handling Covers 7 user stories with 10 functional requirements. Closes #114
Add proper error reporting when artifacts referenced by inject_artifacts cannot be found. Previously silent failures are now detected and reported with a descriptive message listing all missing artifacts.
Add tests for contract validation failure scenarios: - JSON schema validation with invalid data - Type mismatches, missing required fields, extra properties - Test suite execution failures with non-zero exit codes - Multi-error aggregation and clear error messages - Edge cases for malformed schemas and circular refs
Add tests for adapter failure scenarios: - Non-zero exit code detection and proper error propagation - Timeout handling with context cancellation - Error classification and recovery hints - Process cleanup on adapter failures
Add comprehensive pipeline failure tests: - DAG dependency resolution failures and cycle detection - Permission denial enforcement with deny patterns - Workspace validation and corruption detection - Step timeout handling with proper cleanup - Missing artifact injection failures - Executor error propagation and state management
Add integration tests validating full pipeline failure scenarios: - Contract validation failures with schema mismatch - Step timeout with proper state recording - Missing artifact detection across steps - Permission enforcement at runtime - Non-zero adapter exit handling - Multi-step failure propagation and recovery
Add reusable contracts and pipelines for testing failure scenarios: - artifact-verification.schema.json for artifact validation - contract-validation-test.schema.json for validation testing - failure-modes-validation.schema.json for failure mode output - contract-validation-test.yaml pipeline for contract testing - failure-modes-validation.yaml pipeline for comprehensive validation
Collaborator
Author
|
Wrong folder: .wave -> internal/defaults |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
executor.gospecs/114-pipeline-failure-tests/Changes
Production Code
internal/pipeline/executor.go: Fixed silent artifact injection failures - now reports all missing artifacts with clear error messagesTest Coverage
Artifacts
.wave/contracts/.wave/pipelines/specs/114-pipeline-failure-tests/Spec
See specs/114-pipeline-failure-tests/spec.md for full specification covering 7 user stories and 10 functional requirements.
Test Plan
go test -race ./...Known Limitations
Closes #114