feat: Add AST versioning system with automatic migration support#72
Merged
feat: Add AST versioning system with automatic migration support#72
Conversation
This commit introduces a comprehensive AST versioning system that allows
the codebase to evolve while maintaining backward compatibility with
older AST formats.
## Changes
### Core Versioning
- Add `ast_version` field to SerializableStreamSpec and SerializableStackSpec
- Define CURRENT_AST_VERSION constant (1.0.0) in both AST type modules
- Use serde default to ensure backwards compatibility (defaults to 1.0.0)
### Versioned Loader Module
- Create `versioned` module in both hyperstack-macros and interpreter
- Implement `load_stack_spec()` and `load_stream_spec()` functions with:
- Automatic version detection from JSON
- Version routing to appropriate deserializer
- Clear error messages for unsupported versions
- Provide VersionedStackSpec and VersionedStreamSpec enums for explicit version handling
- Add `detect_ast_version()` utility for debugging
### Backwards Compatibility
- Old ASTs without ast_version field default to "1.0.0"
- All existing code paths continue to work
- No breaking changes to public APIs
### Usage
```rust
// Load with automatic version detection
let spec = load_stack_spec(&json)?;
// Or detect version first
let version = detect_ast_version(&json)?;
println!("AST version: {}", version);
```
### Future Breaking Changes
When AST changes are needed:
1. Bump CURRENT_AST_VERSION (e.g., "1.1.0")
2. Add migration logic in versioned loader
3. Add new variant to VersionedStackSpec/VersionedStreamSpec enums
4. Old versions remain supported until explicit deprecation
## Testing
- Unit tests for version detection
- Tests for loading v1.0.0 specs
- Tests for unsupported version errors
- Tests for backwards compatibility (no version field)
## Files Modified
- hyperstack-macros/src/ast/types.rs (+15 lines)
- hyperstack-macros/src/ast/mod.rs (+4 lines)
- hyperstack-macros/src/ast/versioned.rs (new, 283 lines)
- hyperstack-macros/src/stream_spec/ast_writer.rs (+1 line)
- hyperstack-macros/src/stream_spec/idl_spec.rs (+1 line)
- hyperstack-macros/src/stream_spec/module.rs (+1 line)
- interpreter/src/ast.rs (+16 lines)
- interpreter/src/lib.rs (+1 line)
- interpreter/src/versioned.rs (new, 283 lines)
- interpreter/src/typescript.rs (+1 line)
- cli/src/commands/sdk.rs (+7/-3 lines)
Adds docs/ast-versioning-guide.md with: - Quick reference for version bumping decisions - Step-by-step instructions for breaking changes - Complete migration examples - Best practices and FAQ - Deprecation strategies This guide enables developers to confidently evolve the AST while maintaining backward compatibility and clear migration paths.
- Add warning comments to CURRENT_AST_VERSION in both crates explaining the circular dependency issue and the need to keep versions in sync - Add test_ast_version_sync_* tests in both versioned.rs files that verify the constants match between crates - Update the versioning guide to explain WHY versions must be bumped in 2 places and how the sync test helps catch mismatches
Starting versioning from 0.0.1 as the initial version before any breaking changes. This gives us room to iterate before reaching 1.0.0 stability.
- Remove unused re-exports from ast/mod.rs - Add #![allow(dead_code)] to versioned.rs module with explanation that these are public API items for future use
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR introduces a comprehensive AST versioning system across Key observations:
Confidence Score: 3/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[JSON input] --> B[load_stack_spec / load_stream_spec]
B --> C[serde_json::from_str → raw Value]
C --> D{ast_version field present?}
D -- No --> E[default: '0.0.1']
D -- Yes --> F[read version string]
E --> G{version == CURRENT_AST_VERSION?}
F --> G
G -- Yes --> H[Deserialize directly as SerializableStackSpec]
G -- No --> I{Migration arm exists?}
I -- Yes --> J[Deserialize as old type → migrate to latest]
I -- No --> K[Err: UnsupportedVersion]
H --> L[Ok: SerializableStackSpec]
J --> L
subgraph "Enum API (VersionedStackSpec)"
M[JSON input] --> N[serde_json::from_str]
N --> O{ast_version field present?}
O -- No --> P[❌ Serde error: missing tag]
O -- 0.0.1 --> Q[V1 variant → into_latest]
O -- Other --> R[❌ Serde error: unknown variant]
Q --> S[SerializableStackSpec with original ast_version]
end
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Changed version routing in load_stack_spec and load_stream_spec to use CURRENT_AST_VERSION constant instead of hardcoded "0.0.1". This ensures newly written ASTs with bumped versions are recognized as current format instead of falling through to UnsupportedVersion error.
Changed test_ast_version_sync_* tests to fail with a clear assertion instead of silently passing when the referenced source file doesn't exist. Also fixed the path construction to use only one '..' component since crates are siblings in the workspace. This ensures the tests provide actual safety in CI and don't give false confidence when run in environments where the source tree layout differs.
Remove duplicate unreachable '0.0.1' pattern in the migration example. The example now shows proper progression from oldest to newest: - '2.0.0' = current version (deserialize directly) - '1.0.0' = old v1 (migrate to v2) - '0.0.1' = old v0 (migrate to v2)
Fixed version progression examples to follow proper semver: - Minor bumps: 0.0.1 → 0.1.0 (not 1.1.0) - Major bumps: 0.0.1 → 1.0.0 (not 2.0.0) Also fixed the FAQ example about skipping versions to show correct intermediate version progression.
- Add comprehensive tests for load_stream_spec in both crates (happy path, no-version backward compat, unsupported version) - Fix FAQ about version-skipping: only need migrations for released versions, not every intermediate version number - Replace module-level #![allow(dead_code)] with targeted attributes on specific public API items that aren't yet used within the crate
- Fix misleading context message in CLI that blamed 'version detection' for all load errors; now uses generic 'Failed to load' message - Fix example in versioning guide to use .map_err() with ? instead of bare ? which won't compile (VersionedLoadError doesn't implement From<serde_json::Error>) - Fix migration example to use CURRENT_AST_VERSION constant instead of hardcoded version string
…licate keys The VersionedStackSpec and VersionedStreamSpec enums used #[serde(tag = ast_version)] which conflicted with the ast_version field in the inner struct types, causing duplicate keys in serialized JSON (invalid per RFC 8259). Fixed by removing the Serialize derive from these enums since they're only used for deserialization and conversion via into_latest(), not for serialization. The struct types themselves handle serialization with their ast_version field. Also removed now-unused Serialize imports.
…pport Changed error message from 'Current supported versions' to 'Latest supported version' and added note that older versions are supported via automatic migration. This prevents confusion when old versions are supported through migration arms but the error made it seem like only the latest version works.
Changed examples to use the guard pattern instead of hardcoded version strings. This ensures the current version arm stays correct when the constant is bumped, and teaches the idiomatic pattern used in the production code.
…string Changed test assertions from hardcoded 0.0.1 to CURRENT_AST_VERSION constant. This ensures tests won't fail when a migration arm is added and specs are upgraded to the current version during loading.
…d parsing - Add warning doc comments to all into_latest() methods explaining that the returned spec's ast_version field is not updated to CURRENT_AST_VERSION and that load_*_spec functions should be used for round-trip safety - Fix hardcoded version strings in guide test assertions to use CURRENT_AST_VERSION constant - Make sync test parsing more robust by splitting on '=' first, then extracting the quoted value from the right-hand side only
…t_version docs - Add⚠️ IMPORTANT warnings to VersionedStackSpec and VersionedStreamSpec explaining they require ast_version field and don't handle version-less legacy JSON; recommend using load_*_spec functions instead - Fix detect_ast_version doc to correctly state it returns 0.0.1 as the default, not unknown Fixes in both interpreter and hyperstack-macros crates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AST Versioning System
Summary
This PR introduces a comprehensive AST versioning system that enables the codebase to evolve while maintaining backward compatibility with older AST formats. This allows us to make breaking changes to the AST structure while providing clear migration paths for existing users.
Motivation
Currently, any change to the AST structure breaks all previously generated AST files. This makes it difficult to:
Solution
Core Changes
Version Fields: Added
ast_versionfield to bothSerializableStreamSpecandSerializableStackSpec0.0.1(semver format)serde(default)for backwards compatibility with existing ASTsVersioned Loader Module: Created
versioned.rsin both crates with:load_stack_spec()- Auto-detects version and migrates to latestload_stream_spec()- Same for entity specsdetect_ast_version()- Utility to peek at version without full parseVersionedStackSpec/VersionedStreamSpecenums for explicit handlingSync Verification: Added tests that verify
CURRENT_AST_VERSIONconstants match betweenhyperstack-macrosandinterpretercrates (they must be kept in sync due to circular dependency constraints)Backwards Compatibility
ast_versionfield default to0.0.1Usage
Future Breaking Changes
When AST changes are needed:
CURRENT_AST_VERSIONin bothhyperstack-macros/src/ast/types.rsandinterpreter/src/ast.rsversioned.rsfilesVersionedStackSpec/VersionedStreamSpecenumsSee the AST Versioning Guide for detailed instructions on adding breaking changes.
Testing
Files Changed
New Files
hyperstack-macros/src/ast/versioned.rs(322 lines)interpreter/src/versioned.rs(322 lines)docs/ast-versioning-guide.md(413 lines)Modified Files
hyperstack-macros/src/ast/types.rs- Addast_versionfield and version constanthyperstack-macros/src/ast/mod.rs- Export versioned modulehyperstack-macros/src/stream_spec/ast_writer.rs- Set version when creating specshyperstack-macros/src/stream_spec/idl_spec.rs- Set version when creating specshyperstack-macros/src/stream_spec/module.rs- Set version when creating specsinterpreter/src/ast.rs- Addast_versionfield and version constantinterpreter/src/lib.rs- Export versioned moduleinterpreter/src/typescript.rs- Set version in test specscli/src/commands/sdk.rs- Use versioned loaderMigration Path for Users
No action required for existing users. The system is fully backwards compatible:
Checklist
Related
This PR establishes the foundation for future AST evolution. See the versioning guide for how to add breaking changes in subsequent PRs.