Enhance pipeline stages, progress tracking, and UI for restores#14
Merged
Enhance pipeline stages, progress tracking, and UI for restores#14
Conversation
Introduce a typed pipeline stage system and structured progress APIs across runner, logs, and UI. Add PIPELINE_STAGES, STAGE_ORDER, STAGE_PROGRESS_MAP and stageProgress() to calculate global progress from per-stage percent. Update runner to use setStage(), updateDetail(), and updateStageProgress() (track stage durations and emit completion log entries). Revamp LogViewer grouping/sorting to use pipeline order, show pending stages, per-stage durations and nicer icons/badges. Improve adapters/logging: buffer mongodump stderr and emit as single log entry, make SQLite dump logs typed and include details, and propagate metadata.detail to history UI. Update types and changelog accordingly.
Introduce end-to-end transfer progress and restore-specific stage handling across the runner, adapters, and UI. Key changes: - Added scripts/setup-dev-debian.sh to install common DB clients/tools for Debian/Ubuntu (Postgres 14/16/17/18 symlinks, MongoDB tools, sqlite3, redis-cli, ssh/rsync/smbclient). - Storage adapters (Google Drive, OneDrive, S3, WebDAV, FTP, SFTP) now report real-time bytes transferred and total size using stream transformers / client size/stat where available, enabling accurate progress and speed displays during downloads/uploads. - ProgressMonitorStream now reports throughput (bytes/s) in addition to percent. - Upload flow: ProgressMonitor reports speed; upload UI details now show transferred/total bytes and throughput; post-upload integrity checks for local destinations are moved to a dedicated Verifying stage executed after all uploads complete (queued instead of inline). - RestoreService: added structured setStage() and updateDetail() helpers, records stage durations as log entries, and displays byte progress and speed for download, decryption, decompression, and restore operations; properly transitions to RESTORE-specific stages (Downloading, Decrypting, Decompressing, Restoring Database, Completed/Failed/Cancelled). - Frontend LogViewer: accepts executionType, uses separate BACKUP_STAGE_ORDER and RESTORE_STAGE_ORDER to order and show pending stages correctly for restores; grouped stage logic updated and memo dependencies expanded. - Small UX/runner tweaks: dump monitoring now shows throughput; various progress/detail text improvements. - Updated wiki/changelog.md to reflect features, improvements and bug fixes. These changes improve visibility into transfer progress and speeds, keep restore executions distinct from backups, and centralize stage/duration reporting for more accurate logs and UI presentation.
Add byte-level progress reporting for restores and SFTP uploads. SshClient.uploadFile now accepts an optional progress callback (uses sftp.fastPut step), MySQL/Postgres restore adapters and the core DatabaseAdapter interface propagate an optional detail string with progress, and MySQL SSH restore reports upload percent, transferred/total bytes and speed. RestoreService was updated to consume percentage+detail, reset progress on stage change, and fix ordering so Cancelled/Failed stages appear correctly in logs. Also minor UI tweak hides raw percent when a stage detail is shown, added formatBytes import, made setup script executable, and updated the changelog.
Prevent leftover upload/byte progress from persisting when the remote restore command starts. Pass an explicit empty string to onProgress in the MySQL SSH restore to clear the detail, normalize currentDetail using `detail || null` when flushing logs, and update the changelog to document the fixes (progress detail clearing and verification ordering). Files changed: src/lib/adapters/database/mysql/restore.ts, src/services/restore-service.ts, wiki/changelog.md.
Update project and wiki package versions to 1.4.0 and update OpenAPI docs (api-docs/openapi.yaml and public/openapi.yaml) to match. Mark v1.4.0 as released in wiki/changelog.md, consolidate and clean up feature/improvement/bugfix entries, and remove the CI/CD setup script note. These changes reflect the 1.4.0 release metadata and documentation updates.
Cleanup unused variables and minor refactors, and update tests to include newly-required mocks. - src/lib/adapters/database/postgres/restore.ts: remove unused `totalSize` variable before SFTP upload. - src/services/restore-service.ts: rename `updateProgress` to `_updateProgress`, remove unused percent calculations for decryption/decompression, and adjust an `updateDetail` call indentation. - tests/unit/...: add `setStage`, `updateDetail`, and `updateStageProgress` mocked functions to affected unit tests so they match the service API changes. These changes are primarily dead-code removal and test updates to keep tests aligned with the service interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.