feat(pipeline-loader, pipeline-server): improve pipeline loading and sourcing#527
Draft
feat(pipeline-loader, pipeline-server): improve pipeline loading and sourcing#527
Conversation
- Introduced `BundleResult` interface to include `dataUrl` in the bundling process. - Updated `bundle` function to return both `code` and `dataUrl`. - Refactored `loadPipelineFile` to utilize the new bundling structure. - Improved error handling and added tests for GitHub and GitLab sources. - Added utility functions for parsing remote source URLs.
- Removed `parseRepoString` function and replaced it with `parseRemoteSourceUrl` for better clarity and error handling. - Consolidated remote source addition logic into a single `addRemoteSource` function to reduce code duplication.
- Implemented a new loader function to fetch pipelines from the API. - Added error handling for failed fetch requests and pipeline errors. - Updated the HomePage component to display loading and error states.
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
🌏 Preview Deployments
Built from commit: 🤖 This comment will be updated automatically when you push new commits to this PR. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
packages/pipelines/pipeline-server/src/client/components/keyboard-shortcuts-help.tsx
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/routes/$sourceId/$fileId/$pipelineId/index.tsx
Fixed
Show fixed
Hide fixed
- Introduced `pipelineQueryOptions` for fetching pipeline data. - Created lazy-loaded route for pipeline details, including version selection and error handling.
…query options Refactor `QuickActionsPanel`, `RecentExecutionsPanel`, and `RecentOutputsPanel` to use new route parameters for `sourceId`, `fileId`, and `pipelineId`. Introduce `executionsQueryOptions` for fetching execution data with improved API structure. Add lazy loading for the new pipeline overview route.
…for pipeline code
Added lazy loading routes for inspecting and viewing graphs of pipelines under the `$sourceId/$fileId/$pipelineId` path. This enhances the routing capabilities for pipeline details.
packages/pipelines/pipeline-server/src/client/lib/query-options/pipeline.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/pipeline.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/pipeline.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/source.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/source.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/source.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/execution.ts
Fixed
Show fixed
Hide fixed
…dex router - Consolidate source routes by removing the `sourcesIndexRouter`. - Update the `sourcesSourceRouter` to handle additional source details. - Refactor query options imports for better organization. - Introduce new components for route details and list display in the UI.
packages/pipelines/pipeline-server/src/client/lib/query-options/execution.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/source.ts
Fixed
Show fixed
Hide fixed
packages/pipelines/pipeline-server/src/client/lib/query-options/source.ts
Fixed
Show fixed
Hide fixed
- Introduced `dashboardRouter` to handle dashboard statistics and recent executions. - Updated existing routes to include the new dashboard functionality. - Refactored execution data handling to improve clarity and maintainability.
packages/pipelines/pipeline-server/src/client/components/pipeline-command-palette.tsx
Fixed
Show fixed
Hide fixed
| } | ||
| }; | ||
|
|
||
| const handleBack = () => { |
…ionality - Added `./types` export to `package.json` for type definitions. - Updated `tsdown.config.ts` to include `./src/types.ts` as an entry point. - Introduced `sourcesOverviewRouter` for aggregated source data retrieval. - Created `fetch-overview.ts` for fetching overview data with React Query integration. - Updated various components and schemas to utilize new overview data structure.
…hing - Consolidated execution-related functions into a single module. - Introduced new `fetchOverview` and `overviewQueryOptions` functions for fetching overview data. - Updated imports in `use-execute` to reflect new structure.
- Added `fetchOnMount` parameter to various query options for better control over data fetching. - Refactored `useExecute` to utilize `useMutation` for executing pipelines. - Updated `QuickActionsPanel` to handle execution results more effectively. - Removed unused `useEventView` hook to streamline codebase.
Updated query options across the pipeline UI to accept a `baseUrl` parameter. This change enhances the flexibility of API calls by allowing the base URL to be specified dynamically. Refactored functions for fetching executions, pipelines, and execution details to utilize the new structure, improving code maintainability and readability.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Linked issue
📚 Description
This PR fixes the issues with pipeline loading, and in general rewrites the entire system.
We now always bundle via rolldown, this is to ensure that imports and other things work correctly. For remote sources we download the "pipeline" project into the
.config/ucd/cache/repos/<source>/<owner>/<repo>/<commit-sha>.Important
For security purposes, we don't install any dependencies. Dependencies can be installed via the CLI, to ensure that we can bundle the pipeline in the most efficient way. When a user uses the CLI, to run on a remote pipeline source, we download the pipeline, and will ask them to install. If they don't install, we will return a nice error message.