Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
| prime eval run configs/eval/benchmark-hosted.toml --hosted | ||
| ``` | ||
|
|
||
| For the full hosted workflow and hosted-only flags such as `--follow`, `--timeout-minutes`, `--allow-sandbox-access`, and `--custom-secrets`, see the official [Hosted Evaluations](https://docs.primeintellect.ai/tutorials-environments/hosted-evaluations) guide. |
There was a problem hiding this comment.
Missing skill update for hosted evaluations workflow
Low Severity
The PR adds a new Hosted Evaluations section to docs/evaluation.md documenting the --hosted flag, --follow, and other hosted-only flags, but skills/evaluate-environments/SKILL.md was not updated. The skill file only mentions "hosted eval workflows" in passing (line 63) and the example command there doesn't even use --hosted. The project rule requires that changes to docs/evaluation.md that affect user-facing workflows are reflected in the corresponding skill file.
Triggered by project rule: BugBot Instructions
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aee2394a5d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| prime env push my-env | ||
| prime eval run my-team/my-env --hosted |
There was a problem hiding this comment.
Keep the pushed namespace consistent with the hosted slug
This example fails if someone copies it literally: prime env push my-env publishes under the caller's own namespace, but the next command runs my-team/my-env. Publishing to a team requires an explicit --team <team> on the push, so the doc should either keep the owner the same in both commands or show the team flag; otherwise readers end up targeting a hosted slug they never created.
Useful? React with 👍 / 👎.
| Hosted runs also support TOML configs: | ||
|
|
||
| ```bash | ||
| prime eval run configs/eval/benchmark-hosted.toml --hosted |
There was a problem hiding this comment.
Explain that hosted TOML configs must use Hub slugs
The new hosted TOML example omits the main difference from the local TOML flow: hosted [[eval]].env_id entries must point at already-published Hub slugs like owner/my-env. Later in this same file, env_id is documented as an environment module name and every config example uses local IDs, so readers following this new snippet will naturally reuse gsm8k/my-env and get hosted runs that cannot resolve the environment.
Useful? React with 👍 / 👎.


Summary
docs/evaluation.mdprime eval run --hostedflow, including publishing first,--follow, TOML config usage, and the official hosted eval guideTesting
Note
Low Risk
Low risk documentation-only change; no code paths or CLI behavior are modified.
Overview
Adds a new Hosted Evaluations section to
docs/evaluation.md, linked from the table of contents, describing how to runprime eval run --hostedagainst Hub-published environments (includingprime env push,--follow, and TOML config usage) and pointing to the official hosted-evals guide for hosted-only flags.Written by Cursor Bugbot for commit aee2394. This will update automatically on new commits. Configure here.