[Feature]: Adding configurable llmservice endpoint which assumes backend stores the system prompt#392
Open
JMN09 wants to merge 4 commits intodi-sukharev:devfrom
Open
[Feature]: Adding configurable llmservice endpoint which assumes backend stores the system prompt#392JMN09 wants to merge 4 commits intodi-sukharev:devfrom
JMN09 wants to merge 4 commits intodi-sukharev:devfrom
Conversation
… of AiEngine that supports a configurable backend, added the corresponding configuration variables, and integrated it in the factory method
…ture/llmservice
Owner
|
thanks for the contribution, i first need to merge refactoring_v1 PR and alter this PR to match the refactoring (if needed, hope not) what are the model files? 🤔 |
Owner
|
@JMN09 please refer to this comment #391 (comment) |
Contributor
Author
|
Hey. A model file is used to create an instance of LLMs with the system prompt wired into them. They are used to gain more context size in prompts without losing accuracy due to large prompts. Especially useful in smaller models such as llama3.1 8b |
Owner
|
please resolve conflicts |
Owner
|
@JMN09 all tests pass, you only need to resolve the conflicts, please lmk when i can merge it |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relying on model files decreases the context size in prompts allowing LLMs to perform better. By adding this endpoint we allow communication with APIs that use model files.
I added a class that implements AiEngine to do so.
In addition to that added the possibility to confire OCO_AI_PROVIDER to llmservice
And added to configuration variable OCO_BACKEND_ENDPOINT and OCO_BACKEND_PATH
closes #386