Fix: support/summarizer model lifecycle

[giveen]

Purpose: Fix support/summarizer model lifecycle and improve auto‑compaction so the support agent no longer holds server context, avoids repeating exhausted approaches, and doesn’t crash the runner.
What Changed

Flush support model: Added async def cleanup() to OpenAIChatCompletionsModel to close the underlying async client, remove the instance from ACTIVE_MODEL_INSTANCES, and clear message_history.

Call cleanup after summarization: Updated _ai_summarize_history to call model_inst.cleanup() in a finally block so temporary summary/support model instances are best-effort flushed after Runner.run. See memory.py.
Compaction & resume hardening (context): earlier related fixes included moving _last_user_input assignment earlier, injecting compacted summary back into message_history to prevent repeat work, and improving _format_history_for_summary to preserve tool args and larger tool outputs so the summarizer can list “Exhausted Approaches”. See cli.py, openai_chatcompletions.py, and memory.py.
Why

CAI_SUPPORT_MODEL / auto‑compaction path was leaving ephemeral summary/support model instances with open HTTP clients and retained message history, causing downstream LLM servers (e.g., llama-support) to accumulate large context (n_ctx_slot) and consume slots/tokens. Flushing prevents slot/context growth and avoids repeated retries of exhausted approaches.

Config checks

export CAI_SUPPORT_MODEL=llama-support
export CAI_SUPPORT_INTERVAL=3

My current setup for testing is as follows:

LLAMA.CPP and LiteLLM(proxy)

.env

OPENAI_API_BASE="http://192.168.0.165:4000/v1"
CAI_MODEL="openai/reasoner"
CAI_SUPPORT_MODEL="openai/support"
CAI_SUPPORT_INTERVAL=25
OPENAI_API_KEY="sk-asdfasdfs"
PROMPT_TOOLKIT_NO_CPR=1
CAI_STREAM=false
CAI_AGENT_TYPE="redteam_agent"
CAI_GUARDRAILS="true"
CAI_TOOL_TIMEOUT=120
CAI_MEMORY="episodic"
CAI_MEMORY_ONLINE="true"
CAI_MEMORY_ONLINE_INTERVAL=5

Main Model setup with LLAMA.CPP

ExecStart=/mnt/storage/llama.cpp/build/bin/llama-server \
    --model /mnt/storage/models/qwen3.5-27b-opus-distill-Q6_K.gguf \
    --model-draft /mnt/storage/models/Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf \
    --n-gpu-layers 99 \
    --n-gpu-layers-draft 99 \
    --ctx-size 65536 \
    --threads 16 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --flash-attn on \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --sleep-idle-seconds 300 \
    --host 0.0.0.0 \
    --port 8080 \
    --cont-batching \
    --jinja

Support Model setup with LLAMA.CPP

ExecStart=/mnt/storage/llama.cpp/build/bin/llama-server \
    --model /mnt/storage/models/Qwen2.5-Coder-1.5B-Instruct-Q4_K_M.gguf \
    --n-gpu-layers 99 \
    --ctx-size 32768 \
    --port 8081 \
    --slot-save-path /tmp \
    --host 0.0.0.0 \
    --flash-attn on

With this setup, every 25 turns (user input and LLM input/out), the support model will summarize the work that was completed, provide a list of things that were done, or failed and suggested methods moving forward.

This is then saved to memory.

Context is cleared for main model and summary

Summary is loaded back into main model as its prompt and it continues its work. This prevents local LLM's from running out of context space and causing local models to shutdown under load.

CODED WITH CLaude Sonnet 4.6 - High thinking mode

[Copilot]

Pull request overview

Fixes the support/summarizer model lifecycle to avoid leaking client resources/history, and hardens auto-compaction so the CLI can restart cleanly after mid-run compaction.

Changes:

• Added OpenAIChatCompletionsModel.cleanup() and invoked it after summarization to best-effort close async clients and clear temporary model state.
• Updated OpenAI Chat Completions request building to avoid double-including history when CLI already passes it in conversation_input.
• Added a ContextCompactedError restart path and support-interval-based auto-compaction/replay behavior in the CLI/model.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/cai/sdk/agents/models/openai_chatcompletions.py	Adds explicit model cleanup and a support-interval auto-compact trigger (raising ContextCompactedError) and avoids double-prepending history.
src/cai/repl/commands/memory.py	Improves summary prompt/formatting and ensures temporary summarizer models are cleaned up in a finally block.
src/cai/cli.py	Catches ContextCompactedError to restart Runner and replays the last user task; prints auto-compact status and includes a CLI-side interval compaction block.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In the CAI_SUPPORT_INTERVAL trigger, ContextCompactedError is raised unconditionally after the summarization attempt, even if summarization/compaction fails (e.g., import/runtime error). In that failure case, message_history may remain unchanged and the next Runner restart will immediately hit the same condition again, potentially causing an infinite restart loop. Consider only raising ContextCompactedError after successfully clearing/reseeding history (or add a guard/backoff flag so failures fall back to normal execution and don’t retrigger immediately).

[Copilot]

This CLI-level CAI_SUPPORT_INTERVAL auto-compact runs whenever CAI_SUPPORT_MODEL/CAI_SUPPORT_INTERVAL are set, but it does not check CAI_AUTO_COMPACT. As a result, setting CAI_AUTO_COMPACT=false will still trigger count-based compaction from the CLI loop. Consider honoring the same disable flag here (or introducing a dedicated flag for support-interval compaction) so users can reliably turn auto-compaction off.

Suggested change

	if _support_model and _support_interval_raw:
	_auto_compact_raw = os.getenv("CAI_AUTO_COMPACT", "true")
	_auto_compact_enabled = _auto_compact_raw.strip().lower() not in {
	"0", "false", "no", "off"
	}
	if _auto_compact_enabled and _support_model and _support_interval_raw:

[Copilot]

When applying the new in-memory summary, COMPACTED_SUMMARIES[self.agent_name] is overwritten but APPLIED_MEMORY_IDS is only initialized for the first-time case. If the agent already has entries, the old APPLIED_MEMORY_IDS[self.agent_name] values remain and no longer correspond to the newly applied summary, which can make /memory status misleading. Consider clearing/updating APPLIED_MEMORY_IDS[self.agent_name] alongside COMPACTED_SUMMARIES when overwriting the summary (even if no memory_id is available).

Suggested change

	if self.agent_name not in COMPACTED_SUMMARIES:
	COMPACTED_SUMMARIES[self.agent_name] = []
	APPLIED_MEMORY_IDS[self.agent_name] = []
	COMPACTED_SUMMARIES[self.agent_name] = [_summary]
	COMPACTED_SUMMARIES[self.agent_name] = [_summary]
	APPLIED_MEMORY_IDS[self.agent_name] = []