Serverjonas/sjcode
SHA256
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FIX SOME BUGS

Browse Source
This commit is contained in:
jhkaktus
2026-09-12 10:14:55 +02:00
parent 02d48a304a
commit 8f1c3207c9
15 changed files with 1356 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+61 -8
View File
@@ -1,18 +1,52 @@
# sjcode
ein terminal coding agent, der die ollama API nutzt
a terminal coding agent, powered by the ollama API
## What makes it an agent
The client is not a chat front end: the model can **really work in your
codebase**. It runs shell commands, reads and edits files — on your machine,
in the directory you started `./coder` from. Every model reply can contain
tool calls; the client executes them and feeds the real output back to the
model until the task is done.
Tools the agent can use:
| tool | what it does | asks first |
|---------|--------------------------------------------------|------------|
| `run` | run one shell command in the project directory | yes |
| `read` | read a file (numbered lines, `start`/`limit`) | no |
| `files` | list project files (skips `.git`, caches, ...) | no |
| `write` | create a file, or overwrite a small one | yes |
| `edit` | exact-text replace in a file, shows a diff | yes |
Before every `run`/`write`/`edit` you see exactly what will happen (command
line, or a unified diff for file changes) and press Enter to allow it, `n` to
decline, `a` to stop the agent — or just type a short instruction ("use ripgrep
instead") and the model adapts. `--yes` (or `/auto` in the session) runs
everything without asking; the sandbox keeps all file tools inside the project
directory.
## Setup
```bash
cp config.toml.example config.toml # then edit port / model / limits
cp server/config.toml.example server/config.toml # then edit port/model/limits
python3 server/server.py # starts the daemon + admin CLI
```
`config.toml` is machine-local and git-ignored; `config.toml.example` is the
template. On first start the daemon creates the SQLite database (`sjcode.db`).
## Daemon (server.py)
Pick a coder model in `[ollama] model` — one that follows structured output
well, e.g. `qwen2.5-coder:7b` or bigger. The agent transcript is long, so the
daemon requests a large context window from ollama:
```toml
[ollama]
num_ctx = 16384 # context window; do not go below 8192 for agent mode
```
## Daemon (server/server.py)
On startup it prints its settings and drops into a minimal admin CLI
(accounts are stored in SQLite):
@@ -32,12 +66,18 @@ On startup it prints its settings and drops into a minimal admin CLI
While the CLI runs, an HTTP API is served in the background:
- `POST /login` — `{"username", "password"}` → `{"token", ...}`
- `POST /chat` — `{"token", "message", "model"?}` → `{"response", ...}`
- `POST /chat` — `{"token", "message", "model"?, "stream"?}` → reply as JSON,
or (with `"stream": true`) as NDJSON lines `{"chunk": ...}` ending with one
final `{"response", "used_minutes", "remaining_minutes", ...}` line
- `GET /models?token=...` — list ollama models
- `GET /status` — health check
Every user row stores: `username`, `password_hash`, `last_logged_in`,
`role`, `used_today` (minutes of AI usage; resets on a new day).
`SJCODE_CONFIG=/path/to/config.toml` lets you run a second daemon (tests,
staging) next to a regular one.
## Roles and limits
In `config.toml` each role gets a daily limit in minutes:
@@ -56,6 +96,7 @@ Login and every chat request are blocked once `used_today` reaches the limit.
./coder # or: python3 client.py
python3 client.py 10.0.0.5:8420 # explicit daemon address
python3 client.py --model qwen3-coder # skip the model picker
python3 client.py --yes # auto-approve every tool call (careful)
python3 client.py --no-probe # skip the reachability check (use behind
# HTTP proxies that answer on every port)
```
@@ -66,7 +107,10 @@ ones, or add a new one (`n`). Every server you use is saved in
model, so the next start is just Enter-Enter. Addresses without a port use
8420; `host:port`, `http(s)://...` URLs and IPv6 literals are accepted.
After login and model choice you get the agent banner and a chat REPL.
After login and model choice you get the agent banner and a REPL. Type tasks,
not chat messages — "rename the function foo to bar in all files" is a task,
"what is a monad" is just a question the agent answers in place.
Commands:
| command | effect |
@@ -76,14 +120,23 @@ Commands:
| `/models` | list models on the server |
| `/model <n>` | switch model (saved for this server) |
| `/limit` | remaining minutes today (last known) |
| `/auto` | toggle confirmations (auto-approve all) |
| `/state` | auto mode, project dir, history size |
| `/cwd` | show the project directory |
| `/clear` | forget the conversation history |
| `/quit` | exit |
Anything else is sent to the model. The client keeps the last few turns as
context (the daemon API itself is stateless) and renders the reply in the
agent style (`✓` lines for actions the model reports).
The agent keeps the last few finished tasks as context. It stops after 30
model round-trips per task; the daemon API itself is stateless.
## Requirements
- Python 3.11+ (stdlib only; `tomllib` is included)
- ollama listening on the address from `config.toml`
## Tests
```bash
python3 -m unittest test_client.py -v # unit tests (no network)
python3 test_e2e.py # fake ollama + daemon + client
```
a.txt
+50
View File
@@ -0,0 +1,50 @@
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
app.py
+2
View File
@@ -0,0 +1,2 @@
x = 1
x = 1
big.txt
+1
View File
@@ -0,0 +1 @@
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client.py
+643 -56
View File
@@ -3,27 +3,37 @@
Terminal coding-agent front end for the sjcode daemon (server/server.py).
This is a real agent: it can run shell commands and change files in the
directory it was started from (the "project directory"). The model emits
tool calls, the client executes them locally and feeds the real output
back to the model until the task is done. You confirm every command that
runs and every file that is written, unless you pass --yes or toggle
/auto in the session.
On start it asks which server to connect to. Servers you used before are
remembered in ~/.config/sjcode/client.json together with username and the
last model, so next time you just press Enter. New servers are saved too.
last model, so next time you just press Enter.
Flow:
coder -> banner, pick saved server (or add one),
login, pick model, chat REPL
login, pick model, agent REPL
coder 10.0.0.5:8420 -> skip the server picker
coder --model qwen3-coder -> skip the model picker
coder --yes -> auto-approve every tool call (careful)
REPL commands:
/help /server /model /models /limit /clear /quit
/help /server /model /models /limit /auto /state /cwd /clear /quit
"""
from __future__ import annotations
import argparse
import difflib
import getpass
import json
import os
import re
import subprocess
import sys
import time
import urllib.error
@@ -35,8 +45,19 @@ CONFIG_DIR = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config")) /
CONFIG_FILE = CONFIG_DIR / "client.json"
DEFAULT_PORT = 8420
REQUEST_TIMEOUT = 300
HISTORY_TURNS = 10 # prior turns sent with each message for context
REQUEST_TIMEOUT = 600
HISTORY_TURNS = 10 # prior task/summary pairs sent with each task for context
# The project directory the agent works in (where ./coder was started).
CWD = Path.cwd()
MAX_TURNS = 30 # model round-trips per user task
NO_TOOL_LIMIT = 2 # plain-text replies before the turn ends by itself
RUN_TIMEOUT = 300 # seconds per shell command
MAX_RUN_CHARS = 10_000 # shell output fed back / shown, per command
READ_DEFAULT_LIMIT = 250 # lines per read
WRITE_MAX_CHARS = 60_000 # refuse to write bigger files in one go
DIFF_PREVIEW_LINES = 60 # diff lines shown in a confirmation prompt
HOST_RE = re.compile(
r"^(?=.{1,253}\z)" # total length limit
@@ -48,14 +69,77 @@ HOST_RE = re.compile(
re.IGNORECASE,
)
# Makes the model behave like a terminal agent: short lines, check marks.
AGENT_PREAMBLE = (
"You are sj, a coding agent working in the user's terminal.\n"
"Answer concisely and practically.\n"
"Describe what you do as short lines; put every action you performed "
"or verified on its own line starting with '\u2713 ' (check mark).\n"
"End your reply with the single line 'Done.'"
# Tool-call protocol: the model answers with normal text plus fenced blocks.
# ```run
# git status --short
# ```
# ```sj:write
# {"path": "x.txt", "content": "hi"}
# ```
# The closing fence must be a line containing only ``` so that content which
# itself contains backticks (inside a JSON string) cannot end the block early.
# (?![a-z]) keeps language fences like ```rust or ```ruby from being read as
# run-tool calls.
TAG_RE = re.compile(
r"(?ms)^[ \t]*```(?:sj:([a-z]+)|run)(?![a-z])[^\n]*\n(.*?)(?=\n[ \t]*```[ \t]*(?:\n|\Z)|\Z)"
)
STRAY_FENCE_RE = re.compile(r"(?m)^[ \t]*```[^\n]*$")
AGENT_PREAMBLE = """\
You are sj, a real coding agent working in the user's project directory on
their machine. You can actually run commands and change files. Work like an
experienced engineer:
- Plan silently, act step by step. After every step, look at the real output
you got before deciding the next step.
- Explore before you change code you have not seen: list files, read them,
then edit. Never guess file contents.
- Keep replies short. Say what you are doing, then do it.
- Every action you performed or verified goes on its own line starting with
"✓ " (check mark).
- If something fails, read the error and change your approach; do not repeat
a failing call unchanged and do not apologize.
- When the whole task is finished, end your reply with the single line "Done."
You act by emitting tool calls. A tool call is a fenced code block tagged
"run" or "sj:<tool>" whose body is plain text (run) or a JSON object (all
other tools). Examples:
```run
git status --short
```
```sj:write
{"path": "notes/hello.txt", "content": "hi\\n"}
```
```sj:edit
{"path": "app.py", "old": "old text", "new": "new text", "all": false}
```
```sj:read
{"path": "app.py", "start": 1, "limit": 200}
```
```sj:files
{"path": ".", "limit": 200}
```
Tools:
- run: run one shell command in the project directory. Every run starts
in the project directory again (cd does not persist).
- read: read a text file. Args: path, start (line number), limit (lines).
- files: list project files. Args: path (subdirectory), limit.
- write: create a new file or overwrite a small one. Args: path, content.
- edit: replace exact text in a file. Args: path, old, new, all (optional).
Rules:
- Emit at most ONE tool call per reply, then stop. You will receive its real
output as the next message and then continue.
- The user confirms dangerous actions. If a result says "cancelled by user",
accept that, adjust, and move on; never repeat the same call.
- Never invent tool output. If you need information, call a tool to get it.
"""
# ---------------------------------------------------------------------------
@@ -64,6 +148,7 @@ AGENT_PREAMBLE = (
class Style:
def __init__(self, enabled: bool) -> None:
self.enabled = enabled
self.dim = "\033[2m" if enabled else ""
self.bold = "\033[1m" if enabled else ""
self.green = "\033[32m" if enabled else ""
@@ -100,13 +185,13 @@ def banner(model: str, project: str, server: str) -> None:
for line in lines:
print(f"\u2502 {line.ljust(width)} \u2502")
print(f"\u2570{'\u2500' * (width + 2)}\u256f")
print(f"{ST.dim}server: {server} (\u2018/help\u2019 for commands){ST.reset}")
print(f"{ST.dim}server: {server} ('/help' for commands){ST.reset}")
def shorten_home(path: Path) -> str:
def shorten_home(path: Path) -> Path:
home = Path.home()
try:
return "~" / path.relative_to(home)
return Path("~") / path.relative_to(home)
except ValueError:
return path
@@ -171,8 +256,8 @@ def pick_server(state: dict) -> tuple[dict, bool]:
servers = state["servers"]
print(f"\n{ST.bold}Connect to which server?{ST.reset}")
for i, entry in enumerate(servers, 1):
default = " (last)" if server_label(entry) == state.get("last") else ""
user = f" as {entry['username']}" if entry.get("username") else ""
default = f" {ST.dim}(last){ST.reset}" if server_label(entry) == state.get("last") else ""
user = f" {ST.dim}({entry['username']}){ST.reset}" if entry.get("username") else ""
print(f" {i}) {server_label(entry)}{user}{default}")
print(f" n) add a new server\n")
@@ -198,7 +283,7 @@ def pick_server(state: dict) -> tuple[dict, bool]:
def add_server(state: dict) -> dict:
"""Ask for a new server address, probe it and save it."""
"""Ask for a new server address, and save it."""
while True:
raw = ask("server address (domain or ip:port): ").strip()
try:
@@ -218,8 +303,20 @@ def add_server(state: dict) -> dict:
# daemon API
# ---------------------------------------------------------------------------
def _feed_chunk(line: str, on_chunk, final: dict) -> dict:
"""Handle one JSON line of a streamed reply: chunk -> callback, else final."""
try:
obj = json.loads(line)
except json.JSONDecodeError:
return final
if isinstance(obj, dict) and "chunk" in obj:
on_chunk(obj)
return final
return obj if isinstance(obj, dict) else final
def http_json(method: str, base: str, path: str, payload: dict | None = None,
timeout: int = REQUEST_TIMEOUT) -> tuple[int, dict]:
timeout: int = REQUEST_TIMEOUT, on_chunk=None) -> tuple[int, dict]:
url = base + path
data = None if payload is None else json.dumps(payload).encode()
req = urllib.request.Request(
@@ -228,7 +325,21 @@ def http_json(method: str, base: str, path: str, payload: dict | None = None,
)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
if on_chunk is None:
return resp.status, json.loads(resp.read().decode())
# streamed reply: one JSON object per line
status = resp.status
final: dict = {}
buf = ""
for raw in resp:
buf += raw.decode(errors="replace")
while "\n" in buf:
line, buf = buf.split("\n", 1)
if line.strip():
final = _feed_chunk(line.strip(), on_chunk, final)
if buf.strip():
final = _feed_chunk(buf.strip(), on_chunk, final)
return status, final
except urllib.error.HTTPError as exc:
try:
return exc.code, json.loads(exc.read().decode())
@@ -266,10 +377,14 @@ def api_models(entry: dict, token: str) -> list[str]:
return list(body.get("models", [])) if status == 200 else []
def api_chat(entry: dict, token: str, message: str, model: str) -> tuple[int, dict]:
def api_chat(entry: dict, token: str, message: str, model: str,
on_chunk=None) -> tuple[int, dict]:
base = f"http://{entry['host']}:{entry['port']}"
return http_json("POST", base, "/chat",
{"token": token, "message": message, "model": model})
payload: dict = {"token": token, "message": message, "model": model}
if on_chunk is not None:
payload["stream"] = True
return http_json("POST", base, "/chat", payload, on_chunk=on_chunk)
return http_json("POST", base, "/chat", payload)
# ---------------------------------------------------------------------------
@@ -333,48 +448,513 @@ def pick_model(entry: dict, models: list[str]) -> str:
# ---------------------------------------------------------------------------
# the agent look
# tools the agent can use (executed locally, in the project directory)
# ---------------------------------------------------------------------------
class ToolError(Exception):
"""A tool refused to run; the message goes back to the model."""
class ToolAbort(Exception):
"""The user aborted the whole agent turn at a confirmation prompt."""
ABORT = "!!abort"
EXCLUDED_DIRS = {
".git", "__pycache__", "node_modules", ".venv", "venv", ".mypy_cache",
".ruff_cache", ".pytest_cache", ".tox", ".idea", ".vscode", ".eggs",
"dist", "build", ".hg", ".svn",
}
def _resolve(path: str) -> Path:
"""Resolve a path against the project dir; refuse anything outside it."""
if not path:
raise ToolError("path required")
p = Path(path).expanduser()
if not p.is_absolute():
p = CWD / p
try:
rp = p.resolve()
except OSError as exc:
raise ToolError(f"cannot resolve path: {exc}")
inside = rp == CWD or str(rp).startswith(str(CWD) + os.sep)
if not inside:
raise ToolError(
f"outside the project directory ({CWD}): {path!r}")
return rp
def _display(p: Path) -> str:
try:
rel = str(p.relative_to(CWD))
return rel or "."
except ValueError:
return str(p)
def _unified_diff(old: str, new: str, display: str) -> list[str]:
diff = difflib.unified_diff(
old.splitlines(), new.splitlines(),
fromfile=f"{display} (old)", tofile=f"{display} (new)", lineterm="",
)
lines = list(diff)
if len(lines) > DIFF_PREVIEW_LINES:
lines = lines[:DIFF_PREVIEW_LINES]
lines.append(f"... (+{len(lines)} more diff lines)")
return lines
def _run_args(args: dict) -> tuple[str, int, Path]:
"""Validate run-tool args -> (command, timeout, working dir)."""
cmd_text = str(args.get("command") or args.get("cmd") or "").strip()
if not cmd_text:
raise ToolError("'command' required")
timeout = args.get("timeout")
try:
timeout = min(max(int(timeout), 5), RUN_TIMEOUT) if timeout else RUN_TIMEOUT
except (TypeError, ValueError):
timeout = RUN_TIMEOUT
wd = _resolve(str(args.get("cwd") or "."))
if not wd.is_dir():
raise ToolError(f"not a directory: {_display(wd)}")
return cmd_text, timeout, wd
def tool_run_confirm(args: dict) -> list[str]:
cmd_text, timeout, wd = _run_args(args)
return [
f"$ {cmd_text}",
f"cwd: {_display(wd)} timeout: {timeout}s",
]
def tool_run_execute(args: dict) -> list[str]:
cmd_text, timeout, wd = _run_args(args)
try:
proc = subprocess.run(
cmd_text, shell=True, cwd=str(wd), capture_output=True,
text=True, timeout=timeout,
)
except subprocess.TimeoutExpired:
return [f"\u2717 $ {cmd_text}", f"timed out after {timeout}s"]
except OSError as exc:
return [f"\u2717 $ {cmd_text}", f"cannot run: {exc}"]
ok = proc.returncode == 0
head = (f"\u2713 $ {cmd_text}" if ok
else f"\u2717 $ {cmd_text} (exit {proc.returncode})")
parts = []
if (proc.stdout or "").strip():
parts.append(proc.stdout.rstrip())
if (proc.stderr or "").strip():
parts.append(proc.stderr.rstrip())
text = "\n".join(parts).strip()
if len(text) > MAX_RUN_CHARS:
text = text[:MAX_RUN_CHARS] + f"\n... (output truncated at {MAX_RUN_CHARS} chars)"
return [head] + (text.splitlines() if text else [])
def tool_read(args: dict) -> list[str]:
path = str(args.get("path", "")).strip()
p = _resolve(path)
if not p.is_file():
raise ToolError(f"not a file: {_display(p)}")
try:
text = p.read_text(encoding="utf-8", errors="replace")
except OSError as exc:
raise ToolError(str(exc))
lines = text.splitlines()
total = len(lines)
try:
start = max(1, int(args.get("start", 1)))
except (TypeError, ValueError):
start = 1
try:
limit = int(args.get("limit", READ_DEFAULT_LIMIT))
except (TypeError, ValueError):
limit = READ_DEFAULT_LIMIT
limit = max(1, min(limit, 1000))
start = min(start, max(total, 1))
sel = lines[start - 1:start - 1 + limit]
out = [f"\u2713 read {_display(p)} ({total} lines)"]
for i, line in enumerate(sel, start):
out.append(f"{i}\t{line[:400]}")
if start - 1 + limit < total:
out.append(f"(... {total - (start - 1 + limit)} more lines - "
f"read again with 'start'/'limit')")
return out
def tool_files(args: dict) -> list[str]:
sub = str(args.get("path") or ".").strip()
base = _resolve(sub)
if not base.is_dir():
raise ToolError(f"not a directory: {_display(base)}")
try:
limit = max(1, min(int(args.get("limit", 200)), 500))
except (TypeError, ValueError):
limit = 200
out: list[str] = []
truncated = False
for root, dirs, fnames in os.walk(base):
dirs[:] = sorted(
d for d in dirs
if d not in EXCLUDED_DIRS and not d.endswith(".egg-info")
)
for fn in sorted(fnames):
if len(out) >= limit:
truncated = True
break
out.append(_display(Path(root) / fn))
if truncated:
break
head = (f"\u2713 files under {_display(base)} ({len(out)} shown"
+ (", truncated - narrow with 'path'" if truncated else "") + ")")
return [head] + out
def tool_write_confirm(args: dict) -> list[str]:
path = str(args.get("path", "")).strip()
if not path:
raise ToolError("path required")
content = args.get("content")
if not isinstance(content, str):
raise ToolError("'content' (string) required")
if len(content) > WRITE_MAX_CHARS:
raise ToolError(
f"content too large ({len(content)} chars, max {WRITE_MAX_CHARS}) - "
f"write it in smaller pieces or use 'run'")
p = _resolve(path)
exists = p.is_file()
if exists:
old = p.read_text(encoding="utf-8", errors="replace")
if len(old) > 2000:
raise ToolError(
f"{_display(p)} exists and is {len(old)} chars; refusing to "
f"overwrite it wholesale - use 'edit' instead")
else:
old = ""
summary = [
f"{'overwrite' if exists else 'create'} {_display(p)} "
f"({len(content.splitlines())} lines)"
]
summary += _unified_diff(old, content, _display(p))
return summary
def tool_write_execute(args: dict) -> list[str]:
p = _resolve(str(args.get("path", "")).strip())
content = str(args.get("content", ""))
p.parent.mkdir(parents=True, exist_ok=True)
text = content if content.endswith("\n") else content + "\n"
p.write_text(text, encoding="utf-8")
return [f"\u2713 wrote {_display(p)} ({len(text.splitlines())} lines)"]
def _flag(args: dict, key: str) -> bool:
return args.get(key) in (True, "true", "True", 1, "1", "yes")
def tool_edit_confirm(args: dict) -> list[str]:
path = str(args.get("path", "")).strip()
if not path:
raise ToolError("path required")
old, new = args.get("old"), args.get("new")
if not isinstance(old, str) or not old:
raise ToolError("'old' (exact existing text) required")
if not isinstance(new, str):
raise ToolError("'new' (replacement text) required")
p = _resolve(path)
if not p.is_file():
raise ToolError(f"not a file: {_display(p)}")
body = p.read_text(encoding="utf-8", errors="replace")
n = body.count(old)
if n == 0:
raise ToolError(
f"search text not found in {_display(p)} - 'read' the file and "
f"copy the text exactly")
all_flag = _flag(args, "all")
if n > 1 and not all_flag:
raise ToolError(
f"'old' matches {n} times in {_display(p)} - include more "
f"surrounding text or set \"all\": true")
updated = body.replace(old, new) if all_flag else body.replace(old, new, 1)
summary = [
f"edit {_display(p)} "
f"({'all' if all_flag else 'first'} of {n} match{'es' if n != 1 else ''})"
]
summary += _unified_diff(body, updated, _display(p))
return summary
def tool_edit_execute(args: dict) -> list[str]:
p = _resolve(str(args.get("path", "")).strip())
old, new = str(args.get("old", "")), str(args.get("new", ""))
all_flag = _flag(args, "all")
body = p.read_text(encoding="utf-8", errors="replace")
updated = body.replace(old, new) if all_flag else body.replace(old, new, 1)
p.write_text(updated, encoding="utf-8")
return [f"\u2713 edited {_display(p)}"]
# name -> {confirm, execute, confirm_first}
TOOLS: dict[str, dict] = {
"run": {"confirm": tool_run_confirm, "execute": tool_run_execute, "confirm_first": True},
"read": {"confirm": tool_read, "execute": tool_read, "confirm_first": False},
"files": {"confirm": tool_files, "execute": tool_files, "confirm_first": False},
"write": {"confirm": tool_write_confirm, "execute": tool_write_execute, "confirm_first": True},
"edit": {"confirm": tool_edit_confirm, "execute": tool_edit_execute, "confirm_first": True},
}
def confirm_tool(name: str, summary: list[str], auto: bool) -> str:
"""Ask the user to approve a tool call.
Returns "y", "n", ABORT, or a free-text instruction from the user.
"""
if auto:
return "y"
print(f"{ST.bold}{ST.yellow}? {name}{ST.reset}")
for line in summary:
print(f" {line}")
try:
answer = input(
f" {ST.bold}[Enter=yes n=no a=abort or type a new instruction]{ST.reset} "
).strip()
except KeyboardInterrupt:
print()
return "n"
except EOFError:
print()
return ABORT
low = answer.lower()
if low in ("", "y", "yes"):
return "y"
if low in ("n", "no", "c", "cancel"):
return "n"
if low in ("a", "abort"):
return ABORT
return answer
def handle_tool_call(name: str, raw_body: str, auto: bool) -> tuple[bool, list[str]]:
"""Validate, confirm and execute one tool call. Returns (ok, lines)."""
spec = TOOLS.get(name)
if spec is None:
return False, [f"\u2717 unknown tool '{name}' - "
f"available: {', '.join(sorted(TOOLS))}"]
if name == "run":
# Accept the plain-text command (as documented) or a JSON object
# with a "command" key - some models emit JSON anyway.
body = (raw_body or "").strip()
args = {"command": body}
if body.startswith("{"):
try:
parsed = json.loads(body)
if isinstance(parsed, dict) and "command" in parsed:
args = parsed
except json.JSONDecodeError:
pass # e.g. a brace group in shell - run it as-is
else:
try:
args = json.loads(raw_body or "{}")
if not isinstance(args, dict):
raise ValueError("arguments must be a JSON object")
except json.JSONDecodeError as exc:
return False, [f"\u2717 bad JSON for '{name}': {exc} - fix it and retry"]
except ValueError as exc:
return False, [f"\u2717 {exc}"]
try:
prepared = spec["confirm"](args) # validates; for read/files it already runs
except ToolError as exc:
return False, [f"\u2717 {exc}"]
except (KeyError, TypeError, ValueError, OSError) as exc:
return False, [f"\u2717 {exc}"]
if not spec["confirm_first"]: # read/files: safe, results are in
return True, prepared
decision = "y" if auto else confirm_tool(name, prepared, auto)
if decision == ABORT:
raise ToolAbort()
if decision != "y":
note = "cancelled by user" if decision == "n" else f"user says: {decision}"
return False, [f"\u2717 {note} - do not repeat this exact call; "
f"adjust your approach or stop"]
try:
lines = spec["execute"](args)
# A failing command is not a tool error, but the model must see it
# as a failure: the first output line starts with the cross mark.
ok = not (lines and lines[0].startswith("\u2717"))
return ok, lines
except ToolError as exc:
return False, [f"\u2717 {exc}"]
except (OSError, ValueError, RuntimeError) as exc:
return False, [f"\u2717 {exc}"]
# ---------------------------------------------------------------------------
# agent turn: model reply -> tool calls -> real output -> model ...
# ---------------------------------------------------------------------------
def read_tool_tags(raw: str) -> list[tuple[str, str]]:
"""Extract tool calls in order: [(name, body), ...]."""
tags: list[tuple[str, str]] = []
for m in TAG_RE.finditer(raw):
name = m.group(1) or "run"
tags.append((name, (m.group(2) or "").strip()))
return tags
def print_agent_reply(text: str) -> None:
print(f"{ST.bold}Agent:{ST.reset}")
for line in text.rstrip().splitlines():
stripped = line.strip()
if stripped.startswith("\u2713"):
print(f" {ST.green}\u2713{ST.reset} {stripped.lstrip('\u2713').strip()}")
elif stripped.lower() == "done.":
print(f" {ST.bold}{stripped}{ST.reset}")
if line.startswith("\u2713"):
print(f"{ST.green}{line}{ST.reset}")
elif line.startswith("\u2717"):
print(f"{ST.red}{line}{ST.reset}")
else:
print(f" {line.strip() if stripped else ''}")
print(line)
def build_prompt(history: list[tuple[str, str]], message: str) -> str:
"""Plain-text transcript + new user message (server endpoint is stateless)."""
parts = [AGENT_PREAMBLE]
def process_reply(raw: str, auto: bool, show_text: bool = True
) -> tuple[bool, str, list]:
"""Print the reply, run its tool calls. Returns (had_tools, text, results)."""
# Tool blocks out, leftover closing-fence lines out, blank edges trimmed.
reply_text = STRAY_FENCE_RE.sub("", TAG_RE.sub("", raw)).strip()
if reply_text and show_text: # already shown live when streamed
print_agent_reply(reply_text)
tags = read_tool_tags(raw)
if not tags:
return False, reply_text, []
results: list[tuple[str, bool, list[str]]] = []
for name, body in tags:
print(f"{ST.dim}\u2014 tool: {name}{ST.reset}")
ok, lines = handle_tool_call(name, body, auto)
results.append((name, ok, lines))
for i, line in enumerate(lines):
if i == 0 and not ok:
print(f"{ST.red}{line}{ST.reset}")
elif line.startswith("\u2713"):
print(f"{ST.green}{line}{ST.reset}")
else:
print(f" {line}")
return True, reply_text, results
def agent_turn(entry: dict, token: str, model: str,
history: list[tuple[str, str]], task: str, auto: bool
) -> tuple[str, str, float | None]:
"""Run one user task through the agent loop.
Returns (status, final_text, remaining_minutes):
status is "done", "stopped", "cancelled", "limit" or "error".
"""
transcript = ""
for user_text, answer in history[-HISTORY_TURNS:]:
parts.append(f"User: {user_text}\nAssistant: {answer}")
parts.append(f"User: {message}\nAssistant:")
return "\n\n".join(parts)
transcript += f"User: {user_text}\nAssistant: {answer}\n\n"
messages = [AGENT_PREAMBLE
+ (("\n\n" + transcript) if transcript else "")
+ f"\n\nUser: {task}\nAssistant:"]
remaining: float | None = None
no_tool = 0
streamed = {"any": False}
def on_chunk(obj: dict) -> None:
piece = obj.get("chunk")
if not piece:
return
if not streamed["any"]:
streamed["any"] = True
print(f"{ST.bold}Agent:{ST.reset}")
sys.stdout.write(f"{ST.dim}{piece}{ST.reset}" if ST.enabled else piece)
sys.stdout.flush()
try:
for turn in range(1, MAX_TURNS + 1):
try:
status, body = api_chat(
entry, token, "\n\n".join(messages), model,
on_chunk=on_chunk)
except KeyboardInterrupt:
print()
return "cancelled", "", remaining
if status != 200 or body.get("error"):
print(f"\n{ST.red}error: "
f"{body.get('error', 'unknown')}{ST.reset}")
if status == 403:
return "limit", "", remaining
return "error", "", remaining
remaining = body.get("remaining_minutes", remaining)
raw = str(body.get("response", ""))
if streamed["any"]:
print()
had_tools, reply_text, results = process_reply(
raw, auto, show_text=not streamed["any"])
streamed["any"] = False
if not had_tools:
no_tool += 1
if no_tool >= NO_TOOL_LIMIT:
return "done", reply_text, remaining
messages.append(
"(no tool call received) If the task is complete, reply "
"with exactly: Done.\nOtherwise continue with the next "
"tool call.")
continue
no_tool = 0
if turn >= MAX_TURNS:
print(f"{ST.yellow}step limit ({MAX_TURNS}) reached - "
f"stopping{ST.reset}")
return "stopped", reply_text, remaining
out = []
for name, ok, lines in results:
out.append(f"{name}: {'ok' if ok else 'FAILED'}")
out.extend(" " + ln for ln in lines)
messages.append(
"Tool results (real output from your last tool call):\n"
+ ("\n".join(out) or "(no output)")
+ "\n\nContinue with the next step.")
except ToolAbort:
print(f"{ST.yellow}stopped by user{ST.reset}")
return "stopped", "", remaining
except KeyboardInterrupt:
print()
return "cancelled", "", remaining
print(f"{ST.yellow}step limit ({MAX_TURNS}) reached - stopping{ST.reset}")
return "stopped", "", remaining
# ---------------------------------------------------------------------------
# chat REPL
# ---------------------------------------------------------------------------
HELP = """
HELP = f"""\
commands:
/help show this help
/server show the connected server
/models list models on the server
/model <name> switch model (also saved for this server)
/limit show remaining minutes today (last known)
/auto toggle confirmations (auto-approve every tool call)
/state show auto mode, project dir, history size
/cwd show the project directory
/clear forget the conversation history
/quit exit (also Ctrl+D)
anything else is sent to the model.
anything else is a task for the agent. It can run commands and change files
in the project directory; you confirm every action unless auto mode is on.
available tools: {', '.join(sorted(TOOLS))}
"""
def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> None:
def chat_loop(entry: dict, token: str, model: str,
remaining: float | None, auto: bool) -> None:
history: list[tuple[str, str]] = []
while True:
try:
@@ -409,6 +989,18 @@ def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> N
elif cmd == "/limit":
left = "unlimited" if remaining is None else f"{remaining:g} min"
print(f"time left today: {left}")
elif cmd == "/auto":
auto = not auto
print("auto mode ON - tool calls run without confirmation"
if auto else
"auto mode OFF - every action asks first")
elif cmd == "/state":
print(f"auto: {'on' if auto else 'off'}")
print(f"project dir: {CWD}")
print(f"history: {len(history)} turn(s)")
print(f"model: {model or '(daemon default)'}")
elif cmd == "/cwd":
print(str(CWD))
elif cmd == "/clear":
history.clear()
print("history cleared")
@@ -416,23 +1008,12 @@ def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> N
print(f"unknown command {cmd} - try /help")
continue
try:
status, body = api_chat(entry, token, build_prompt(history, message), model)
except KeyboardInterrupt:
print(f"\n{ST.yellow}request cancelled{ST.reset}")
continue
if status != 200:
print(f"{ST.red}error: {body.get('error', 'unknown')}{ST.reset}")
if status == 403:
status, final, remaining = agent_turn(
entry, token, model, history, message, auto)
if status == "limit":
return # daily limit reached
continue
answer = str(body.get("response", "")).strip() or "(empty reply)"
history.append((message, answer))
print()
print_agent_reply(answer)
remaining = body.get("remaining_minutes")
if status == "done" and final:
history.append((message, final))
if remaining is not None:
print(f"{ST.dim} ({remaining:g} min left today){ST.reset}")
print()
@@ -453,6 +1034,9 @@ def main() -> None:
parser.add_argument("--no-probe", action="store_true",
help="skip the reachability check before login (use behind "
"HTTP proxies that answer for every port)")
parser.add_argument("-y", "--yes", action="store_true",
help="auto mode: run tool calls without confirmation "
"(the agent edits real files - be careful)")
args = parser.parse_args()
state = load_state()
@@ -504,9 +1088,12 @@ def main() -> None:
state["last"] = label
save_state(state)
project = shorten_home(Path.cwd())
project = shorten_home(CWD)
banner(model or "(server default)", str(project), label)
chat_loop(entry, token, model, None)
if args.yes:
print(f"{ST.yellow}auto mode: tool calls run without confirmation{ST.reset}")
print(f"{ST.dim}working dir: {CWD}{ST.reset}")
chat_loop(entry, token, model, None, args.yes)
print("bye")
f.txt
+1
View File
@@ -0,0 +1 @@
a
fake_ollama.py
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""Fake ollama for end-to-end tests of the sjcode daemon + client.
Serves a scripted conversation: every /api/generate call returns the next
reply from SCRIPT. Each prompt it receives is appended to prompts.log so
the test can assert what the client actually fed back to the model.
Usage: python3 fake_ollama.py PORT PROMPT_LOG
"""
import json
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
PORT = int(sys.argv[1])
LOG = sys.argv[2]
RUN_REPLY = (
"I will create the file.\n"
"```run\n"
"printf 'hi' > hello.txt\n"
"```\n"
)
READ_REPLY = (
"Checking it.\n"
"```sj:read\n"
'{"path": "hello.txt"}\n'
"```\n"
)
DONE_REPLY = "The file is in place.\nDone."
SCRIPT = [RUN_REPLY, READ_REPLY, DONE_REPLY]
calls = {"n": 0}
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args): # quiet
pass
def _json(self, obj):
body = json.dumps(obj).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path == "/api/tags":
self._json({"models": [{"name": "fake-model"}]})
else:
self._json({"error": "not found"})
def do_POST(self):
if self.path != "/api/generate":
self._json({"error": "not found"})
return
length = int(self.headers.get("Content-Length", "0"))
payload = json.loads(self.rfile.read(length).decode())
with open(LOG, "a") as fh:
fh.write(json.dumps(payload.get("prompt", "")) + "\n")
i = min(calls["n"], len(SCRIPT) - 1)
calls["n"] += 1
self._json({"model": "fake-model", "response": SCRIPT[i], "done": True})
if __name__ == "__main__":
ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()
greet.txt
+1
View File
@@ -0,0 +1 @@
hi
pkg/mod.py
+1
View File
@@ -0,0 +1 @@
x = 1
server/config.toml.example
+3
View File
@@ -12,6 +12,9 @@ host = "127.0.0.1"
base_url = "http://127.0.0.1:11434"
# Default model used when a client does not request a specific one.
model = "llama3.2"
# Context window the daemon requests from ollama. The agent sends long
# transcripts (system prompt + tool results), so do not drop this below 8192.
num_ctx = 16384
[database]
# SQLite file for user accounts. Relative to the project root.
server/server.py
+115 -14
View File
@@ -24,6 +24,7 @@ import cmd
import hashlib
import hmac
import json
import os
import sqlite3
import sys
import threading
@@ -35,7 +36,9 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
ROOT = Path(__file__).resolve().parent
CONFIG_FILE = ROOT / "config.toml"
# SJCODE_CONFIG lets you run a second daemon (tests, staging) without
# touching the regular config; the database is created next to that file.
CONFIG_FILE = Path(os.environ.get("SJCODE_CONFIG") or (ROOT / "config.toml"))
DB_FILE = ROOT / "sjcode.db" # overwritten by config value once loaded
PEPPER = b"sjcode-v1"
@@ -43,7 +46,8 @@ SESSION_TTL = 24 * 3600 # login token valid for one day
CONF: dict = {
"server": {"host": "127.0.0.1", "port": 8420},
"ollama": {"base_url": "http://127.0.0.1:11434", "model": "llama3.2"},
"ollama": {"base_url": "http://127.0.0.1:11434", "model": "llama3.2",
"num_ctx": 16384},
"database": {"path": "sjcode.db"},
"limits": {"user": 120, "admin": -1},
}
@@ -66,7 +70,7 @@ def load_config() -> None:
for section, values in loaded.items():
if isinstance(values, dict) and isinstance(CONF.get(section), dict):
CONF[section].update(values)
DB_FILE = ROOT / CONF["database"]["path"]
DB_FILE = CONFIG_FILE.parent / CONF["database"]["path"]
# --------------------------------------------------------------------------
@@ -232,15 +236,62 @@ def ollama_json(path: str, payload: dict | None = None, timeout: int = 300) -> t
return 502, {"error": f"ollama not reachable: {exc}"}
def ollama_generate(prompt: str, model: str | None) -> tuple[int, str]:
status, body = ollama_json(
"/api/generate",
{
def ollama_stream(path: str, payload: dict) -> tuple[int, object]:
"""POST to ollama and return the raw response to iterate line by line."""
url = CONF["ollama"]["base_url"].rstrip("/") + path
data = json.dumps(payload).encode()
req = urllib.request.Request(
url, data=data, headers={"Content-Type": "application/json"}
)
try:
resp = urllib.request.urlopen(req, timeout=600)
except urllib.error.HTTPError as exc:
try:
body = json.loads(exc.read().decode())
except Exception:
body = {"error": str(exc)}
return exc.code, str(body.get("error", "ollama error"))
except (urllib.error.URLError, TimeoutError, ConnectionError) as exc:
return 502, {"error": f"ollama not reachable: {exc}"}
return resp.status, resp # caller iterates the lines and closes it
def ollama_generate(prompt: str, model: str | None, stream: bool = False):
"""Generate a reply. Returns (status, text) or (200, token-iterator)."""
options = {"num_ctx": int(CONF["ollama"].get("num_ctx", 16384))}
payload = {
"model": model or CONF["ollama"]["model"],
"prompt": prompt,
"stream": False,
},
)
"stream": stream,
"options": options,
}
if stream:
status, body = ollama_stream("/api/generate", payload)
if status != 200:
return status, str(body)
def tokens():
try:
for raw in body:
line = raw.decode() if isinstance(raw, bytes) else raw
if not line.strip():
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if obj.get("error"):
yield f"\n[ollama error: {obj['error']}]"
return
piece = obj.get("response", "")
if piece:
yield piece
finally:
body.close()
return 200, tokens()
status, body = ollama_json("/api/generate", payload)
if status != 200:
return status, str(body.get("error", "ollama error"))
return 200, str(body.get("response", ""))
@@ -259,7 +310,8 @@ def list_models() -> tuple[int, list[str]]:
# --------------------------------------------------------------------------
class ApiHandler(BaseHTTPRequestHandler):
server_version = "sjcode-daemon/0.1"
server_version = "sjcode-daemon/0.2"
protocol_version = "HTTP/1.1" # required for chunked streamed replies
# -- helpers ----------------------------------------------------------
def _json(self, status: int, payload: dict) -> None:
@@ -277,8 +329,48 @@ class ApiHandler(BaseHTTPRequestHandler):
except (ValueError, json.JSONDecodeError):
return {}
def log_message(self, fmt: str, *args) -> None: # quiet by default
return
def _respond_stream(self, tokens, model: str, started: float,
username: str) -> None:
"""Stream a reply as JSON lines: {"chunk": ...} ... {"response": ...}."""
self.send_response(200)
self.send_header("Content-Type", "application/x-ndjson")
self.send_header("Transfer-Encoding", "chunked")
self.end_headers()
def send_line(obj: dict) -> None:
data = (json.dumps(obj) + "\n").encode()
self.wfile.write(f"{len(data):x}\r\n".encode() + data + b"\r\n")
self.wfile.flush()
parts: list[str] = []
abandoned = False
try:
for piece in tokens:
parts.append(piece)
send_line({"chunk": piece})
except (BrokenPipeError, ConnectionResetError):
abandoned = True # client went away; still bill the usage below
except OSError as exc:
parts.append(f"\n[stream error: {exc}]")
send_line({"chunk": f"\n[stream error: {exc}]"})
finally:
text = "".join(parts)
elapsed = time.monotonic() - started
used_min = elapsed / 60.0
add_used_minutes(username, used_min)
if not abandoned:
left_now = remaining_minutes(username)
send_line({
"response": text,
"model": model,
"used_minutes": round(used_min, 4),
"remaining_minutes": None if left_now == float("inf")
else round(left_now, 2),
})
self.wfile.write(b"0\r\n\r\n")
self.wfile.flush()
print(f"[chat] {username}: {len(text)} chars in "
f"{elapsed:.1f}s (streamed)")
# -- routes -----------------------------------------------------------
def do_GET(self) -> None: # noqa: N802 (http.server API)
@@ -353,9 +445,15 @@ class ApiHandler(BaseHTTPRequestHandler):
return
model = str(data.get("model") or "") or CONF["ollama"]["model"]
want_stream = bool(data.get("stream"))
started = time.monotonic()
status, answer = ollama_generate(message, model)
status, answer = ollama_generate(message, model, stream=want_stream)
if want_stream and status == 200:
self._respond_stream(answer, model, started, username)
return
elapsed = time.monotonic() - started
if status != 200:
@@ -375,6 +473,9 @@ class ApiHandler(BaseHTTPRequestHandler):
},
)
def log_message(self, fmt: str, *args) -> None: # quiet by default
return
# --------------------------------------------------------------------------
# admin CLI
server/sjcode.db-shm
BIN
View File
Binary file not shown.
sub/marker.txt
+1
View File
@@ -0,0 +1 @@
x
test_client.py
+277
View File
@@ -0,0 +1,277 @@
#!/usr/bin/env python3
"""Tests for the sjcode client agent layer.
Run: python3 -m unittest test_client.py -v
"""
from __future__ import annotations
import json
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parent))
import client as c # noqa: E402
class ClientTestCase(unittest.TestCase):
"""Each test runs in its own temp dir which becomes the project dir."""
def setUp(self) -> None:
self._old_cwd = c.CWD
self._old_chdir = os.getcwd()
self._tmp = tempfile.TemporaryDirectory()
c.CWD = Path(self._tmp.name).resolve()
os.chdir(c.CWD) # tools use CWD; keep relative paths consistent
def tearDown(self) -> None:
os.chdir(self._old_chdir)
c.CWD = self._old_cwd
self._tmp.cleanup()
class TestParsing(ClientTestCase):
def test_extracts_run_and_json_tools_in_order(self) -> None:
raw = (
"Let me check the repo.\n"
"```run\n"
"git status --short\n"
"```\n"
"And now the file:\n"
"```sj:read\n"
'{"path": "a.py", "limit": 10}\n'
"```\n"
)
tags = c.read_tool_tags(raw)
self.assertEqual([("run", "git status --short"),
("read", '{"path": "a.py", "limit": 10}')], tags)
def test_language_fence_is_run_tool(self) -> None:
self.assertEqual(c.read_tool_tags("```run\necho hi\n```"),
[("run", "echo hi")])
def test_language_fence_is_not_a_run_tool(self) -> None:
self.assertEqual(c.read_tool_tags("```rust\nfn main() {}\n```"), [])
def test_text_before_and_after_is_stripped_from_reply(self) -> None:
raw = "Thinking...\n```run\necho hi\n```\nDone."
had, reply, results = c.process_reply(raw, auto=True)
self.assertTrue(had)
self.assertNotIn("```", reply)
self.assertNotIn("echo hi", reply)
def test_body_with_triple_backticks_inside_json_survives(self) -> None:
body = '{"path": "x.md", "content": "code: ```py\\nprint(1)\\n```\\n"}'
raw = f"```sj:write\n{body}\n```\n"
self.assertEqual(c.read_tool_tags(raw), [("write", body)])
def test_unclosed_fence_still_captured(self) -> None:
self.assertEqual(c.read_tool_tags("```run\necho hi"),
[("run", "echo hi")])
class TestRunTool(ClientTestCase):
def test_runs_command_and_returns_output(self) -> None:
ok, lines = c.handle_tool_call("run", '{"command": "echo hello"}', auto=True)
self.assertTrue(ok)
self.assertIn("hello", "\n".join(lines))
def test_nonzero_exit_reported(self) -> None:
ok, lines = c.handle_tool_call(
"run", '{"command": "python3 -c \x27import sys; sys.exit(3)\x27"}',
auto=True)
self.assertFalse(ok)
self.assertIn("exit 3", "\n".join(lines))
def test_cwd_argument_scopes_the_command(self) -> None:
os.mkdir("sub")
Path("sub/marker.txt").write_text("x")
ok, lines = c.handle_tool_call(
"run", '{"command": "pwd", "cwd": "sub"}', auto=True)
self.assertTrue(ok)
self.assertIn("sub", "\n".join(lines))
def test_run_outside_project_dir_is_refused(self) -> None:
ok, lines = c.handle_tool_call(
"run", json.dumps({"command": "pwd", "cwd": "/tmp"}), auto=True)
self.assertFalse(ok)
self.assertIn("outside the project directory", "\n".join(lines))
def test_missing_command_is_refused(self) -> None:
ok, lines = c.handle_tool_call("run", "{}", auto=True)
self.assertFalse(ok)
self.assertIn("command", "\n".join(lines))
class TestFileTools(ClientTestCase):
def test_read_shows_numbered_lines(self) -> None:
Path("a.txt").write_text("one\ntwo\nthree\n")
ok, lines = c.handle_tool_call("read", '{"path": "a.txt"}', auto=True)
self.assertTrue(ok)
joined = "\n".join(lines)
self.assertIn("1\tone", joined)
self.assertIn("3\tthree", joined)
def test_read_supports_start_and_limit(self) -> None:
Path("a.txt").write_text("\n".join(str(i) for i in range(1, 51)))
ok, lines = c.handle_tool_call(
"read", '{"path": "a.txt", "start": 40, "limit": 5}', auto=True)
self.assertIn("40\t40", "\n".join(lines))
def test_read_missing_file_gives_error_back(self) -> None:
ok, lines = c.handle_tool_call("read", '{"path": "nope.txt"}', auto=True)
self.assertFalse(ok)
self.assertIn("not a file", "\n".join(lines))
def test_files_lists_and_skips_junk_dirs(self) -> None:
os.mkdir("pkg")
Path("pkg/mod.py").write_text("x = 1\n")
os.mkdir(".git")
Path(".git/config").write_text("x\n")
ok, lines = c.handle_tool_call("files", '{"path": "."}', auto=True)
joined = "\n".join(lines)
self.assertIn("pkg/mod.py", joined)
self.assertNotIn(".git/config", joined)
def test_write_creates_file_after_confirmation(self) -> None:
with mock.patch("client.confirm_tool", return_value="y"):
ok, lines = c.handle_tool_call(
"write", '{"path": "new.txt", "content": "hi\\n"}', auto=False)
self.assertTrue(ok)
self.assertEqual(Path("new.txt").read_text(), "hi\n")
def test_write_refused_outside_project(self) -> None:
ok, lines = c.handle_tool_call(
"write",
json.dumps({"path": "../../evil.txt", "content": "x"}),
auto=True)
self.assertFalse(ok)
self.assertIn("outside the project directory", "\n".join(lines))
def test_edit_replaces_exact_text(self) -> None:
Path("app.py").write_text("def main():\n print('old')\n")
with mock.patch("client.confirm_tool", return_value="y"):
ok, lines = c.handle_tool_call(
"edit",
json.dumps({"path": "app.py", "old": "print('old')",
"new": "print('new')"}),
auto=False)
self.assertTrue(ok)
self.assertIn("print('new')", Path("app.py").read_text())
def test_edit_requires_unique_match_without_all(self) -> None:
Path("app.py").write_text("x = 1\nx = 1\n")
ok, lines = c.handle_tool_call(
"edit", json.dumps({"path": "app.py", "old": "x = 1", "new": "y"}),
auto=True)
self.assertFalse(ok)
self.assertIn("matches 2 times", "\n".join(lines))
def test_edit_with_all_flag_replaces_every_match(self) -> None:
Path("app.py").write_text("x = 1\nx = 1\n")
with mock.patch("client.confirm_tool", return_value="y"):
ok, lines = c.handle_tool_call(
"edit",
json.dumps({"path": "app.py", "old": "x = 1", "new": "y",
"all": True}),
auto=False)
self.assertTrue(ok)
self.assertEqual(Path("app.py").read_text(), "y\ny\n")
def test_edit_not_found_gives_actionable_error(self) -> None:
Path("app.py").write_text("hello\n")
ok, lines = c.handle_tool_call(
"edit", json.dumps({"path": "app.py", "old": "zzz", "new": "y"}),
auto=True)
self.assertFalse(ok)
self.assertIn("search text not found", "\n".join(lines))
def test_overwrite_of_big_existing_file_refused(self) -> None:
Path("big.txt").write_text("x" * 3000)
ok, lines = c.handle_tool_call(
"write", json.dumps({"path": "big.txt", "content": "y"}), auto=True)
self.assertFalse(ok)
self.assertIn("use 'edit' instead", "\n".join(lines))
class TestConfirmation(ClientTestCase):
def test_declined_call_returns_cancelled_to_model(self) -> None:
Path("f.txt").write_text("a\n")
with mock.patch("client.confirm_tool", return_value="n"):
ok, lines = c.handle_tool_call(
"edit", json.dumps({"path": "f.txt", "old": "a", "new": "b"}),
auto=False)
self.assertFalse(ok)
self.assertIn("cancelled by user", "\n".join(lines))
def test_user_instruction_is_passed_to_model(self) -> None:
Path("f.txt").write_text("a\n")
with mock.patch("client.confirm_tool",
return_value="use python instead"):
ok, lines = c.handle_tool_call("run", '{"command": "ls -la"}',
auto=False)
self.assertFalse(ok)
self.assertIn("user says: use python instead", "\n".join(lines))
def test_abort_raises_toolabort(self) -> None:
with mock.patch("client.confirm_tool", return_value=c.ABORT):
with self.assertRaises(c.ToolAbort):
c.handle_tool_call("run", '{"command": "ls"}', auto=False)
def test_auto_mode_skips_confirmation(self) -> None:
with mock.patch("client.confirm_tool",
side_effect=AssertionError("must not be called")):
ok, lines = c.handle_tool_call("run", '{"command": "echo ok"}',
auto=True)
self.assertTrue(ok)
class TestAgentTurn(ClientTestCase):
def _chat(self, replies):
"""Patch api_chat to serve canned replies in order."""
calls = {"n": 0}
def fake(entry, token, message, model, on_chunk=None):
i = calls["n"]
calls["n"] += 1
return 200, {"response": replies[min(i, len(replies) - 1)],
"remaining_minutes": 99}
return mock.patch("client.api_chat", side_effect=fake)
def test_tool_loop_feeds_results_and_finishes_on_done(self) -> None:
Path("greet.txt").write_text("hi\n")
replies = [
'Reading the file.\n```sj:read\n{"path": "greet.txt"}\n```',
'```sj:write\n{"path": "out.txt", "content": "done"}\n```',
"Everything is in place.\nDone.",
]
with self._chat(replies):
status, final, _ = c.agent_turn({}, "t", "m", [], "make out.txt",
auto=True)
self.assertEqual(status, "done")
self.assertEqual(Path("out.txt").read_text(), "done\n")
self.assertEqual(final, "Everything is in place.\nDone.")
def test_plain_text_reply_ends_turn_without_tool_loop(self) -> None:
with self._chat(["Just answering."]):
status, final, _ = c.agent_turn({}, "t", "m", [], "hello", auto=True)
self.assertEqual(status, "done")
self.assertEqual(final, "Just answering.")
def test_error_result_is_sent_back_to_model(self) -> None:
replies = [
'```sj:read\n{"path": "missing.txt"}\n```',
"Task complete.\nDone.",
]
with self._chat(replies):
status, _, _ = c.agent_turn({}, "t", "m", [], "read it", auto=True)
self.assertEqual(status, "done")
if __name__ == "__main__":
unittest.main()
test_e2e.py
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""End-to-end test: fake ollama -> real daemon -> real client agent loop.
Exercises the whole chain the same way a user would: the client connects,
logs in, gets a task, confirms a shell command, and the agent actually
creates a file in the working directory.
Run: python3 test_e2e.py
"""
import json
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parent
def free_ports(count: int) -> list[int]:
"""Find unused ports by binding ephemeral sockets (avoids TIME_WAIT)."""
ports, sockets = [], []
while len(ports) < count:
s = socket.socket()
s.bind(("127.0.0.1", 0))
sockets.append(s)
ports.append(s.getsockname()[1])
for s in sockets:
s.close()
return ports
class E2E(unittest.TestCase):
def setUp(self):
self.ollama_port, self.daemon_port = free_ports(2)
self.tmp = Path(tempfile.mkdtemp(prefix="sj-e2e-"))
self.server_dir = self.tmp / "server"
self.work = self.tmp / "work"
self.server_dir.mkdir()
self.work.mkdir()
(self.server_dir / "config.toml").write_text(
f'[server]\nhost = "127.0.0.1"\nport = {self.daemon_port}\n'
f'\n[ollama]\nbase_url = "http://127.0.0.1:{self.ollama_port}"\n'
'model = "fake-model"\nnum_ctx = 16384\n'
'\n[database]\npath = "sjcode.db"\n'
"\n[limits]\nuser = 120\nadmin = -1\n"
)
self.prompt_log = self.tmp / "prompts.log"
self.fake = subprocess.Popen(
[sys.executable, str(ROOT / "fake_ollama.py"),
str(self.ollama_port), str(self.prompt_log)],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
env = dict(os.environ, SJCODE_CONFIG=str(self.server_dir / "config.toml"))
self.daemon = subprocess.Popen(
[sys.executable, str(ROOT / "server" / "server.py")],
cwd=self.server_dir, env=env, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
self._wait_ready()
# create the test account through the daemon admin CLI
self.daemon.stdin.write("create tester admin\nsecret\nsecret\n")
self.daemon.stdin.flush()
time.sleep(0.5) # let the admin CLI process the lines
def _wait_ready(self, timeout: float = 10.0) -> None:
deadline = time.time() + timeout
url = f"http://127.0.0.1:{self.daemon_port}/status"
while time.time() < deadline:
try:
with urllib.request.urlopen(url, timeout=2) as resp:
if resp.status == 200:
return
except OSError:
time.sleep(0.1)
self.fail("daemon did not come up in time")
def tearDown(self):
for proc in (self.daemon, self.fake):
if proc.poll() is None:
proc.terminate()
for proc in (self.daemon, self.fake):
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
shutil.rmtree(self.tmp, ignore_errors=True)
def _client_run(self, inputs: list[str]) -> subprocess.CompletedProcess:
script = "\n".join(inputs) + "\n"
# keep the client's saved-server config inside the test sandbox
env = dict(os.environ, XDG_CONFIG_HOME=str(self.tmp / "cfg"))
return subprocess.run(
[sys.executable, str(ROOT / "client.py"),
f"127.0.0.1:{self.daemon_port}", "--model", "fake-model"],
cwd=self.work, env=env, input=script, capture_output=True,
text=True, timeout=120)
def test_full_agent_run_with_confirmation(self):
proc = self._client_run([
"tester", # username
"secret", # password
"create hello.txt with the word hi in it",
"y", # confirm the run tool call
"/quit",
])
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
self.assertIn("logged in as tester", proc.stdout)
self.assertIn("Done.", proc.stdout)
hello = self.work / "hello.txt"
self.assertTrue(hello.exists(), "agent did not create hello.txt")
self.assertEqual(hello.read_text().strip(), "hi")
# the model saw the REAL command output, not invented text
prompts = [json.loads(line)
for line in self.prompt_log.read_text().splitlines()]
self.assertEqual(len(prompts), 4) # run, read, done, done-again
self.assertIn("printf", prompts[1]) # run result fed back
self.assertIn("Tool results", prompts[1])
self.assertIn("1\thi", prompts[2]) # read result fed back
if __name__ == "__main__":
unittest.main()