FIX SOME BUGS

README.md

@@ -1,18 +1,52 @@
 # sjcode
 
-ein terminal coding agent, der die ollama API nutzt
+a terminal coding agent, powered by the ollama API
+
+## What makes it an agent
+
+The client is not a chat front end: the model can **really work in your
+codebase**. It runs shell commands, reads and edits files — on your machine,
+in the directory you started `./coder` from. Every model reply can contain
+tool calls; the client executes them and feeds the real output back to the
+model until the task is done.
+
+Tools the agent can use:
+
+| tool    | what it does                                     | asks first |
+|---------|--------------------------------------------------|------------|
+| `run`   | run one shell command in the project directory   | yes        |
+| `read`  | read a file (numbered lines, `start`/`limit`)    | no         |
+| `files` | list project files (skips `.git`, caches, ...)   | no         |
+| `write` | create a file, or overwrite a small one          | yes        |
+| `edit`  | exact-text replace in a file, shows a diff       | yes        |
+
+Before every `run`/`write`/`edit` you see exactly what will happen (command
+line, or a unified diff for file changes) and press Enter to allow it, `n` to
+decline, `a` to stop the agent — or just type a short instruction ("use ripgrep
+instead") and the model adapts. `--yes` (or `/auto` in the session) runs
+everything without asking; the sandbox keeps all file tools inside the project
+directory.
 
 ## Setup
 
 ```bash
-cp config.toml.example config.toml   # then edit port / model / limits
+cp server/config.toml.example server/config.toml   # then edit port/model/limits
 python3 server/server.py                            # starts the daemon + admin CLI
 ```
 
 `config.toml` is machine-local and git-ignored; `config.toml.example` is the
 template. On first start the daemon creates the SQLite database (`sjcode.db`).
 
-## Daemon (server.py)
+Pick a coder model in `[ollama] model` — one that follows structured output
+well, e.g. `qwen2.5-coder:7b` or bigger. The agent transcript is long, so the
+daemon requests a large context window from ollama:
+
+```toml
+[ollama]
+num_ctx = 16384   # context window; do not go below 8192 for agent mode
+```
+
+## Daemon (server/server.py)
 
 On startup it prints its settings and drops into a minimal admin CLI
 (accounts are stored in SQLite):
@@ -32,12 +66,18 @@ On startup it prints its settings and drops into a minimal admin CLI
 While the CLI runs, an HTTP API is served in the background:
 
 - `POST /login` — `{"username", "password"}` → `{"token", ...}`
-- `POST /chat`  — `{"token", "message", "model"?}` → `{"response", ...}`
+- `POST /chat`  — `{"token", "message", "model"?, "stream"?}` → reply as JSON,
+  or (with `"stream": true`) as NDJSON lines `{"chunk": ...}` ending with one
+  final `{"response", "used_minutes", "remaining_minutes", ...}` line
 - `GET  /models?token=...` — list ollama models
+- `GET  /status` — health check
 
 Every user row stores: `username`, `password_hash`, `last_logged_in`,
 `role`, `used_today` (minutes of AI usage; resets on a new day).
 
+`SJCODE_CONFIG=/path/to/config.toml` lets you run a second daemon (tests,
+staging) next to a regular one.
+
 ## Roles and limits
 
 In `config.toml` each role gets a daily limit in minutes:
@@ -56,6 +96,7 @@ Login and every chat request are blocked once `used_today` reaches the limit.
 ./coder                               # or: python3 client.py
 python3 client.py 10.0.0.5:8420       # explicit daemon address
 python3 client.py --model qwen3-coder # skip the model picker
+python3 client.py --yes               # auto-approve every tool call (careful)
 python3 client.py --no-probe          # skip the reachability check (use behind
                                       # HTTP proxies that answer on every port)
 ```
@@ -66,7 +107,10 @@ ones, or add a new one (`n`). Every server you use is saved in
 model, so the next start is just Enter-Enter. Addresses without a port use
 8420; `host:port`, `http(s)://...` URLs and IPv6 literals are accepted.
 
-After login and model choice you get the agent banner and a chat REPL.
+After login and model choice you get the agent banner and a REPL. Type tasks,
+not chat messages — "rename the function foo to bar in all files" is a task,
+"what is a monad" is just a question the agent answers in place.
+
 Commands:
 
 | command       | effect                                  |
@@ -76,14 +120,23 @@ Commands:
 | `/models`     | list models on the server               |
 | `/model <n>`  | switch model (saved for this server)    |
 | `/limit`      | remaining minutes today (last known)    |
+| `/auto`       | toggle confirmations (auto-approve all) |
+| `/state`      | auto mode, project dir, history size    |
+| `/cwd`        | show the project directory              |
 | `/clear`      | forget the conversation history         |
 | `/quit`       | exit                                    |
 
-Anything else is sent to the model. The client keeps the last few turns as
+The agent keeps the last few finished tasks as context. It stops after 30
-context (the daemon API itself is stateless) and renders the reply in the
+model round-trips per task; the daemon API itself is stateless.
-agent style (`✓` lines for actions the model reports).
 
 ## Requirements
 
 - Python 3.11+ (stdlib only; `tomllib` is included)
 - ollama listening on the address from `config.toml`
+
+## Tests
+
+```bash
+python3 -m unittest test_client.py -v   # unit tests (no network)
+python3 test_e2e.py                     # fake ollama + daemon + client
+```

a.txt

@@ -0,0 +1,50 @@
+1
+2
+3
+4
+5
+6
+7
+8
+9
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+40
+41
+42
+43
+44
+45
+46
+47
+48
+49
+50

app.py

@@ -0,0 +1,2 @@
+x = 1
+x = 1

big.txt

@@ -0,0 +1 @@
+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

client.py

@@ -3,27 +3,37 @@
 
 Terminal coding-agent front end for the sjcode daemon (server/server.py).
 
+This is a real agent: it can run shell commands and change files in the
+directory it was started from (the "project directory"). The model emits
+tool calls, the client executes them locally and feeds the real output
+back to the model until the task is done. You confirm every command that
+runs and every file that is written, unless you pass --yes or toggle
+/auto in the session.
+
 On start it asks which server to connect to. Servers you used before are
 remembered in ~/.config/sjcode/client.json together with username and the
-last model, so next time you just press Enter. New servers are saved too.
+last model, so next time you just press Enter.
 
 Flow:
     coder                     -> banner, pick saved server (or add one),
-                                 login, pick model, chat REPL
+                                 login, pick model, agent REPL
     coder 10.0.0.5:8420       -> skip the server picker
     coder --model qwen3-coder -> skip the model picker
+    coder --yes               -> auto-approve every tool call (careful)
 
 REPL commands:
-    /help /server /model /models /limit /clear /quit
+    /help /server /model /models /limit /auto /state /cwd /clear /quit
 """
 
 from __future__ import annotations
 
 import argparse
+import difflib
 import getpass
 import json
 import os
 import re
+import subprocess
 import sys
 import time
 import urllib.error
@@ -35,8 +45,19 @@ CONFIG_DIR = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config")) /
 CONFIG_FILE = CONFIG_DIR / "client.json"
 
 DEFAULT_PORT = 8420
-REQUEST_TIMEOUT = 300
+REQUEST_TIMEOUT = 600
-HISTORY_TURNS = 10  # prior turns sent with each message for context
+HISTORY_TURNS = 10  # prior task/summary pairs sent with each task for context
+
+# The project directory the agent works in (where ./coder was started).
+CWD = Path.cwd()
+
+MAX_TURNS = 30        # model round-trips per user task
+NO_TOOL_LIMIT = 2     # plain-text replies before the turn ends by itself
+RUN_TIMEOUT = 300     # seconds per shell command
+MAX_RUN_CHARS = 10_000   # shell output fed back / shown, per command
+READ_DEFAULT_LIMIT = 250  # lines per read
+WRITE_MAX_CHARS = 60_000  # refuse to write bigger files in one go
+DIFF_PREVIEW_LINES = 60   # diff lines shown in a confirmation prompt
 
 HOST_RE = re.compile(
     r"^(?=.{1,253}\z)"                     # total length limit
@@ -48,14 +69,77 @@ HOST_RE = re.compile(
     re.IGNORECASE,
 )
 
-# Makes the model behave like a terminal agent: short lines, check marks.
+# Tool-call protocol: the model answers with normal text plus fenced blocks.
-AGENT_PREAMBLE = (
+#     ```run
-    "You are sj, a coding agent working in the user's terminal.\n"
+#     git status --short
-    "Answer concisely and practically.\n"
+#     ```
-    "Describe what you do as short lines; put every action you performed "
+#     ```sj:write
-    "or verified on its own line starting with '\u2713 ' (check mark).\n"
+#     {"path": "x.txt", "content": "hi"}
-    "End your reply with the single line 'Done.'"
+#     ```
+# The closing fence must be a line containing only ``` so that content which
+# itself contains backticks (inside a JSON string) cannot end the block early.
+# (?![a-z]) keeps language fences like ```rust or ```ruby from being read as
+# run-tool calls.
+TAG_RE = re.compile(
+    r"(?ms)^[ \t]*```(?:sj:([a-z]+)|run)(?![a-z])[^\n]*\n(.*?)(?=\n[ \t]*```[ \t]*(?:\n|\Z)|\Z)"
 )
+STRAY_FENCE_RE = re.compile(r"(?m)^[ \t]*```[^\n]*$")
+
+AGENT_PREAMBLE = """\
+You are sj, a real coding agent working in the user's project directory on
+their machine. You can actually run commands and change files. Work like an
+experienced engineer:
+
+- Plan silently, act step by step. After every step, look at the real output
+  you got before deciding the next step.
+- Explore before you change code you have not seen: list files, read them,
+  then edit. Never guess file contents.
+- Keep replies short. Say what you are doing, then do it.
+- Every action you performed or verified goes on its own line starting with
+  "✓ " (check mark).
+- If something fails, read the error and change your approach; do not repeat
+  a failing call unchanged and do not apologize.
+- When the whole task is finished, end your reply with the single line "Done."
+
+You act by emitting tool calls. A tool call is a fenced code block tagged
+"run" or "sj:<tool>" whose body is plain text (run) or a JSON object (all
+other tools). Examples:
+
+```run
+git status --short
+```
+
+```sj:write
+{"path": "notes/hello.txt", "content": "hi\\n"}
+```
+
+```sj:edit
+{"path": "app.py", "old": "old text", "new": "new text", "all": false}
+```
+
+```sj:read
+{"path": "app.py", "start": 1, "limit": 200}
+```
+
+```sj:files
+{"path": ".", "limit": 200}
+```
+
+Tools:
+- run:    run one shell command in the project directory. Every run starts
+          in the project directory again (cd does not persist).
+- read:   read a text file. Args: path, start (line number), limit (lines).
+- files:  list project files. Args: path (subdirectory), limit.
+- write:  create a new file or overwrite a small one. Args: path, content.
+- edit:   replace exact text in a file. Args: path, old, new, all (optional).
+
+Rules:
+- Emit at most ONE tool call per reply, then stop. You will receive its real
+  output as the next message and then continue.
+- The user confirms dangerous actions. If a result says "cancelled by user",
+  accept that, adjust, and move on; never repeat the same call.
+- Never invent tool output. If you need information, call a tool to get it.
+"""
 
 
 # ---------------------------------------------------------------------------
@@ -64,6 +148,7 @@ AGENT_PREAMBLE = (
 
 class Style:
     def __init__(self, enabled: bool) -> None:
+        self.enabled = enabled
         self.dim = "\033[2m" if enabled else ""
         self.bold = "\033[1m" if enabled else ""
         self.green = "\033[32m" if enabled else ""
@@ -100,13 +185,13 @@ def banner(model: str, project: str, server: str) -> None:
     for line in lines:
         print(f"\u2502 {line.ljust(width)} \u2502")
     print(f"\u2570{'\u2500' * (width + 2)}\u256f")
-    print(f"{ST.dim}server: {server}  (\u2018/help\u2019 for commands){ST.reset}")
+    print(f"{ST.dim}server: {server}  ('/help' for commands){ST.reset}")
 
 
-def shorten_home(path: Path) -> str:
+def shorten_home(path: Path) -> Path:
     home = Path.home()
     try:
-        return "~" / path.relative_to(home)
+        return Path("~") / path.relative_to(home)
     except ValueError:
         return path
 
@@ -171,8 +256,8 @@ def pick_server(state: dict) -> tuple[dict, bool]:
     servers = state["servers"]
     print(f"\n{ST.bold}Connect to which server?{ST.reset}")
     for i, entry in enumerate(servers, 1):
-        default = "  (last)" if server_label(entry) == state.get("last") else ""
+        default = f"  {ST.dim}(last){ST.reset}" if server_label(entry) == state.get("last") else ""
-        user = f"  as {entry['username']}" if entry.get("username") else ""
+        user = f"  {ST.dim}({entry['username']}){ST.reset}" if entry.get("username") else ""
         print(f"  {i}) {server_label(entry)}{user}{default}")
     print(f"  n) add a new server\n")
 
@@ -198,7 +283,7 @@ def pick_server(state: dict) -> tuple[dict, bool]:
 
 
 def add_server(state: dict) -> dict:
-    """Ask for a new server address, probe it and save it."""
+    """Ask for a new server address, and save it."""
     while True:
         raw = ask("server address (domain or ip:port): ").strip()
         try:
@@ -218,8 +303,20 @@ def add_server(state: dict) -> dict:
 # daemon API
 # ---------------------------------------------------------------------------
 
+def _feed_chunk(line: str, on_chunk, final: dict) -> dict:
+    """Handle one JSON line of a streamed reply: chunk -> callback, else final."""
+    try:
+        obj = json.loads(line)
+    except json.JSONDecodeError:
+        return final
+    if isinstance(obj, dict) and "chunk" in obj:
+        on_chunk(obj)
+        return final
+    return obj if isinstance(obj, dict) else final
+
+
 def http_json(method: str, base: str, path: str, payload: dict | None = None,
-              timeout: int = REQUEST_TIMEOUT) -> tuple[int, dict]:
+              timeout: int = REQUEST_TIMEOUT, on_chunk=None) -> tuple[int, dict]:
     url = base + path
     data = None if payload is None else json.dumps(payload).encode()
     req = urllib.request.Request(
@@ -228,7 +325,21 @@ def http_json(method: str, base: str, path: str, payload: dict | None = None,
     )
     try:
         with urllib.request.urlopen(req, timeout=timeout) as resp:
+            if on_chunk is None:
                 return resp.status, json.loads(resp.read().decode())
+            # streamed reply: one JSON object per line
+            status = resp.status
+            final: dict = {}
+            buf = ""
+            for raw in resp:
+                buf += raw.decode(errors="replace")
+                while "\n" in buf:
+                    line, buf = buf.split("\n", 1)
+                    if line.strip():
+                        final = _feed_chunk(line.strip(), on_chunk, final)
+            if buf.strip():
+                final = _feed_chunk(buf.strip(), on_chunk, final)
+            return status, final
     except urllib.error.HTTPError as exc:
         try:
             return exc.code, json.loads(exc.read().decode())
@@ -266,10 +377,14 @@ def api_models(entry: dict, token: str) -> list[str]:
     return list(body.get("models", [])) if status == 200 else []
 
 
-def api_chat(entry: dict, token: str, message: str, model: str) -> tuple[int, dict]:
+def api_chat(entry: dict, token: str, message: str, model: str,
+             on_chunk=None) -> tuple[int, dict]:
     base = f"http://{entry['host']}:{entry['port']}"
-    return http_json("POST", base, "/chat",
+    payload: dict = {"token": token, "message": message, "model": model}
-                     {"token": token, "message": message, "model": model})
+    if on_chunk is not None:
+        payload["stream"] = True
+        return http_json("POST", base, "/chat", payload, on_chunk=on_chunk)
+    return http_json("POST", base, "/chat", payload)
 
 
 # ---------------------------------------------------------------------------
@@ -333,48 +448,513 @@ def pick_model(entry: dict, models: list[str]) -> str:
 
 
 # ---------------------------------------------------------------------------
-# the agent look
+# tools the agent can use (executed locally, in the project directory)
 # ---------------------------------------------------------------------------
 
+class ToolError(Exception):
+    """A tool refused to run; the message goes back to the model."""
+
+
+class ToolAbort(Exception):
+    """The user aborted the whole agent turn at a confirmation prompt."""
+
+
+ABORT = "!!abort"
+
+EXCLUDED_DIRS = {
+    ".git", "__pycache__", "node_modules", ".venv", "venv", ".mypy_cache",
+    ".ruff_cache", ".pytest_cache", ".tox", ".idea", ".vscode", ".eggs",
+    "dist", "build", ".hg", ".svn",
+}
+
+def _resolve(path: str) -> Path:
+    """Resolve a path against the project dir; refuse anything outside it."""
+    if not path:
+        raise ToolError("path required")
+    p = Path(path).expanduser()
+    if not p.is_absolute():
+        p = CWD / p
+    try:
+        rp = p.resolve()
+    except OSError as exc:
+        raise ToolError(f"cannot resolve path: {exc}")
+    inside = rp == CWD or str(rp).startswith(str(CWD) + os.sep)
+    if not inside:
+        raise ToolError(
+            f"outside the project directory ({CWD}): {path!r}")
+    return rp
+
+
+def _display(p: Path) -> str:
+    try:
+        rel = str(p.relative_to(CWD))
+        return rel or "."
+    except ValueError:
+        return str(p)
+
+
+def _unified_diff(old: str, new: str, display: str) -> list[str]:
+    diff = difflib.unified_diff(
+        old.splitlines(), new.splitlines(),
+        fromfile=f"{display} (old)", tofile=f"{display} (new)", lineterm="",
+    )
+    lines = list(diff)
+    if len(lines) > DIFF_PREVIEW_LINES:
+        lines = lines[:DIFF_PREVIEW_LINES]
+        lines.append(f"... (+{len(lines)} more diff lines)")
+    return lines
+
+
+def _run_args(args: dict) -> tuple[str, int, Path]:
+    """Validate run-tool args -> (command, timeout, working dir)."""
+    cmd_text = str(args.get("command") or args.get("cmd") or "").strip()
+    if not cmd_text:
+        raise ToolError("'command' required")
+    timeout = args.get("timeout")
+    try:
+        timeout = min(max(int(timeout), 5), RUN_TIMEOUT) if timeout else RUN_TIMEOUT
+    except (TypeError, ValueError):
+        timeout = RUN_TIMEOUT
+    wd = _resolve(str(args.get("cwd") or "."))
+    if not wd.is_dir():
+        raise ToolError(f"not a directory: {_display(wd)}")
+    return cmd_text, timeout, wd
+
+
+def tool_run_confirm(args: dict) -> list[str]:
+    cmd_text, timeout, wd = _run_args(args)
+    return [
+        f"$ {cmd_text}",
+        f"cwd: {_display(wd)}   timeout: {timeout}s",
+    ]
+
+
+def tool_run_execute(args: dict) -> list[str]:
+    cmd_text, timeout, wd = _run_args(args)
+    try:
+        proc = subprocess.run(
+            cmd_text, shell=True, cwd=str(wd), capture_output=True,
+            text=True, timeout=timeout,
+        )
+    except subprocess.TimeoutExpired:
+        return [f"\u2717 $ {cmd_text}", f"timed out after {timeout}s"]
+    except OSError as exc:
+        return [f"\u2717 $ {cmd_text}", f"cannot run: {exc}"]
+
+    ok = proc.returncode == 0
+    head = (f"\u2713 $ {cmd_text}" if ok
+            else f"\u2717 $ {cmd_text}  (exit {proc.returncode})")
+    parts = []
+    if (proc.stdout or "").strip():
+        parts.append(proc.stdout.rstrip())
+    if (proc.stderr or "").strip():
+        parts.append(proc.stderr.rstrip())
+    text = "\n".join(parts).strip()
+    if len(text) > MAX_RUN_CHARS:
+        text = text[:MAX_RUN_CHARS] + f"\n... (output truncated at {MAX_RUN_CHARS} chars)"
+    return [head] + (text.splitlines() if text else [])
+
+
+def tool_read(args: dict) -> list[str]:
+    path = str(args.get("path", "")).strip()
+    p = _resolve(path)
+    if not p.is_file():
+        raise ToolError(f"not a file: {_display(p)}")
+    try:
+        text = p.read_text(encoding="utf-8", errors="replace")
+    except OSError as exc:
+        raise ToolError(str(exc))
+    lines = text.splitlines()
+    total = len(lines)
+    try:
+        start = max(1, int(args.get("start", 1)))
+    except (TypeError, ValueError):
+        start = 1
+    try:
+        limit = int(args.get("limit", READ_DEFAULT_LIMIT))
+    except (TypeError, ValueError):
+        limit = READ_DEFAULT_LIMIT
+    limit = max(1, min(limit, 1000))
+    start = min(start, max(total, 1))
+    sel = lines[start - 1:start - 1 + limit]
+    out = [f"\u2713 read {_display(p)} ({total} lines)"]
+    for i, line in enumerate(sel, start):
+        out.append(f"{i}\t{line[:400]}")
+    if start - 1 + limit < total:
+        out.append(f"(... {total - (start - 1 + limit)} more lines - "
+                   f"read again with 'start'/'limit')")
+    return out
+
+
+def tool_files(args: dict) -> list[str]:
+    sub = str(args.get("path") or ".").strip()
+    base = _resolve(sub)
+    if not base.is_dir():
+        raise ToolError(f"not a directory: {_display(base)}")
+    try:
+        limit = max(1, min(int(args.get("limit", 200)), 500))
+    except (TypeError, ValueError):
+        limit = 200
+    out: list[str] = []
+    truncated = False
+    for root, dirs, fnames in os.walk(base):
+        dirs[:] = sorted(
+            d for d in dirs
+            if d not in EXCLUDED_DIRS and not d.endswith(".egg-info")
+        )
+        for fn in sorted(fnames):
+            if len(out) >= limit:
+                truncated = True
+                break
+            out.append(_display(Path(root) / fn))
+        if truncated:
+            break
+    head = (f"\u2713 files under {_display(base)} ({len(out)} shown"
+            + (", truncated - narrow with 'path'" if truncated else "") + ")")
+    return [head] + out
+
+
+def tool_write_confirm(args: dict) -> list[str]:
+    path = str(args.get("path", "")).strip()
+    if not path:
+        raise ToolError("path required")
+    content = args.get("content")
+    if not isinstance(content, str):
+        raise ToolError("'content' (string) required")
+    if len(content) > WRITE_MAX_CHARS:
+        raise ToolError(
+            f"content too large ({len(content)} chars, max {WRITE_MAX_CHARS}) - "
+            f"write it in smaller pieces or use 'run'")
+    p = _resolve(path)
+    exists = p.is_file()
+    if exists:
+        old = p.read_text(encoding="utf-8", errors="replace")
+        if len(old) > 2000:
+            raise ToolError(
+                f"{_display(p)} exists and is {len(old)} chars; refusing to "
+                f"overwrite it wholesale - use 'edit' instead")
+    else:
+        old = ""
+    summary = [
+        f"{'overwrite' if exists else 'create'} {_display(p)} "
+        f"({len(content.splitlines())} lines)"
+    ]
+    summary += _unified_diff(old, content, _display(p))
+    return summary
+
+
+def tool_write_execute(args: dict) -> list[str]:
+    p = _resolve(str(args.get("path", "")).strip())
+    content = str(args.get("content", ""))
+    p.parent.mkdir(parents=True, exist_ok=True)
+    text = content if content.endswith("\n") else content + "\n"
+    p.write_text(text, encoding="utf-8")
+    return [f"\u2713 wrote {_display(p)} ({len(text.splitlines())} lines)"]
+
+
+def _flag(args: dict, key: str) -> bool:
+    return args.get(key) in (True, "true", "True", 1, "1", "yes")
+
+
+def tool_edit_confirm(args: dict) -> list[str]:
+    path = str(args.get("path", "")).strip()
+    if not path:
+        raise ToolError("path required")
+    old, new = args.get("old"), args.get("new")
+    if not isinstance(old, str) or not old:
+        raise ToolError("'old' (exact existing text) required")
+    if not isinstance(new, str):
+        raise ToolError("'new' (replacement text) required")
+    p = _resolve(path)
+    if not p.is_file():
+        raise ToolError(f"not a file: {_display(p)}")
+    body = p.read_text(encoding="utf-8", errors="replace")
+    n = body.count(old)
+    if n == 0:
+        raise ToolError(
+            f"search text not found in {_display(p)} - 'read' the file and "
+            f"copy the text exactly")
+    all_flag = _flag(args, "all")
+    if n > 1 and not all_flag:
+        raise ToolError(
+            f"'old' matches {n} times in {_display(p)} - include more "
+            f"surrounding text or set \"all\": true")
+    updated = body.replace(old, new) if all_flag else body.replace(old, new, 1)
+    summary = [
+        f"edit {_display(p)} "
+        f"({'all' if all_flag else 'first'} of {n} match{'es' if n != 1 else ''})"
+    ]
+    summary += _unified_diff(body, updated, _display(p))
+    return summary
+
+
+def tool_edit_execute(args: dict) -> list[str]:
+    p = _resolve(str(args.get("path", "")).strip())
+    old, new = str(args.get("old", "")), str(args.get("new", ""))
+    all_flag = _flag(args, "all")
+    body = p.read_text(encoding="utf-8", errors="replace")
+    updated = body.replace(old, new) if all_flag else body.replace(old, new, 1)
+    p.write_text(updated, encoding="utf-8")
+    return [f"\u2713 edited {_display(p)}"]
+
+
+# name -> {confirm, execute, confirm_first}
+TOOLS: dict[str, dict] = {
+    "run":   {"confirm": tool_run_confirm,     "execute": tool_run_execute,   "confirm_first": True},
+    "read":  {"confirm": tool_read,            "execute": tool_read,          "confirm_first": False},
+    "files": {"confirm": tool_files,           "execute": tool_files,         "confirm_first": False},
+    "write": {"confirm": tool_write_confirm,   "execute": tool_write_execute, "confirm_first": True},
+    "edit":  {"confirm": tool_edit_confirm,    "execute": tool_edit_execute,  "confirm_first": True},
+}
+
+
+def confirm_tool(name: str, summary: list[str], auto: bool) -> str:
+    """Ask the user to approve a tool call.
+
+    Returns "y", "n", ABORT, or a free-text instruction from the user.
+    """
+    if auto:
+        return "y"
+    print(f"{ST.bold}{ST.yellow}? {name}{ST.reset}")
+    for line in summary:
+        print(f"  {line}")
+    try:
+        answer = input(
+            f"  {ST.bold}[Enter=yes  n=no  a=abort  or type a new instruction]{ST.reset} "
+        ).strip()
+    except KeyboardInterrupt:
+        print()
+        return "n"
+    except EOFError:
+        print()
+        return ABORT
+    low = answer.lower()
+    if low in ("", "y", "yes"):
+        return "y"
+    if low in ("n", "no", "c", "cancel"):
+        return "n"
+    if low in ("a", "abort"):
+        return ABORT
+    return answer
+
+
+def handle_tool_call(name: str, raw_body: str, auto: bool) -> tuple[bool, list[str]]:
+    """Validate, confirm and execute one tool call. Returns (ok, lines)."""
+    spec = TOOLS.get(name)
+    if spec is None:
+        return False, [f"\u2717 unknown tool '{name}' - "
+                       f"available: {', '.join(sorted(TOOLS))}"]
+    if name == "run":
+        # Accept the plain-text command (as documented) or a JSON object
+        # with a "command" key - some models emit JSON anyway.
+        body = (raw_body or "").strip()
+        args = {"command": body}
+        if body.startswith("{"):
+            try:
+                parsed = json.loads(body)
+                if isinstance(parsed, dict) and "command" in parsed:
+                    args = parsed
+            except json.JSONDecodeError:
+                pass  # e.g. a brace group in shell - run it as-is
+    else:
+        try:
+            args = json.loads(raw_body or "{}")
+            if not isinstance(args, dict):
+                raise ValueError("arguments must be a JSON object")
+        except json.JSONDecodeError as exc:
+            return False, [f"\u2717 bad JSON for '{name}': {exc} - fix it and retry"]
+        except ValueError as exc:
+            return False, [f"\u2717 {exc}"]
+
+    try:
+        prepared = spec["confirm"](args)  # validates; for read/files it already runs
+    except ToolError as exc:
+        return False, [f"\u2717 {exc}"]
+    except (KeyError, TypeError, ValueError, OSError) as exc:
+        return False, [f"\u2717 {exc}"]
+
+    if not spec["confirm_first"]:  # read/files: safe, results are in
+        return True, prepared
+
+    decision = "y" if auto else confirm_tool(name, prepared, auto)
+    if decision == ABORT:
+        raise ToolAbort()
+    if decision != "y":
+        note = "cancelled by user" if decision == "n" else f"user says: {decision}"
+        return False, [f"\u2717 {note} - do not repeat this exact call; "
+                       f"adjust your approach or stop"]
+
+    try:
+        lines = spec["execute"](args)
+        # A failing command is not a tool error, but the model must see it
+        # as a failure: the first output line starts with the cross mark.
+        ok = not (lines and lines[0].startswith("\u2717"))
+        return ok, lines
+    except ToolError as exc:
+        return False, [f"\u2717 {exc}"]
+    except (OSError, ValueError, RuntimeError) as exc:
+        return False, [f"\u2717 {exc}"]
+
+
+# ---------------------------------------------------------------------------
+# agent turn: model reply -> tool calls -> real output -> model ...
+# ---------------------------------------------------------------------------
+
+def read_tool_tags(raw: str) -> list[tuple[str, str]]:
+    """Extract tool calls in order: [(name, body), ...]."""
+    tags: list[tuple[str, str]] = []
+    for m in TAG_RE.finditer(raw):
+        name = m.group(1) or "run"
+        tags.append((name, (m.group(2) or "").strip()))
+    return tags
+
+
 def print_agent_reply(text: str) -> None:
     print(f"{ST.bold}Agent:{ST.reset}")
     for line in text.rstrip().splitlines():
-        stripped = line.strip()
+        if line.startswith("\u2713"):
-        if stripped.startswith("\u2713"):
+            print(f"{ST.green}{line}{ST.reset}")
-            print(f"  {ST.green}\u2713{ST.reset} {stripped.lstrip('\u2713').strip()}")
+        elif line.startswith("\u2717"):
-        elif stripped.lower() == "done.":
+            print(f"{ST.red}{line}{ST.reset}")
-            print(f"  {ST.bold}{stripped}{ST.reset}")
         else:
-            print(f"  {line.strip() if stripped else ''}")
+            print(line)
 
 
-def build_prompt(history: list[tuple[str, str]], message: str) -> str:
+def process_reply(raw: str, auto: bool, show_text: bool = True
-    """Plain-text transcript + new user message (server endpoint is stateless)."""
+                  ) -> tuple[bool, str, list]:
-    parts = [AGENT_PREAMBLE]
+    """Print the reply, run its tool calls. Returns (had_tools, text, results)."""
+    # Tool blocks out, leftover closing-fence lines out, blank edges trimmed.
+    reply_text = STRAY_FENCE_RE.sub("", TAG_RE.sub("", raw)).strip()
+    if reply_text and show_text:  # already shown live when streamed
+        print_agent_reply(reply_text)
+    tags = read_tool_tags(raw)
+    if not tags:
+        return False, reply_text, []
+    results: list[tuple[str, bool, list[str]]] = []
+    for name, body in tags:
+        print(f"{ST.dim}\u2014 tool: {name}{ST.reset}")
+        ok, lines = handle_tool_call(name, body, auto)
+        results.append((name, ok, lines))
+        for i, line in enumerate(lines):
+            if i == 0 and not ok:
+                print(f"{ST.red}{line}{ST.reset}")
+            elif line.startswith("\u2713"):
+                print(f"{ST.green}{line}{ST.reset}")
+            else:
+                print(f"  {line}")
+    return True, reply_text, results
+
+
+def agent_turn(entry: dict, token: str, model: str,
+               history: list[tuple[str, str]], task: str, auto: bool
+               ) -> tuple[str, str, float | None]:
+    """Run one user task through the agent loop.
+
+    Returns (status, final_text, remaining_minutes):
+    status is "done", "stopped", "cancelled", "limit" or "error".
+    """
+    transcript = ""
     for user_text, answer in history[-HISTORY_TURNS:]:
-        parts.append(f"User: {user_text}\nAssistant: {answer}")
+        transcript += f"User: {user_text}\nAssistant: {answer}\n\n"
-    parts.append(f"User: {message}\nAssistant:")
+    messages = [AGENT_PREAMBLE
-    return "\n\n".join(parts)
+                + (("\n\n" + transcript) if transcript else "")
+                + f"\n\nUser: {task}\nAssistant:"]
+    remaining: float | None = None
+    no_tool = 0
+    streamed = {"any": False}
+
+    def on_chunk(obj: dict) -> None:
+        piece = obj.get("chunk")
+        if not piece:
+            return
+        if not streamed["any"]:
+            streamed["any"] = True
+            print(f"{ST.bold}Agent:{ST.reset}")
+        sys.stdout.write(f"{ST.dim}{piece}{ST.reset}" if ST.enabled else piece)
+        sys.stdout.flush()
+
+    try:
+        for turn in range(1, MAX_TURNS + 1):
+            try:
+                status, body = api_chat(
+                    entry, token, "\n\n".join(messages), model,
+                    on_chunk=on_chunk)
+            except KeyboardInterrupt:
+                print()
+                return "cancelled", "", remaining
+            if status != 200 or body.get("error"):
+                print(f"\n{ST.red}error: "
+                      f"{body.get('error', 'unknown')}{ST.reset}")
+                if status == 403:
+                    return "limit", "", remaining
+                return "error", "", remaining
+            remaining = body.get("remaining_minutes", remaining)
+            raw = str(body.get("response", ""))
+            if streamed["any"]:
+                print()
+            had_tools, reply_text, results = process_reply(
+                raw, auto, show_text=not streamed["any"])
+            streamed["any"] = False
+
+            if not had_tools:
+                no_tool += 1
+                if no_tool >= NO_TOOL_LIMIT:
+                    return "done", reply_text, remaining
+                messages.append(
+                    "(no tool call received) If the task is complete, reply "
+                    "with exactly: Done.\nOtherwise continue with the next "
+                    "tool call.")
+                continue
+            no_tool = 0
+
+            if turn >= MAX_TURNS:
+                print(f"{ST.yellow}step limit ({MAX_TURNS}) reached - "
+                      f"stopping{ST.reset}")
+                return "stopped", reply_text, remaining
+
+            out = []
+            for name, ok, lines in results:
+                out.append(f"{name}: {'ok' if ok else 'FAILED'}")
+                out.extend("  " + ln for ln in lines)
+            messages.append(
+                "Tool results (real output from your last tool call):\n"
+                + ("\n".join(out) or "(no output)")
+                + "\n\nContinue with the next step.")
+    except ToolAbort:
+        print(f"{ST.yellow}stopped by user{ST.reset}")
+        return "stopped", "", remaining
+    except KeyboardInterrupt:
+        print()
+        return "cancelled", "", remaining
+    print(f"{ST.yellow}step limit ({MAX_TURNS}) reached - stopping{ST.reset}")
+    return "stopped", "", remaining
 
 
 # ---------------------------------------------------------------------------
 # chat REPL
 # ---------------------------------------------------------------------------
 
-HELP = """
+HELP = f"""\
 commands:
   /help            show this help
   /server          show the connected server
   /models          list models on the server
   /model <name>    switch model (also saved for this server)
   /limit           show remaining minutes today (last known)
+  /auto            toggle confirmations (auto-approve every tool call)
+  /state           show auto mode, project dir, history size
+  /cwd             show the project directory
   /clear           forget the conversation history
   /quit            exit (also Ctrl+D)
-anything else is sent to the model.
+anything else is a task for the agent. It can run commands and change files
+in the project directory; you confirm every action unless auto mode is on.
+
+available tools: {', '.join(sorted(TOOLS))}
 """
 
 
-def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> None:
+def chat_loop(entry: dict, token: str, model: str,
+              remaining: float | None, auto: bool) -> None:
     history: list[tuple[str, str]] = []
     while True:
         try:
@@ -409,6 +989,18 @@ def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> N
             elif cmd == "/limit":
                 left = "unlimited" if remaining is None else f"{remaining:g} min"
                 print(f"time left today: {left}")
+            elif cmd == "/auto":
+                auto = not auto
+                print("auto mode ON - tool calls run without confirmation"
+                      if auto else
+                      "auto mode OFF - every action asks first")
+            elif cmd == "/state":
+                print(f"auto: {'on' if auto else 'off'}")
+                print(f"project dir: {CWD}")
+                print(f"history: {len(history)} turn(s)")
+                print(f"model: {model or '(daemon default)'}")
+            elif cmd == "/cwd":
+                print(str(CWD))
             elif cmd == "/clear":
                 history.clear()
                 print("history cleared")
@@ -416,23 +1008,12 @@ def chat_loop(entry: dict, token: str, model: str, remaining: float | None) -> N
                 print(f"unknown command {cmd} - try /help")
             continue
 
-        try:
+        status, final, remaining = agent_turn(
-            status, body = api_chat(entry, token, build_prompt(history, message), model)
+            entry, token, model, history, message, auto)
-        except KeyboardInterrupt:
+        if status == "limit":
-            print(f"\n{ST.yellow}request cancelled{ST.reset}")
-            continue
-
-        if status != 200:
-            print(f"{ST.red}error: {body.get('error', 'unknown')}{ST.reset}")
-            if status == 403:
             return  # daily limit reached
-            continue
+        if status == "done" and final:
-
+            history.append((message, final))
-        answer = str(body.get("response", "")).strip() or "(empty reply)"
-        history.append((message, answer))
-        print()
-        print_agent_reply(answer)
-        remaining = body.get("remaining_minutes")
         if remaining is not None:
             print(f"{ST.dim}  ({remaining:g} min left today){ST.reset}")
         print()
@@ -453,6 +1034,9 @@ def main() -> None:
     parser.add_argument("--no-probe", action="store_true",
                         help="skip the reachability check before login (use behind "
                              "HTTP proxies that answer for every port)")
+    parser.add_argument("-y", "--yes", action="store_true",
+                        help="auto mode: run tool calls without confirmation "
+                             "(the agent edits real files - be careful)")
     args = parser.parse_args()
 
     state = load_state()
@@ -504,9 +1088,12 @@ def main() -> None:
     state["last"] = label
     save_state(state)
 
-    project = shorten_home(Path.cwd())
+    project = shorten_home(CWD)
     banner(model or "(server default)", str(project), label)
-    chat_loop(entry, token, model, None)
+    if args.yes:
+        print(f"{ST.yellow}auto mode: tool calls run without confirmation{ST.reset}")
+    print(f"{ST.dim}working dir: {CWD}{ST.reset}")
+    chat_loop(entry, token, model, None, args.yes)
     print("bye")
 
 

f.txt

@@ -0,0 +1 @@
+a

fake_ollama.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+"""Fake ollama for end-to-end tests of the sjcode daemon + client.
+
+Serves a scripted conversation: every /api/generate call returns the next
+reply from SCRIPT. Each prompt it receives is appended to prompts.log so
+the test can assert what the client actually fed back to the model.
+
+Usage: python3 fake_ollama.py PORT PROMPT_LOG
+"""
+import json
+import sys
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+PORT = int(sys.argv[1])
+LOG = sys.argv[2]
+
+RUN_REPLY = (
+    "I will create the file.\n"
+    "```run\n"
+    "printf 'hi' > hello.txt\n"
+    "```\n"
+)
+READ_REPLY = (
+    "Checking it.\n"
+    "```sj:read\n"
+    '{"path": "hello.txt"}\n'
+    "```\n"
+)
+DONE_REPLY = "The file is in place.\nDone."
+
+SCRIPT = [RUN_REPLY, READ_REPLY, DONE_REPLY]
+calls = {"n": 0}
+
+
+class Handler(BaseHTTPRequestHandler):
+    def log_message(self, *args):  # quiet
+        pass
+
+    def _json(self, obj):
+        body = json.dumps(obj).encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def do_GET(self):
+        if self.path == "/api/tags":
+            self._json({"models": [{"name": "fake-model"}]})
+        else:
+            self._json({"error": "not found"})
+
+    def do_POST(self):
+        if self.path != "/api/generate":
+            self._json({"error": "not found"})
+            return
+        length = int(self.headers.get("Content-Length", "0"))
+        payload = json.loads(self.rfile.read(length).decode())
+        with open(LOG, "a") as fh:
+            fh.write(json.dumps(payload.get("prompt", "")) + "\n")
+        i = min(calls["n"], len(SCRIPT) - 1)
+        calls["n"] += 1
+        self._json({"model": "fake-model", "response": SCRIPT[i], "done": True})
+
+
+if __name__ == "__main__":
+    ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()

greet.txt

@@ -0,0 +1 @@
+hi

pkg/mod.py

@@ -0,0 +1 @@
+x = 1

server/config.toml.example

@@ -12,6 +12,9 @@ host = "127.0.0.1"
 base_url = "http://127.0.0.1:11434"
 # Default model used when a client does not request a specific one.
 model = "llama3.2"
+# Context window the daemon requests from ollama. The agent sends long
+# transcripts (system prompt + tool results), so do not drop this below 8192.
+num_ctx = 16384
 
 [database]
 # SQLite file for user accounts. Relative to the project root.

server/server.py

@@ -24,6 +24,7 @@ import cmd
 import hashlib
 import hmac
 import json
+import os
 import sqlite3
 import sys
 import threading
@@ -35,7 +36,9 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parent
-CONFIG_FILE = ROOT / "config.toml"
+# SJCODE_CONFIG lets you run a second daemon (tests, staging) without
+# touching the regular config; the database is created next to that file.
+CONFIG_FILE = Path(os.environ.get("SJCODE_CONFIG") or (ROOT / "config.toml"))
 DB_FILE = ROOT / "sjcode.db"  # overwritten by config value once loaded
 
 PEPPER = b"sjcode-v1"
@@ -43,7 +46,8 @@ SESSION_TTL = 24 * 3600  # login token valid for one day
 
 CONF: dict = {
     "server": {"host": "127.0.0.1", "port": 8420},
-    "ollama": {"base_url": "http://127.0.0.1:11434", "model": "llama3.2"},
+    "ollama": {"base_url": "http://127.0.0.1:11434", "model": "llama3.2",
+               "num_ctx": 16384},
     "database": {"path": "sjcode.db"},
     "limits": {"user": 120, "admin": -1},
 }
@@ -66,7 +70,7 @@ def load_config() -> None:
     for section, values in loaded.items():
         if isinstance(values, dict) and isinstance(CONF.get(section), dict):
             CONF[section].update(values)
-    DB_FILE = ROOT / CONF["database"]["path"]
+    DB_FILE = CONFIG_FILE.parent / CONF["database"]["path"]
 
 
 # --------------------------------------------------------------------------
@@ -232,15 +236,62 @@ def ollama_json(path: str, payload: dict | None = None, timeout: int = 300) -> t
         return 502, {"error": f"ollama not reachable: {exc}"}
 
 
-def ollama_generate(prompt: str, model: str | None) -> tuple[int, str]:
+def ollama_stream(path: str, payload: dict) -> tuple[int, object]:
-    status, body = ollama_json(
+    """POST to ollama and return the raw response to iterate line by line."""
-        "/api/generate",
+    url = CONF["ollama"]["base_url"].rstrip("/") + path
-        {
+    data = json.dumps(payload).encode()
+    req = urllib.request.Request(
+        url, data=data, headers={"Content-Type": "application/json"}
+    )
+    try:
+        resp = urllib.request.urlopen(req, timeout=600)
+    except urllib.error.HTTPError as exc:
+        try:
+            body = json.loads(exc.read().decode())
+        except Exception:
+            body = {"error": str(exc)}
+        return exc.code, str(body.get("error", "ollama error"))
+    except (urllib.error.URLError, TimeoutError, ConnectionError) as exc:
+        return 502, {"error": f"ollama not reachable: {exc}"}
+    return resp.status, resp  # caller iterates the lines and closes it
+
+
+def ollama_generate(prompt: str, model: str | None, stream: bool = False):
+    """Generate a reply. Returns (status, text) or (200, token-iterator)."""
+    options = {"num_ctx": int(CONF["ollama"].get("num_ctx", 16384))}
+    payload = {
         "model": model or CONF["ollama"]["model"],
         "prompt": prompt,
-            "stream": False,
+        "stream": stream,
-        },
+        "options": options,
-    )
+    }
+    if stream:
+        status, body = ollama_stream("/api/generate", payload)
+        if status != 200:
+            return status, str(body)
+
+        def tokens():
+            try:
+                for raw in body:
+                    line = raw.decode() if isinstance(raw, bytes) else raw
+                    if not line.strip():
+                        continue
+                    try:
+                        obj = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    if obj.get("error"):
+                        yield f"\n[ollama error: {obj['error']}]"
+                        return
+                    piece = obj.get("response", "")
+                    if piece:
+                        yield piece
+            finally:
+                body.close()
+
+        return 200, tokens()
+
+    status, body = ollama_json("/api/generate", payload)
     if status != 200:
         return status, str(body.get("error", "ollama error"))
     return 200, str(body.get("response", ""))
@@ -259,7 +310,8 @@ def list_models() -> tuple[int, list[str]]:
 # --------------------------------------------------------------------------
 
 class ApiHandler(BaseHTTPRequestHandler):
-    server_version = "sjcode-daemon/0.1"
+    server_version = "sjcode-daemon/0.2"
+    protocol_version = "HTTP/1.1"  # required for chunked streamed replies
 
     # -- helpers ----------------------------------------------------------
     def _json(self, status: int, payload: dict) -> None:
@@ -277,8 +329,48 @@ class ApiHandler(BaseHTTPRequestHandler):
         except (ValueError, json.JSONDecodeError):
             return {}
 
-    def log_message(self, fmt: str, *args) -> None:  # quiet by default
+    def _respond_stream(self, tokens, model: str, started: float,
-        return
+                        username: str) -> None:
+        """Stream a reply as JSON lines: {"chunk": ...} ... {"response": ...}."""
+        self.send_response(200)
+        self.send_header("Content-Type", "application/x-ndjson")
+        self.send_header("Transfer-Encoding", "chunked")
+        self.end_headers()
+
+        def send_line(obj: dict) -> None:
+            data = (json.dumps(obj) + "\n").encode()
+            self.wfile.write(f"{len(data):x}\r\n".encode() + data + b"\r\n")
+            self.wfile.flush()
+
+        parts: list[str] = []
+        abandoned = False
+        try:
+            for piece in tokens:
+                parts.append(piece)
+                send_line({"chunk": piece})
+        except (BrokenPipeError, ConnectionResetError):
+            abandoned = True  # client went away; still bill the usage below
+        except OSError as exc:
+            parts.append(f"\n[stream error: {exc}]")
+            send_line({"chunk": f"\n[stream error: {exc}]"})
+        finally:
+            text = "".join(parts)
+            elapsed = time.monotonic() - started
+            used_min = elapsed / 60.0
+            add_used_minutes(username, used_min)
+            if not abandoned:
+                left_now = remaining_minutes(username)
+                send_line({
+                    "response": text,
+                    "model": model,
+                    "used_minutes": round(used_min, 4),
+                    "remaining_minutes": None if left_now == float("inf")
+                    else round(left_now, 2),
+                })
+                self.wfile.write(b"0\r\n\r\n")
+                self.wfile.flush()
+                print(f"[chat] {username}: {len(text)} chars in "
+                      f"{elapsed:.1f}s (streamed)")
 
     # -- routes -----------------------------------------------------------
     def do_GET(self) -> None:  # noqa: N802 (http.server API)
@@ -353,9 +445,15 @@ class ApiHandler(BaseHTTPRequestHandler):
             return
 
         model = str(data.get("model") or "") or CONF["ollama"]["model"]
+        want_stream = bool(data.get("stream"))
 
         started = time.monotonic()
-        status, answer = ollama_generate(message, model)
+        status, answer = ollama_generate(message, model, stream=want_stream)
+
+        if want_stream and status == 200:
+            self._respond_stream(answer, model, started, username)
+            return
+
         elapsed = time.monotonic() - started
 
         if status != 200:
@@ -375,6 +473,9 @@ class ApiHandler(BaseHTTPRequestHandler):
             },
         )
 
+    def log_message(self, fmt: str, *args) -> None:  # quiet by default
+        return
+
 
 # --------------------------------------------------------------------------
 # admin CLI

sub/marker.txt

@@ -0,0 +1 @@
+x

test_client.py

@@ -0,0 +1,277 @@
+#!/usr/bin/env python3
+"""Tests for the sjcode client agent layer.
+
+Run:  python3 -m unittest test_client.py -v
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import client as c  # noqa: E402
+
+
+class ClientTestCase(unittest.TestCase):
+    """Each test runs in its own temp dir which becomes the project dir."""
+
+    def setUp(self) -> None:
+        self._old_cwd = c.CWD
+        self._old_chdir = os.getcwd()
+        self._tmp = tempfile.TemporaryDirectory()
+        c.CWD = Path(self._tmp.name).resolve()
+        os.chdir(c.CWD)  # tools use CWD; keep relative paths consistent
+
+    def tearDown(self) -> None:
+        os.chdir(self._old_chdir)
+        c.CWD = self._old_cwd
+        self._tmp.cleanup()
+
+
+class TestParsing(ClientTestCase):
+    def test_extracts_run_and_json_tools_in_order(self) -> None:
+        raw = (
+            "Let me check the repo.\n"
+            "```run\n"
+            "git status --short\n"
+            "```\n"
+            "And now the file:\n"
+            "```sj:read\n"
+            '{"path": "a.py", "limit": 10}\n'
+            "```\n"
+        )
+        tags = c.read_tool_tags(raw)
+        self.assertEqual([("run", "git status --short"),
+                          ("read", '{"path": "a.py", "limit": 10}')], tags)
+
+    def test_language_fence_is_run_tool(self) -> None:
+        self.assertEqual(c.read_tool_tags("```run\necho hi\n```"),
+                         [("run", "echo hi")])
+
+    def test_language_fence_is_not_a_run_tool(self) -> None:
+        self.assertEqual(c.read_tool_tags("```rust\nfn main() {}\n```"), [])
+
+    def test_text_before_and_after_is_stripped_from_reply(self) -> None:
+        raw = "Thinking...\n```run\necho hi\n```\nDone."
+        had, reply, results = c.process_reply(raw, auto=True)
+        self.assertTrue(had)
+        self.assertNotIn("```", reply)
+        self.assertNotIn("echo hi", reply)
+
+    def test_body_with_triple_backticks_inside_json_survives(self) -> None:
+        body = '{"path": "x.md", "content": "code: ```py\\nprint(1)\\n```\\n"}'
+        raw = f"```sj:write\n{body}\n```\n"
+        self.assertEqual(c.read_tool_tags(raw), [("write", body)])
+
+    def test_unclosed_fence_still_captured(self) -> None:
+        self.assertEqual(c.read_tool_tags("```run\necho hi"),
+                         [("run", "echo hi")])
+
+
+class TestRunTool(ClientTestCase):
+    def test_runs_command_and_returns_output(self) -> None:
+        ok, lines = c.handle_tool_call("run", '{"command": "echo hello"}', auto=True)
+        self.assertTrue(ok)
+        self.assertIn("hello", "\n".join(lines))
+
+    def test_nonzero_exit_reported(self) -> None:
+        ok, lines = c.handle_tool_call(
+            "run", '{"command": "python3 -c \x27import sys; sys.exit(3)\x27"}',
+            auto=True)
+        self.assertFalse(ok)
+        self.assertIn("exit 3", "\n".join(lines))
+
+    def test_cwd_argument_scopes_the_command(self) -> None:
+        os.mkdir("sub")
+        Path("sub/marker.txt").write_text("x")
+        ok, lines = c.handle_tool_call(
+            "run", '{"command": "pwd", "cwd": "sub"}', auto=True)
+        self.assertTrue(ok)
+        self.assertIn("sub", "\n".join(lines))
+
+    def test_run_outside_project_dir_is_refused(self) -> None:
+        ok, lines = c.handle_tool_call(
+            "run", json.dumps({"command": "pwd", "cwd": "/tmp"}), auto=True)
+        self.assertFalse(ok)
+        self.assertIn("outside the project directory", "\n".join(lines))
+
+    def test_missing_command_is_refused(self) -> None:
+        ok, lines = c.handle_tool_call("run", "{}", auto=True)
+        self.assertFalse(ok)
+        self.assertIn("command", "\n".join(lines))
+
+
+class TestFileTools(ClientTestCase):
+    def test_read_shows_numbered_lines(self) -> None:
+        Path("a.txt").write_text("one\ntwo\nthree\n")
+        ok, lines = c.handle_tool_call("read", '{"path": "a.txt"}', auto=True)
+        self.assertTrue(ok)
+        joined = "\n".join(lines)
+        self.assertIn("1\tone", joined)
+        self.assertIn("3\tthree", joined)
+
+    def test_read_supports_start_and_limit(self) -> None:
+        Path("a.txt").write_text("\n".join(str(i) for i in range(1, 51)))
+        ok, lines = c.handle_tool_call(
+            "read", '{"path": "a.txt", "start": 40, "limit": 5}', auto=True)
+        self.assertIn("40\t40", "\n".join(lines))
+
+    def test_read_missing_file_gives_error_back(self) -> None:
+        ok, lines = c.handle_tool_call("read", '{"path": "nope.txt"}', auto=True)
+        self.assertFalse(ok)
+        self.assertIn("not a file", "\n".join(lines))
+
+    def test_files_lists_and_skips_junk_dirs(self) -> None:
+        os.mkdir("pkg")
+        Path("pkg/mod.py").write_text("x = 1\n")
+        os.mkdir(".git")
+        Path(".git/config").write_text("x\n")
+        ok, lines = c.handle_tool_call("files", '{"path": "."}', auto=True)
+        joined = "\n".join(lines)
+        self.assertIn("pkg/mod.py", joined)
+        self.assertNotIn(".git/config", joined)
+
+    def test_write_creates_file_after_confirmation(self) -> None:
+        with mock.patch("client.confirm_tool", return_value="y"):
+            ok, lines = c.handle_tool_call(
+                "write", '{"path": "new.txt", "content": "hi\\n"}', auto=False)
+        self.assertTrue(ok)
+        self.assertEqual(Path("new.txt").read_text(), "hi\n")
+
+    def test_write_refused_outside_project(self) -> None:
+        ok, lines = c.handle_tool_call(
+            "write",
+            json.dumps({"path": "../../evil.txt", "content": "x"}),
+            auto=True)
+        self.assertFalse(ok)
+        self.assertIn("outside the project directory", "\n".join(lines))
+
+    def test_edit_replaces_exact_text(self) -> None:
+        Path("app.py").write_text("def main():\n    print('old')\n")
+        with mock.patch("client.confirm_tool", return_value="y"):
+            ok, lines = c.handle_tool_call(
+                "edit",
+                json.dumps({"path": "app.py", "old": "print('old')",
+                            "new": "print('new')"}),
+                auto=False)
+        self.assertTrue(ok)
+        self.assertIn("print('new')", Path("app.py").read_text())
+
+    def test_edit_requires_unique_match_without_all(self) -> None:
+        Path("app.py").write_text("x = 1\nx = 1\n")
+        ok, lines = c.handle_tool_call(
+            "edit", json.dumps({"path": "app.py", "old": "x = 1", "new": "y"}),
+            auto=True)
+        self.assertFalse(ok)
+        self.assertIn("matches 2 times", "\n".join(lines))
+
+    def test_edit_with_all_flag_replaces_every_match(self) -> None:
+        Path("app.py").write_text("x = 1\nx = 1\n")
+        with mock.patch("client.confirm_tool", return_value="y"):
+            ok, lines = c.handle_tool_call(
+                "edit",
+                json.dumps({"path": "app.py", "old": "x = 1", "new": "y",
+                            "all": True}),
+                auto=False)
+        self.assertTrue(ok)
+        self.assertEqual(Path("app.py").read_text(), "y\ny\n")
+
+    def test_edit_not_found_gives_actionable_error(self) -> None:
+        Path("app.py").write_text("hello\n")
+        ok, lines = c.handle_tool_call(
+            "edit", json.dumps({"path": "app.py", "old": "zzz", "new": "y"}),
+            auto=True)
+        self.assertFalse(ok)
+        self.assertIn("search text not found", "\n".join(lines))
+
+    def test_overwrite_of_big_existing_file_refused(self) -> None:
+        Path("big.txt").write_text("x" * 3000)
+        ok, lines = c.handle_tool_call(
+            "write", json.dumps({"path": "big.txt", "content": "y"}), auto=True)
+        self.assertFalse(ok)
+        self.assertIn("use 'edit' instead", "\n".join(lines))
+
+
+class TestConfirmation(ClientTestCase):
+    def test_declined_call_returns_cancelled_to_model(self) -> None:
+        Path("f.txt").write_text("a\n")
+        with mock.patch("client.confirm_tool", return_value="n"):
+            ok, lines = c.handle_tool_call(
+                "edit", json.dumps({"path": "f.txt", "old": "a", "new": "b"}),
+                auto=False)
+        self.assertFalse(ok)
+        self.assertIn("cancelled by user", "\n".join(lines))
+
+    def test_user_instruction_is_passed_to_model(self) -> None:
+        Path("f.txt").write_text("a\n")
+        with mock.patch("client.confirm_tool",
+                        return_value="use python instead"):
+            ok, lines = c.handle_tool_call("run", '{"command": "ls -la"}',
+                                           auto=False)
+        self.assertFalse(ok)
+        self.assertIn("user says: use python instead", "\n".join(lines))
+
+    def test_abort_raises_toolabort(self) -> None:
+        with mock.patch("client.confirm_tool", return_value=c.ABORT):
+            with self.assertRaises(c.ToolAbort):
+                c.handle_tool_call("run", '{"command": "ls"}', auto=False)
+
+    def test_auto_mode_skips_confirmation(self) -> None:
+        with mock.patch("client.confirm_tool",
+                        side_effect=AssertionError("must not be called")):
+            ok, lines = c.handle_tool_call("run", '{"command": "echo ok"}',
+                                           auto=True)
+        self.assertTrue(ok)
+
+
+class TestAgentTurn(ClientTestCase):
+    def _chat(self, replies):
+        """Patch api_chat to serve canned replies in order."""
+        calls = {"n": 0}
+
+        def fake(entry, token, message, model, on_chunk=None):
+            i = calls["n"]
+            calls["n"] += 1
+            return 200, {"response": replies[min(i, len(replies) - 1)],
+                         "remaining_minutes": 99}
+
+        return mock.patch("client.api_chat", side_effect=fake)
+
+    def test_tool_loop_feeds_results_and_finishes_on_done(self) -> None:
+        Path("greet.txt").write_text("hi\n")
+        replies = [
+            'Reading the file.\n```sj:read\n{"path": "greet.txt"}\n```',
+            '```sj:write\n{"path": "out.txt", "content": "done"}\n```',
+            "Everything is in place.\nDone.",
+        ]
+        with self._chat(replies):
+            status, final, _ = c.agent_turn({}, "t", "m", [], "make out.txt",
+                                            auto=True)
+        self.assertEqual(status, "done")
+        self.assertEqual(Path("out.txt").read_text(), "done\n")
+        self.assertEqual(final, "Everything is in place.\nDone.")
+
+    def test_plain_text_reply_ends_turn_without_tool_loop(self) -> None:
+        with self._chat(["Just answering."]):
+            status, final, _ = c.agent_turn({}, "t", "m", [], "hello", auto=True)
+        self.assertEqual(status, "done")
+        self.assertEqual(final, "Just answering.")
+
+    def test_error_result_is_sent_back_to_model(self) -> None:
+        replies = [
+            '```sj:read\n{"path": "missing.txt"}\n```',
+            "Task complete.\nDone.",
+        ]
+        with self._chat(replies):
+            status, _, _ = c.agent_turn({}, "t", "m", [], "read it", auto=True)
+        self.assertEqual(status, "done")
+
+
+if __name__ == "__main__":
+    unittest.main()

test_e2e.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+"""End-to-end test: fake ollama -> real daemon -> real client agent loop.
+
+Exercises the whole chain the same way a user would: the client connects,
+logs in, gets a task, confirms a shell command, and the agent actually
+creates a file in the working directory.
+
+Run:  python3 test_e2e.py
+"""
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+import urllib.request
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+
+
+def free_ports(count: int) -> list[int]:
+    """Find unused ports by binding ephemeral sockets (avoids TIME_WAIT)."""
+    ports, sockets = [], []
+    while len(ports) < count:
+        s = socket.socket()
+        s.bind(("127.0.0.1", 0))
+        sockets.append(s)
+        ports.append(s.getsockname()[1])
+    for s in sockets:
+        s.close()
+    return ports
+
+
+class E2E(unittest.TestCase):
+    def setUp(self):
+        self.ollama_port, self.daemon_port = free_ports(2)
+        self.tmp = Path(tempfile.mkdtemp(prefix="sj-e2e-"))
+        self.server_dir = self.tmp / "server"
+        self.work = self.tmp / "work"
+        self.server_dir.mkdir()
+        self.work.mkdir()
+        (self.server_dir / "config.toml").write_text(
+            f'[server]\nhost = "127.0.0.1"\nport = {self.daemon_port}\n'
+            f'\n[ollama]\nbase_url = "http://127.0.0.1:{self.ollama_port}"\n'
+            'model = "fake-model"\nnum_ctx = 16384\n'
+            '\n[database]\npath = "sjcode.db"\n'
+            "\n[limits]\nuser = 120\nadmin = -1\n"
+        )
+        self.prompt_log = self.tmp / "prompts.log"
+
+        self.fake = subprocess.Popen(
+            [sys.executable, str(ROOT / "fake_ollama.py"),
+             str(self.ollama_port), str(self.prompt_log)],
+            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        env = dict(os.environ, SJCODE_CONFIG=str(self.server_dir / "config.toml"))
+        self.daemon = subprocess.Popen(
+            [sys.executable, str(ROOT / "server" / "server.py")],
+            cwd=self.server_dir, env=env, stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+        self._wait_ready()
+        # create the test account through the daemon admin CLI
+        self.daemon.stdin.write("create tester admin\nsecret\nsecret\n")
+        self.daemon.stdin.flush()
+        time.sleep(0.5)  # let the admin CLI process the lines
+
+    def _wait_ready(self, timeout: float = 10.0) -> None:
+        deadline = time.time() + timeout
+        url = f"http://127.0.0.1:{self.daemon_port}/status"
+        while time.time() < deadline:
+            try:
+                with urllib.request.urlopen(url, timeout=2) as resp:
+                    if resp.status == 200:
+                        return
+            except OSError:
+                time.sleep(0.1)
+        self.fail("daemon did not come up in time")
+
+    def tearDown(self):
+        for proc in (self.daemon, self.fake):
+            if proc.poll() is None:
+                proc.terminate()
+        for proc in (self.daemon, self.fake):
+            try:
+                proc.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def _client_run(self, inputs: list[str]) -> subprocess.CompletedProcess:
+        script = "\n".join(inputs) + "\n"
+        # keep the client's saved-server config inside the test sandbox
+        env = dict(os.environ, XDG_CONFIG_HOME=str(self.tmp / "cfg"))
+        return subprocess.run(
+            [sys.executable, str(ROOT / "client.py"),
+             f"127.0.0.1:{self.daemon_port}", "--model", "fake-model"],
+            cwd=self.work, env=env, input=script, capture_output=True,
+            text=True, timeout=120)
+
+    def test_full_agent_run_with_confirmation(self):
+        proc = self._client_run([
+            "tester",     # username
+            "secret",     # password
+            "create hello.txt with the word hi in it",
+            "y",          # confirm the run tool call
+            "/quit",
+        ])
+        self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
+        self.assertIn("logged in as tester", proc.stdout)
+        self.assertIn("Done.", proc.stdout)
+
+        hello = self.work / "hello.txt"
+        self.assertTrue(hello.exists(), "agent did not create hello.txt")
+        self.assertEqual(hello.read_text().strip(), "hi")
+
+        # the model saw the REAL command output, not invented text
+        prompts = [json.loads(line)
+                   for line in self.prompt_log.read_text().splitlines()]
+        self.assertEqual(len(prompts), 4)  # run, read, done, done-again
+        self.assertIn("printf", prompts[1])          # run result fed back
+        self.assertIn("Tool results", prompts[1])
+        self.assertIn("1\thi", prompts[2])           # read result fed back
+
+
+if __name__ == "__main__":
+    unittest.main()