Allan's Blog
Friday, September 25, 2026
A Jev-like wrapper for LLMs, including vision models
I was intrigued by Jev and the self-hostable projects appearing around it, such as OpenJev and SemIf . Reading about them introduced me to a neat trick: reading an LLM's token probabilities.
Apparently this is an old trick for some people. See e.g. OpenAI's logprobs cookbook . But it was new to me.
I believe the basic idea is to write a prompt like this:
State: My order arrived broken and I want a refund. Question: Which team should handle this? [A] billing [B] shipping [C] returns Answer with the letter of the best option only.
Then add a few JSON request parameters to a compatible Chat Completions request:
{ "max_completion_tokens": 1, "logprobs": true, "top_logprobs": 20 }
The LLM API will return the letter plus the model's log probabilities for alternative tokens.
Repeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it.
The fun part: this works with vision models too. Jev's documented request format currently describes only text/JSON state. I added an attachments field for images for my local experiments.
My example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? With Gemma 4 12B on my RTX 3090, I get around 1 frames per second , with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn't make any effort to avoid the cost of a separate connection through their system per question per frame.
Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text.
Here's the standalone Python example (OpenCV is just used for convenient access to the webcam, not for any actual computer vision):
#!/usr/bin/env -S uv run --script # /// script # dependencies = ["opencv-python"] # /// """Preview and score webcam frames with llama.cpp or OpenAI. uv run webcam.py uv run webcam.py https://api.openai.com/v1 gpt-6-luna OpenAI reads OPENAI_API_KEY. """ import argparse import base64 import concurrent.futures import datetime import json import math import mimetypes import os import pathlib import time import urllib.parse import urllib.request import cv2 # attachments is our custom addition to the Jev request format. data = json.loads(""" { "state": "Inspect this webcam frame. Judge only what is visibly present.", "attachments": [], "questions": { "person": { "type": "noul", "instructions": "Is a person visible?" }, "plant": { "type": "noul", "instructions": "Is a plant visible?" }, "setting": { "type": "choice", "instructions": "Where is the camera?", "criteria": { "indoors": null, "outdoors": null, "unclear": null } }, "light": { "type": "score", "instructions": "How bright is the scene?", "criteria": [ "dark", "dim", "bright" ] } } } """) def score(data, url, model): state = data["state"] if not isinstance(state, str): state = json.dumps(state) # Attachments are our extension to the Jev-style request format: # image file paths or base64 data URLs. Load them once for all questions. images = [] for attachment in data.get("attachments", []): if attachment.startswith("data:image/"): images.append(attachment) continue path = pathlib.Path(attachment).expanduser() mime_type, _ = mimetypes.guess_type(path) if mime_type not in {"image/png", "image/jpeg", "image/webp", "image/gif"}: raise ValueError(f"Unsupported image file: {path}") encoded = base64.b64encode(path.read_bytes()).decode() images.append(f"data:{mime_type};base64,{encoded}") # Send the API key only to OpenAI. is_openai = urllib.parse.urlsplit(url).hostname == "api.openai.com" headers = {"Content-Type": "application/json"} if is_openai: headers["Authorization"] = "Bearer " + os.environ["OPENAI_API_KEY"] answers = {} for name, question in data["questions"].items(): # Represent choices, booleans, and ordinal levels as lettered options. if question["type"] == "choice": options = question["criteria"] elif question["type"] == "noul": options = {"true": None, "false": None} | question.get("criteria", {}) elif question["type"] == "score": options = {str(i): description for i, description in enumerate(question["criteria"])} else: raise ValueError(f"Unknown question type: {question['type']}") if not 2 <= len(options) <= 20: raise ValueError("Provide 2 to 20 criteria per question.") letters = "ABCDEFGHIJKLMNOPQRST"[:len(options)] # Ask for a single option letter, so its logprob represents that option. instructions = question["instructions"] if not isinstance(instructions, str): instructions = json.dumps(instructions) lines = [f"State:\n{state}\n\nQuestion: {instructions}\nOptions:"] for letter, (key, description) in zip(letters, options.items()): line = f"[{letter}] {key}" if description is not None: line += f": {description}" lines.append(line) prompt = "\n".join(lines) + "\n\nAnswer with the letter of the best option only." # OpenAI needs Responses for enough alternatives; llama.cpp needs Chat for logprobs. # top_p=1 avoids pruning alternatives. if is_openai: endpoint = "/responses" content = [{"type": "input_text", "text": prompt}] content.extend({"type": "input_image", "image_url": image} for image in images) body = { "model": model, "input": [{"role": "user", "content": content}], "reasoning": {"effort": "none"}, "max_output_tokens": 16, "top_p": 1, "top_logprobs": 20, "include": ["message.output_text.logprobs"], } else: endpoint = "/chat/completions" content = [{"type": "text", "text": prompt}] content.extend({"type": "image_url", "image_url": {"url": image}} for image in images) body = { "model": model, "messages": [{"role": "user", "content": content}], "max_completion_tokens": 1, "temperature": 0, "reasoning_effort": "none", "logprobs": True, "top_logprobs": 1024, } # Send the request and read the first output token's alternatives. request = urllib.request.Request( url.rstrip("/") + endpoint, headers=headers, data=json.dumps(body).encode(), ) with urllib.request.urlopen(request) as response: result = json.load(response) if is_openai: message = next(item for item in result["output"] if item["type"] == "message") candidates = message["content"][0]["logprobs"][0]["top_logprobs"] else: candidates = result["choices"][0]["logprobs"]["content"][0]["top_logprobs"] logprobs = {item["token"]: item["logprob"] for item in candidates} # Normalize the returned option scores; missing options initially get zero. missing = [letter for letter in letters if letter not in logprobs or logprobs[letter] <= -9999] if len(missing) == len(letters): raise ValueError("API did not return usable scores for any option") peak = max(logprobs[letter] for letter in letters if letter not in missing) weights = [math.exp(logprobs[letter] - peak) if letter not in missing else 0 for letter in letters] total = sum(weights) # An omitted token cannot outrank the last returned alternative. # Allow zero only when their combined normalized probability is below 1e-6. if missing: cutoff = min(value for value in logprobs.values() if value > -9999) missing_weight = len(missing) * math.exp(cutoff - peak) if missing_weight / (total + missing_weight) >= 1e-6: raise ValueError(f"API omitted non-negligible option scores for: {', '.join(missing)}") probabilities = {key: weight / total for key, weight in zip(options, weights)} # Return the winning choice, probability of true, or expected ordinal level. if question["type"] == "choice": answers[name] = { "type": "choice", "choice": max(probabilities, key=probabilities.get), "probabilities": probabilities, } elif question["type"] == "noul": answers[name] = {"type": "noul", "noul": probabilities["true"]} else: answers[name] = { "type": "score", "score": sum(int(key) * probability for key, probability in probabilities.items()), "legend": options, "probabilities": probabilities, } return {"answers": answers} # Choose the server and model before opening the camera. parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("url", nargs="?", default="http://localhost:8060/v1") parser.add_argument("model", nargs="?", default="gemma-4-12b") args = parser.parse_args() # Point OpenCV's bundled Qt at the installed system fonts. os.environ["QT_QPA_FONTDIR"] = "/usr/share/fonts/truetype/noto" # Open the default Linux webcam with a small capture buffer. camera = cv2.VideoCapture(0, cv2.CAP_V4L2) if not camera.isOpened(): raise RuntimeError("Could not open /dev/video0") camera.set(cv2.CAP_PROP_BUFFERSIZE, 1) print(f"Webcam -> {args.model}. Noul: yes %; score: value/max. Ctrl-C or Esc to stop.", flush=True) print(f"{'time':<8}" + "".join(f"{name:>10}" for name in data["questions"]) + f"{'fps':>10}", flush=True) # Preview continuously while a background worker scores one frame at a time. executor = concurrent.futures.ThreadPoolExecutor(max_workers=1) pending = None try: while True: ok, frame = camera.read() if not ok: raise RuntimeError("Could not read a webcam frame") cv2.imshow("Webcam", frame) if cv2.waitKey(1) == 27 or cv2.getWindowProperty("Webcam", cv2.WND_PROP_VISIBLE) < 1: break # Print a completed result, then submit the latest frame. if pending is not None: if not pending.done(): continue result = pending.result() columns = [] for name in data["questions"]: answer = result["answers"][name] if answer["type"] == "noul": value = f"{answer['noul']:.1%}" elif answer["type"] == "choice": value = answer["choice"] else: value = f"{answer['score']:.2f}/{len(data['questions'][name]['criteria']) - 1}" columns.append(f"{value:>10}") columns.append(f"{1 / (time.perf_counter() - started):>10.2f}") print(captured + "".join(columns), flush=True) # Measure throughput for evaluated frames, including image encoding. started = time.perf_counter() captured = datetime.datetime.now().strftime("%H:%M:%S") ok, jpeg = cv2.imencode(".jpg", frame) if not ok: raise RuntimeError("Could not encode the webcam frame") image = "data:image/jpeg;base64," + base64.b64encode(jpeg.tobytes()).decode() data["attachments"] = [image] pending = executor.submit(score, data, args.url, args.model) except KeyboardInterrupt: print("\nStopped.") finally: camera.release() cv2.destroyAllWindows() executor.shutdown()
The script handles the API differences: llama.cpp uses Chat Completions and OpenAI uses Responses to get it to show alternatives.
I ran Gemma 4 12B QAT through llama.cpp. On Linux with NVIDIA drivers, curl , zstd , and uv installed:
# Model (~7 GB) and multimodal projector (~175 MB). mkdir -p ~/models/gemma-4-12b/ cd ~/models/gemma-4-12b/ curl -fL -C - -o gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/gemma-4-12b-it-qat-q4_0.gguf curl -fL -C - -o mmproj-gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf # Standalone llama.cpp binary for RTX 3090 (CUDA architecture 86). curl -fL -o llama.zst https://huggingface.co/buckets/ggml-org/install.sh/resolve/b11160/x86_64/linux/cuda/86/llama-app.zst mkdir -p ~/bin/ zstd -d llama.zst -o ~/bin/llama chmod +x ~/bin/llama ~/bin/llama serve --models-dir ~/models/ --port 8060
Save the Python example as webcam.py . In another terminal, from that directory:
uv run webcam.py http://localhost:8060/v1 gemma-4-12b # Or use OpenAI, with OPENAI_API_KEY set in your environment. uv run webcam.py https://api.openai.com/v1 gpt-6-luna
1 comment:
Anonymous 26 September, 2026 06:55 Current transformer models only attend to past tokens when processing input. So, you probably want the options _before_ your input so that every “word” on the state will be processed taking into account the options. If you do that then you can disable reasoning saving some time (and money) and maybe improve radically the results. Reply Delete Replies Reply
Blog Archive
▼ 2026 (1) ▼ September (1) A Jev-like wrapper for LLMs, including vision models