H harnevalHARNESS EVALUATION

DOCUMENTATION / GETTING STARTED

Your harness.
From terminal to leaderboard.

A practical guide to packaging, uploading, and evaluating your harness with the Harneval CLI.

CLI + DockerLinux + Windows/WineOperator-funded model access

01 / Upload your first harness

Your harness is a program you write to investigate another program. It can use analysis tools and an AI model to work out what that program does. You upload your harness as a Docker image, then choose a task for it to solve. Harneval runs your harness, checks its answers, and measures its model usage and cost.

From an installed checkout, activate the CLI and sign in. Use your application password, which can differ from your operating-system password.

bash
cd /home/kali/Desktop/harneval
source .venv/bin/activate
harneval login --url BACKEND_URL --name YOUR_USERNAME

The password is prompted privately. Then follow this sequence:

bash
# 1. Find an executable and copy its target ID
harneval targets

# 2. Build and save your image (plain TAR, not gzip)
docker build -t my-harness ./my-harness
docker save -o my-harness.tar my-harness

# 3. Upload and copy the returned harness ID
harneval upload my-harness.tar --name my-harness

# 4. Start an evaluation and copy its run ID
harneval submit HARNESS_ID TARGET_ID

# 5. Check progress, then inspect the leaderboard
harneval status RUN_ID
harneval leaderboard TARGET_ID

Replace uppercase IDs with the values returned by the previous command. Submission queues the job; it does not wait for completion. Re-run status RUN_ID to follow progress.

02 / Install and authenticate

You need Python 3.11 or newer for the CLI, a provisioned Harneval account, and Docker wherever you build your image. Docker does not need to run on the computer used only for uploading.

From a checkout:

bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
harneval --help

Or install a wheel supplied by your operator with python -m pip install /path/to/harneval.whl. The installed CLI can then be used outside the checkout.

bash
harneval login --url BACKEND_URL --name YOUR_USERNAME

The URL is the API/frontend address, not OpenRouter. Remote servers require HTTPS; HTTP is accepted only for loopback development. Sessions expire after 24 hours. CLI credentials are saved with mode 0600 in ~/.config/harneval/credentials.json, or beneath $XDG_CONFIG_HOME when set. Browser and CLI sign-in are separate sessions.

03 / What a task is and how to solve it

What is a task?

A task is an assignment for your harness: here is a program; answer these questions about it. For example, a task might ask what kind of file it is, what it prints when run with a certain argument, or what exit code it returns.

Harneval gives your harness two files when it starts:

  • The program to investigate. This is the executable file, usually at /input/target.
  • The task instructions. This is a JSON file, usually at /input/task.json. It tells your harness where the program is and which questions to answer. JSON is a text format for storing named values.

You choose a task with harneval targets and pass its target ID to harneval submit. You do not need to create the task file yourself; Harneval puts it in the container for each evaluation.

What should my harness do?

  1. Read the assignment. Open the file named by HARNEVAL_TASK. Read its executable path and questions.
  2. Investigate the program. Inspect the file, run it when useful, or use tools to understand its behavior. Your harness decides which steps to take based on the questions.
  3. Use the AI model as needed. Send the questions and useful findings to the provided model API. Your harness controls what evidence the model sees and what to do with its response.
  4. Save the answers. Write a JSON file to the path in HARNEVAL_OUTPUT. Use the same question IDs as the task, with your answer as each value.
  5. Finish successfully. Exit with code zero. Harneval then compares your saved answers with its private answer sheet and records your score, tokens, and cost.

Package your harness

Your Docker image must contain your harness and everything it needs to run, including its programming language and analysis tools. Build the image before uploading it. Set the command Docker should run when the image starts, for example ENTRYPOINT ["python", "/app/harness.py"].

Input / settingMeaning
HARNEVAL_TASKThe file containing the assignment and questions; normally /input/task.json.
task.executableThe program your harness should investigate. You can read or run it, but cannot change this file. Normally /input/target.
task.questionsThe questions to answer. Each has a short ID, such as format. Use that same ID as the key in your answers file.
HARNEVAL_OUTPUTWrite your final JSON answer object here; normally /tmp/answers.json.
OPENAI_BASE_URL / OPENROUTER_BASE_URLInjected local API base, http://127.0.0.1:8000/v1 inside the container.
OPENAI_API_KEY / OPENROUTER_API_KEYInjected placeholder credentials. The real operator key stays on the host.
HOMEPrivate writable home at /tmp/harneval-home.

Example: the assignment your harness receives

In this example, platform says the program runs on Linux, executable tells your harness which file to investigate, and questions lists what it needs to find out.

json
{
  "platform": "linux",
  "executable": "/input/target",
  "questions": {
    "format": "What is the executable file format?",
    "probe_output": "What line does it print with --probe?",
    "probe_exit": "What exit code does it return with --probe?"
  }
}

The example program supports an argument called --probe. Your harness could run /input/target --probe, read what it prints, and record its exit code. Other tasks may ask different questions or need different commands. Always read the task you are given.

Example: the answers your harness saves

If your investigation found that the file is an ELF program, it prints HARNEVAL_PROBE_OK, and it exits with code 7, your harness would save:

json
{
  "format": "ELF",
  "probe_output": "HARNEVAL_PROBE_OK",
  "probe_exit": "7"
}

Save this object to the file named by HARNEVAL_OUTPUT, usually /tmp/answers.json. Printing answers in the terminal is not enough: Harneval reads the saved file. Keep it under 1 MiB and exit with code zero. Use your own findings for each task; the values above only illustrate this example. Harneval keeps the correct answers private and gives credit for each question you answer correctly.

04 / Build your Python harness

Your harness is a Python program packaged in a Linux Docker image. You do not need Go. Include Python and your analysis libraries in the image; Harneval starts the image’s Python entrypoint inside the sandbox.

The checkout includes examples/harness.py, a complete Python example for the supplied probe tasks. It reads the task, runs the executable, sends the observed evidence to the model, and writes its answers. Adapt inspect_target() for other tasks.

Build and upload the supplied example

bash
# Run from the Harneval checkout.
docker build -t my-python-harness -f examples/Dockerfile examples

# This local helper keeps the outer archive as plain TAR and compresses
# only its image layers, allowing the supplied Python example to fit
# the current 64 MiB upload allowance.
python examples/save_image.py my-python-harness my-python-harness.tar
harneval upload my-python-harness.tar --name my-python-harness
harneval targets
harneval submit HARNESS_ID TARGET_ID

On an installation with a larger upload allowance, ordinary docker save -o my-python-harness.tar my-python-harness also works. Do not gzip the outer TAR. Always check its size before uploading; adding dependencies can push an image past the limit.

Python harness source

The example uses only Python’s standard library. It never executes model-generated code. Target output is written to bounded guest storage and only a small excerpt is sent to the model.

python
"""Python example for the supplied executable probes. Adapt inspect_target for other tasks."""
import json
import os
import subprocess
import urllib.request
from pathlib import Path


def inspect_target(task):
    artifact = Path(task['executable'])
    with artifact.open('rb') as file:
        header = file.read(64)
    arguments = json.loads(os.environ.get('HARNEVAL_PROBE_ARGUMENTS', '["--probe"]'))
    if not isinstance(arguments, list) or any(not isinstance(arg, str) for arg in arguments):
        raise ValueError('Probe arguments must be a JSON array of strings')
    command = [str(artifact), *arguments]
    if task['platform'] == 'windows-wine':
        command.insert(0, 'wine')
    timeout = int(os.environ.get('HARNEVAL_PROBE_TIMEOUT', '30'))
    if not 1 <= timeout <= 120:
        raise ValueError('Probe timeout must be 1–120 seconds')
    evidence = {
        'header_hex': header.hex(),
        'format_from_header': 'ELF' if header.startswith(b'\x7fELF') else 'PE' if header.startswith(b'MZ') else 'unknown',
        'guest_uid': os.getuid(),
        'python_version': list(__import__('sys').version_info[:3]),
    }
    output, diagnostic = Path('/tmp/probe-output.log'), Path('/tmp/probe-stderr.log')
    # The worker bounds file size and tmpfs; never accumulate child output in RAM.
    try:
        with output.open('wb') as stdout, diagnostic.open('wb') as stderr:
            result = subprocess.run(command, stdin=subprocess.DEVNULL, stdout=stdout, stderr=stderr, timeout=timeout, check=False)
        evidence['probe_exit'] = result.returncode
    except (OSError, subprocess.TimeoutExpired) as error:
        evidence['probe_error'] = type(error).__name__
    for name, path in [('probe_output', output), ('probe_stderr', diagnostic)]:
        if path.exists():
            with path.open('rb') as file:
                evidence[name] = file.read(8192).decode(errors='replace')
    return evidence


def evaluate():
    task = json.loads(Path(os.environ['HARNEVAL_TASK']).read_text())
    evidence = inspect_target(task)
    body = {
        'model': os.environ.get('HARNESS_MODEL', 'openai/gpt-4o-mini'),
        'messages': [{'role': 'user', 'content': 'Answer the executable questions using this observed evidence. Return only a JSON object keyed by question ID. For file format use the short name ELF or PE. Do not invent observations. ' + json.dumps({'questions': task['questions'], 'evidence': evidence})}],
        'response_format': {'type': 'json_object'},
        'max_tokens': 1024,
    }
    request = urllib.request.Request(
        os.environ['OPENAI_BASE_URL'].rstrip('/') + '/chat/completions',
        data=json.dumps(body).encode(),
        headers={'Content-Type': 'application/json', 'Authorization': 'Bearer ' + os.environ['OPENAI_API_KEY']},
        method='POST',
    )
    with urllib.request.urlopen(request, timeout=150) as response:
        raw = response.read(8 * 1024 * 1024 + 1)
    if len(raw) > 8 * 1024 * 1024:
        raise ValueError('Model response too large')
    result = json.loads(raw)
    answers = json.loads(result['choices'][0]['message']['content'])
    if not isinstance(answers, dict):
        raise ValueError('Model must return a JSON answer object')
    encoded = json.dumps(answers).encode()
    if len(encoded) > 1024 * 1024:
        raise ValueError('Answer file too large')
    Path(os.environ['HARNEVAL_OUTPUT']).write_bytes(encoded)


if __name__ == '__main__':
    evaluate()

Dockerfile

dockerfile
FROM python:3.13-slim
WORKDIR /app
COPY harness.py /app/harness.py
ENV HARNESS_MODEL=openai/gpt-4o-mini PYTHONDONTWRITEBYTECODE=1
ENTRYPOINT ["python3", "-I", "-B", "/app/harness.py"]
CMD []

Python’s -I starts the interpreter in isolated mode; -B avoids writing bytecode beside your source on the read-only image. Install any required Python packages while building the image. No package downloads are available during an evaluation.

05 / Call the metered model API

Read the injected base URL and placeholder key from the environment. Do not hardcode a public OpenAI/OpenRouter URL or include a real API key in your image. The model must be on the operator’s allowlist.

The available route is POST /v1/chat/completions. Requests support model/messages, temperature, top-p, output-token limits, tools/tool choice, response format, seed, stop, and streaming. Provider routing overrides, plugins, alternate destinations, and other API routes are rejected.

Streaming is buffered: the host receives and accounts for the complete response before returning one SSE completion frame. It is not incremental token delivery. Native token counts and exact provider-reported costs are committed before output is returned.

06 / Windows executables with Wine

Choose a target whose platform is windows-wine. Your uploaded image is still a Linux image, but must include Wine and any libraries needed by your harness and target. Invoke the executable with wine /input/target.

Use a private Wine prefix under $HOME, such as WINEPREFIX=/tmp/harneval-home/wine. Do not install packages or download Wine during an evaluation; there is no Internet access. The checkout includes examples/Dockerfile.wine as a starting recipe.

The known 64-bit console PE fixture has passed an actual Docker/gVisor/Wine evaluation. Wine compatibility depends on the executable. Native Windows kernels, arbitrary GUI programs, and universal Windows compatibility are not provided. Wine images will typically require more than the current local upload allowance.

07 / Follow runs and read results

bash
harneval harnesses             # Your uploaded harness IDs
harneval targets               # Available executable IDs
harneval submit HARNESS_ID TARGET_ID
harneval status RUN_ID         # One run and its metered calls
harneval status                # Your recent runs
harneval leaderboard TARGET_ID
harneval logout
StatusWhat it means
queuedWaiting for the worker. A missing/stopped worker leaves runs queued.
runningThe worker is preparing or executing the image.
finalizingAnswers are graded; container cleanup must finish before publication.
completedEvaluation and cleanup succeeded. Eligible accounted results enter the leaderboard.
failedInspect the run’s error and request accounting states before retrying.
cleanup_requiredContainer removal could not be confirmed. Operator intervention is needed.

A completed run can score below 100%. Scores reflect the fraction of question IDs whose answers match an accepted value after trimming and case normalization. Missing answers lose credit. Rankings use accuracy first, then lower cost, fewer tokens, and finish time.

In the website, sign in, select an executable, and press Refresh to load its current leaderboard and your recent runs. The page does not continuously poll.

Provider timeouts can incur charges even when no response is returned. Uncertain accounting excludes the run from rankings. Ask the operator to reconcile retained generation IDs; blindly rerunning can create another charge. Logout revokes your current CLI session and removes its local credential file.

08 / Runtime limits and permissions

Your image runs as UID/GID 65532, with a read-only root, no capabilities, and no direct Internet access. Make your entrypoint, tools, and libraries readable/executable by that user. Write temporary files under /tmp or the provided private home.

LimitWorker default
CPU / memory / host tasks2 cores / 2 GiB / 128 tasks
Evaluation deadline600 seconds
Temporary filesystem256 MiB for Linux; 1 GiB for Wine
Model calls / output tokens20 calls; up to 8,192 output tokens per call
API request / provider response256 KiB / 8 MiB
Final answer JSON1 MiB
Local upload allowance64 MiB per saved archive; operator configurable

Keep the exact image architecture aligned with the worker. Build dependencies into the image. Docker’s saved-archive size, not just the compressed size shown by a registry, determines whether an upload fits.

Operator allowances and cost limits may be lower. A run’s spending threshold prevents subsequent calls once reached; it does not cap the final accepted call’s charge. You do not need to configure or possess the operator key.

09 / Troubleshooting

Login fails
Use a provisioned application account, the correct backend URL, and its application password. Reauthenticate if your 24-hour session has expired.
413: image archive too large
Check the size of the plain saved TAR. The local limit is 64 MiB. Use a smaller image or request more capacity; gzipping the outer archive is not supported.
Archive preprocessing fails
Submit exactly one Linux image with docker save, not docker export, a Dockerfile, or an outer-compressed archive. Malformed, sparse, oversized, or conflicting archive layouts are rejected.
Image cannot start / permission denied
Check architecture, executable permissions, required interpreters/shared libraries, and the image entrypoint. The guest user is 65532 and cannot write the image filesystem.
Model request fails
Use the injected base URL and key, an allowlisted model, and only supported parameters. Direct public API calls and package downloads have no network access.
Missing answer file / unsuccessful exit
Write a JSON object to the exact HARNEVAL_OUTPUT path and exit zero. Do not rely on stdout. A target returning nonzero may be expected evidence; your harness must handle that separately from its own exit status.
Low score despite a completed run
Match question IDs and answer wording. A descriptive phrase may not match a short accepted label. Completion means the evaluation ran successfully, not that every answer was correct.
Queued forever
Ask the operator to check the worker service. Repeated submission creates additional queued jobs; it does not accelerate the existing one.
Accounting uncertain / cleanup required
Ask the operator to inspect the existing run before submitting again. These conditions deliberately prevent leaderboard publication.
bash
harneval --help
harneval upload --help
harneval submit --help
harneval status RUN_ID
← Back to Harneval