Hyderabad, IN

17°23'N78°29'E

Versioned.AI

Overview

Versioned.AI is a local workbench for test-driven prompt engineering. Running npx versioned-ai dev starts a web UI on localhost with nothing to install. A prompt_eval.yaml file in the project defines the providers, prompts and test cases, and every test runs against every configured provider.

The problem

A prompt change can quietly break behaviour, and the same prompt can behave differently from one model to the next. Checking that by hand across providers is slow and hard to repeat, and it doesn't fit into a CI pipeline. Each repeat run also costs money when it calls a paid API again.

Approach

I treated prompts like code under test. Tests live in one YAML file, assertions are explicit, and the same engine runs behind both the UI and a headless command. The CLI ships as a single npm package with the UI already compiled into it, so a project needs no setup beyond the config file and an API key or a local Ollama instance.

Architecture

  1. CLI and bundled UI

    A monorepo with two packages. The Next.js UI compiles to a static export, which the build copies into the CLI's dist folder. Only the CLI package is published, and its Express server serves both that UI and a local REST API.

  2. Config and execution engine

    The YAML config is validated with Zod. The engine resolves template variables, then runs each test against each provider in parallel with Promise.allSettled, so one failed call doesn't stop the rest.

  3. Provider adapters

    OpenAI, Anthropic, AWS Bedrock and Ollama each have an adapter behind a shared interface. A fifth adapter reuses the OpenAI SDK with a custom base URL for any server that implements /v1/chat/completions. The hosted-API adapters retry rate limits and server errors with exponential backoff and jitter.

  4. Graders

    Five deterministic assertions (is-json, equals-json, matches-schema, contains-substring, latency) sit alongside two model-based ones (llm-rubric, semantic-similarity). Model grading asks an LLM for a JSON verdict with a 0 to 100 score and treats 70 or above as a pass.

  5. SQLite response cache

    Responses are stored in a .prompt_cache.db file in the project, keyed by a SHA-256 hash of the resolved prompts and the provider ID. Identical runs return from the cache without calling the provider. The cache uses sql.js, so it needs no native dependency.

Outcomes

  • The UI's results table and inspector show each test and provider with its response, pass or fail status, latency, token usage and per-assertion messages.
Evaluation matrix, running on demo data

Takeaways

  1. Published to npm as versioned-ai (version 0.1.0).

  2. Five provider integrations and seven assertion types behind one YAML format.

  3. run --ci exits with code 1 on any failed test, so it works in GitHub Actions, GitLab CI and similar systems.