About
ChainForge is an open-source, visual prompt engineering and LLM evaluation platform created by researchers at Harvard's HCI group. It gives developers and researchers a drag-and-drop flow interface to design, run, and compare batches of prompts across multiple large language models simultaneously. Unlike ad-hoc testing, ChainForge lets you systematically measure prompt robustness — for example, testing susceptibility to prompt injection attacks, evaluating consistency when enforcing a specific output format, or measuring the impact of different system messages on ChatGPT's behavior. Results are visualized in-app and can be exported to Excel for further analysis. The platform is available as a no-install web version at chainforge.ai/play, and as a locally-installed Python package (pip install chainforge) that unlocks advanced features: loading API keys from environment variables, writing custom Python evaluators, and querying locally-hosted Alpaca/Llama models via Dalai. ChainForge supports a wide range of use cases — from prompt injection robustness testing and format consistency checks to parametrized prompt batches, OpenAI evals integration, and cross-model performance comparisons. It is actively developed and open to community contributions via GitHub, making it an ideal tool for AI engineers, researchers, and ML practitioners who need reliable, evidence-based prompt evaluation.
Key Features
- Visual Flow Builder: Drag-and-drop node-based interface to design and run complex prompt evaluation pipelines without writing code.
- Multi-LLM Comparison: Send the same prompts to multiple language models simultaneously and compare their outputs side-by-side with statistical rigor.
- Robustness & Injection Testing: Built-in support for testing prompt injection attacks, output format consistency, and the effect of system message variations.
- Parametrized Prompt Batching: Run large batches of parametrized prompts, cache results, and export to Excel — all without custom scripting.
- Local & Custom Evaluators: When installed locally, write Python code to create custom evaluators and query locally-hosted models like Llama/Alpaca via Dalai.
Use Cases
- Testing which prompt variant performs best across GPT-4, Claude, and other LLMs to make data-driven model selection decisions.
- Evaluating a prompt's robustness to injection attacks before deploying an LLM-powered product feature.
- Verifying that an LLM consistently returns structured output (e.g., JSON or code-only) across dozens of varied inputs.
- Benchmarking the effect of different system message phrasings on model behavior and response quality.
- Exporting large parametrized prompt batches and their LLM responses to Excel for offline analysis or stakeholder reporting.
Pros
- Truly Open Source & Free: Fully open-source under an academic license, free to use on the web or self-hosted — no paywalls or usage limits.
- No-Code Evaluation Pipelines: The visual flow interface means non-engineers and researchers can run rigorous LLM evaluations without writing any code.
- Evidence-Based Insights: Replaces anecdotal prompt comparisons with real visualizations and exportable data, enabling data-driven decisions.
- Active Research Backing: Developed by Harvard HCI researchers with NSF funding, ensuring a principled, academically-grounded approach to LLM evaluation.
Cons
- Limited Web Feature Set: The hosted web version lacks advanced features like custom Python evaluators and local model querying — a full install is required for these.
- Browser Restriction: Only works on Chrome, Firefox, Edge, or Brave — Safari and other browsers are not supported.
- Early-Stage Beta: Still in open beta, meaning the API and UI may change between releases and some features may be unstable.
Frequently Asked Questions
Is ChainForge free to use?
Yes. ChainForge is fully open-source and free. You can use the web version at chainforge.ai/play with no account required, or install it locally via pip for free.
What LLMs does ChainForge support?
ChainForge supports major LLM APIs (including OpenAI's GPT models) and, in the local version, locally-hosted models like Alpaca and Llama via Dalai. Support continues to expand as the project evolves.
Do I need to write code to use ChainForge?
No. The core prompt engineering and evaluation flows are built visually without any code. The local version optionally allows Python code for custom evaluators.
How do I install ChainForge locally?
Run 'pip install chainforge' followed by 'chainforge serve', then open localhost:8000 in a supported browser. This unlocks the full feature set including custom evaluators and local model support.
What makes ChainForge different from just testing prompts manually?
ChainForge runs structured, repeatable evaluations at scale — comparing multiple prompts and models simultaneously, caching results, and producing visualizations and exportable data rather than one-off anecdotal observations.
