Magnitude - AI-Powered Web Application Testing Framework

Project Overview

Magnitude is an open-source, AI-native testing framework designed specifically for web applications. It's powered by a visual AI agent that can see your interface and adapt to any changes within it.

Core Features

🎯 Key Features

• ✍️ Natural Language Test Case Construction - Easily build test cases using natural language.
• 🧠 Powerful Reasoning Agent - Intelligently plans and adjusts the testing process.
• 👁️ Fast Visual Agent - Reliably executes test runs.
• 📄 Test Plan Saving - Save plans to execute runs in the same way.
• 🛠 Intelligent Problem Handling - The reasoning agent intervenes to handle issues when encountered.
• 🏃♂️ Flexible Deployment - Run tests locally or in CI/CD pipelines.

Quick Start

1. Install the Test Runner

npm install --save-dev magnitude-test

2. Initialize the Project

npx magnitude init

This will create a basic test directory tests/magnitude containing:

• magnitude.config.ts - Magnitude test configuration file
• example.mag.ts - Example test file

3. Configure LLM Clients

Magnitude requires setting up two LLM clients:

Planner

• Recommended Model: Gemini 2.5 Pro
• Configuration:
  • Use Gemini through Google AI Studio or Vertex AI.
  • Create an API key in Google AI Studio and export it as GOOGLE_API_KEY.
  • Alternative providers supported: ANTHROPIC_API_KEY / OPENAI_API_KEY

Executor

• Model Used: Moondream
• Configuration:
  • Register with Moondream and create an API key.
  • Set the environment variable MOONDREAM_API_KEY.
  • Offers 5,000 free requests daily (approximately hundreds of test cases).
  • Fully open-source and supports self-hosting.

Running Tests

Basic Run Command

npx magnitude

This will run all Magnitude test files discovered using the *.mag.ts pattern.

Parallel Testing

npx magnitude -w <workers>

Test Case Examples

Basic Syntax

import { test } from 'magnitude-test';

test('can add and complete todos', { url: 'https://magnitodo.com' })
  .step('create 3 todos')
  .data('Take out the trash, Buy groceries, Build more test cases with Magnitude')
  .check('should see all 3 todos')
  .step('mark each todo complete')
  .check('says 0 items left')

Complex Scenario Example

import { test } from 'magnitude-test';

test('can log in and create company')
  .step('Log in to the app')
  .data({ username: '[email protected]', password: 'test' })
  .check('Can see dashboard')
  .step('Create a new company')
  .data('Make up the first 2 values and use defaults for the rest')
  .check('Company added successfully');

Core Concepts

Natural Language Description

• Steps: Describe the actions to be performed.
• Checks: Natural language assertions.
• Data: Test data, supports arbitrary key-value pairs.

Design Philosophy

Writing test cases is like describing to a colleague how to test a specific process:

• What steps they need to take.
• What they should check.
• What test data to use.

CI/CD Integration

Magnitude tests can be run in any CI environment capable of running Playwright tests, simply by including the LLM client credentials.

GitHub Actions Support

The project provides detailed instructions for running test cases on GitHub Actions.

Technical Architecture

Dual-Model Design

Magnitude uses a separated planning/execution model architecture:

• Planning Model: Formulates effective testing strategies.
• Execution Model: Executes tests quickly and reliably.

Differentiation from Other Solutions

Compared to OpenAI or Anthropic's Computer Use APIs:

• Faster Speed: Specifically optimized for test cases.
• Higher Reliability: Specially designed agent architecture.
• Lower Cost: Cost-effective optimization for testing scenarios.
• Native Test Runner: Purpose-built test design and execution tool.

Summary

Magnitude represents a significant innovation in the field of test automation. By combining the strengths of AI technology and traditional testing frameworks, it provides a powerful, flexible, and easy-to-use testing solution for web applications. The combination of natural language test descriptions and visual AI agents makes writing and maintaining test cases easier than ever before.