Auto-Test Generation

CodePeel can automatically generate unit tests for your PR changes and open a separate pull request containing those tests. The test generation uses full source file context, detects your language and testing framework, and validates the output through a second pass before committing. Auto-test is a Pro and Max feature that runs after the standard review completes.

---

Overview

When auto-test is enabled, CodePeel analyzes the diff from your pull request, extracts the full source code of changed files, detects the appropriate testing framework, and generates comprehensive unit tests. The generated tests are committed to a new branch and a pull request is opened targeting your feature branch, so the tests can be merged alongside your code.

The test generation pipeline is designed to produce tests that actually compile and run. It uses a two-pass approach: the first pass generates tests with full source context, and the second pass validates and fixes compilation issues. Tests that fail validation (missing assertions, placeholder markers, or insufficient content) are rejected before the PR is created.

Auto-test runs as a background operation. If test generation fails for any reason, it does not affect the main review. Your PR still receives its normal review findings regardless of whether tests were successfully generated.

---

How to Enable

Auto-test is disabled by default and must be explicitly enabled. It requires a Pro or Max plan.

Option 1: Via .codepeel.yml

Add to your repository's configuration file:

auto_test:
  enabled: true

Option 2: Via the Dashboard

1. Go to Settings in the sidebar
2. Navigate to the Automation section
3. Toggle Auto-Generate Tests on

The .codepeel.yml setting takes priority if both are configured. See the Configuration documentation for details on how settings merge.

---

How It Works

The auto-test pipeline runs immediately after the main review analysis completes, in parallel with comment posting and auto-fix generation. This minimizes the total time before the test PR appears.

Step-by-step flow

1. Trigger check. After the review completes, the system checks if auto_test.enabled is true in the config AND the user is on a Pro or Max plan. If either condition is false, test generation is skipped.

2. Diff processing. The PR diff is truncated to 35,000 characters if it exceeds that limit. This keeps test generation focused on the most relevant content.

3. Source extraction. Full source file contents are extracted from the diff. For new files, the complete content is available. For modified files, only the added lines are used. Files with fewer than 5 added lines are skipped (too little context to generate meaningful tests).

4. Framework detection. The system analyzes the diff content and repository name to detect the programming language and testing framework. Detection is based on file extensions, import statements, and framework-specific patterns.

5. Test generation (first pass). CodePeel constructs the test generation input with the full source code, detected framework, and test placement conventions. Tests are returned with file paths as keys and test code as values.

6. Validation and fixing (second pass). Each generated test file is reviewed alongside its corresponding source file. Compilation errors, incorrect mock patterns, and private field access issues are fixed.

7. Output validation. Generated tests are validated against language-specific criteria: presence of test blocks, assertions, minimum length, and absence of placeholder markers. Invalid tests are rejected.

8. Quota check. Before creating the PR, the system verifies the user has not exceeded their monthly quota (Pro users are capped at 500 reviews). If at the limit, test generation is skipped.

9. Branch and PR creation. Valid tests are committed to a new branch (codepeel/tests-pr-{number}-{timestamp}) and a pull request is opened targeting the original PR's feature branch.

10. Notification. A comment is posted on the original PR linking to the test PR.

Credit consumption

Auto-test generation consumes 1 review from your monthly quota. This deduction happens only if valid tests are generated and a PR is created. If generation fails or produces no valid tests, no review is consumed.

---

Supported Languages and Frameworks

The test generation system automatically detects your language and testing framework from the diff content. No manual configuration is required.

Detection logic

The framework is detected by scanning the diff for language-specific indicators:

• Dart/Flutter: Presence of package:flutter, .dart files, or pubspec.yaml
• Python (pytest): Presence of pytest or conftest imports
• Python (unittest): Presence of .py files without pytest indicators
• Go: Presence of .go files, package declarations, or func keywords
• Rust: Presence of .rs files, fn keywords, or use statements
• TypeScript: Presence of .tsx or .ts files, React imports, or ES module imports
• JavaScript: Presence of .jsx or .js files

If no language can be detected, the system defaults to TypeScript with Jest/Vitest.

Test file placement

Generated test files follow language-specific conventions:

---

What Gets Generated

The AI generates tests based on the full source code of changed files, not just the diff. This gives it complete context about constructors, method signatures, dependencies, and class structure.

Test coverage targets

What the AI considers

• Full source file content (not just changed lines)
• Constructor parameters and dependency injection patterns
• Public vs private method visibility
• External dependencies that need mocking
• Return types and expected outputs
• Error conditions and edge cases

Language-specific handling

Dart/Flutter:

• Uses @GenerateMocks annotation or manual mock classes extending Mock
• Respects private field conventions (prefixed with _)
• Uses constructor injection for dependencies
• Handles CollectionReference generic types correctly

TypeScript/JavaScript:

• Uses vi.fn() for Vitest or jest.fn() for Jest
• Applies type assertions for mocks
• Handles async/await patterns

Python:

• Uses unittest.mock.patch for mocking
• Follows test_ prefix convention for test functions
• Matches actual module import paths

---

Generated PR Format

When tests are successfully generated, a new pull request is created with the following structure:

Branch naming

codepeel/tests-pr-{pullNumber}-{timestamp}

For example: codepeel/tests-pr-42-1717200000000

PR title

🤖 [CodePeel] Auto-Generated Tests for PR #42

PR body

The PR description includes:

• A reference to the original PR number
• The count of test files created
• A note that tests may need minor adjustments

PR target

The test PR targets your feature branch, not main. This means the tests merge into your feature branch and ship together with your code changes.

Commit message

test(codepeel): autogenerate 3 test suites

Co-authored-by: CodePeel <bot@codepeel.com>

Notification comment

A comment is posted on your original PR:

**Auto-Generated Tests Available**

CodePeel has generated tests for this PR: https://github.com/owner/repo/pull/43

---

Validation Pipeline

Generated tests go through multiple validation steps before being committed. This ensures the PR contains only tests that are likely to compile and provide value.

First pass: Generation

CodePeel generates tests with full source context. The input includes:

• Complete source file contents (capped at 8,000 characters per file)
• Language and framework information
• Test file placement conventions
• Critical rules about mocking, private field access, and assertions

Second pass: Validation and fixing

Each generated test file is reviewed with its corresponding source file. Common issues are fixed:

• Private field access from outside the class
• Incorrect mock patterns
• Missing imports
• Type errors
• Framework-specific issues (e.g., late keyword in Dart, generic types)

If the fixed version is shorter than 100 characters or fewer than 5 lines, the original is used instead (the fix was likely garbage).

Output validation

After both passes, each test file is validated against language-specific criteria:

Tests that fail validation are rejected and not included in the PR. The rejection reason is logged for debugging.

---

Configuration

Minimal configuration

auto_test:
  enabled: true

This is the only configuration needed. The system automatically detects your language, framework, and test conventions.

Combined with auto-fix

Auto-test and auto-fix can both be enabled simultaneously. They run in parallel and create separate PRs:

auto_test:
  enabled: true
auto_fix:
  enabled: true

Disabling for specific repositories

If auto-test is enabled in your dashboard settings but you want to disable it for a specific repository, add to that repo's .codepeel.yml:

auto_test:
  enabled: false

The .codepeel.yml setting overrides the dashboard setting.

---

Plan Requirements

Auto-test generation requires a Pro or Max plan. On the Free tier, the auto_test.enabled setting is ignored and no tests are generated.

If a Pro user has exhausted their monthly quota (500 reviews), auto-test generation is skipped for that PR. The main review still runs (it was already deducted), but the test generation is not attempted.

---

Limitations

Diff size cap

The diff is truncated to 35,000 characters before processing. For very large PRs, some files may not have tests generated because their content was cut off. Consider splitting large PRs into smaller ones for better test coverage.

Source file extraction

Tests are only generated for files where at least 5 lines were added in the diff. Files with minor changes (1-4 added lines) are skipped because there is insufficient context to generate meaningful tests.

Source file size cap

Each extracted source file is capped at 8,000 characters. Very large files may have their content truncated, which can affect test quality for code at the end of the file.

No existing test awareness

The current implementation does not read existing test files in your repository. Generated tests may duplicate coverage that already exists. Always review the generated PR before merging.

Framework detection accuracy

Framework detection is based on heuristics (file extensions, import patterns). In rare cases, it may detect the wrong framework. The generated tests will still be syntactically valid for the detected framework but may not match your project's actual setup.

Background operation

Test generation runs asynchronously and does not block the main review. If it fails, no notification is posted. The main review is unaffected.

---

Example

Input: New TypeScript function

Your PR adds a calculateDiscount function:

export function calculateDiscount(
  price: number,
  tier: 'basic' | 'pro' | 'enterprise'
): number {
  const rates = { basic: 0.05, pro: 0.15, enterprise: 0.25 };
  return price * (rates[tier] || 0);
}

Output: Generated test file

CodePeel generates src/lib/__tests__/pricing.test.ts:

import { calculateDiscount } from '../pricing';

describe('calculateDiscount', () => {
  it('applies basic tier discount (5%)', () => {
    expect(calculateDiscount(100, 'basic')).toBe(5);
  });

  it('applies pro tier discount (15%)', () => {
    expect(calculateDiscount(100, 'pro')).toBe(15);
  });

  it('applies enterprise tier discount (25%)', () => {
    expect(calculateDiscount(200, 'enterprise')).toBe(50);
  });

  it('returns 0 for zero price', () => {
    expect(calculateDiscount(0, 'pro')).toBe(0);
  });

  it('handles negative prices', () => {
    expect(calculateDiscount(-100, 'basic')).toBe(-5);
  });
});

Input: New Dart service class

Your PR adds a UserRepository class:

class UserRepository {
  final FirebaseFirestore _firestore;

  UserRepository(this._firestore);

  Future<User?> getById(String id) async {
    final doc = await _firestore.collection('users').doc(id).get();
    if (!doc.exists) return null;
    return User.fromMap(doc.data()!);
  }
}

Output: Generated Dart test

CodePeel generates test/repositories/user_repository_test.dart:

import 'package:flutter_test/flutter_test.dart';
import 'package:mockito/mockito.dart';
import 'package:mockito/annotations.dart';
import 'package:cloud_firestore/cloud_firestore.dart';

@GenerateMocks([FirebaseFirestore, CollectionReference, DocumentReference, DocumentSnapshot])
import 'user_repository_test.mocks.dart';

void main() {
  late UserRepository repository;
  late MockFirebaseFirestore mockFirestore;
  late MockCollectionReference<Map<String, dynamic>> mockCollection;
  late MockDocumentReference<Map<String, dynamic>> mockDoc;

  setUp(() {
    mockFirestore = MockFirebaseFirestore();
    mockCollection = MockCollectionReference();
    mockDoc = MockDocumentReference();
    repository = UserRepository(mockFirestore);

    when(mockFirestore.collection('users')).thenReturn(mockCollection);
  });

  group('getById', () {
    test('returns user when document exists', () async {
      final mockSnapshot = MockDocumentSnapshot();
      when(mockCollection.doc('user-1')).thenReturn(mockDoc);
      when(mockDoc.get()).thenAnswer((_) async => mockSnapshot);
      when(mockSnapshot.exists).thenReturn(true);
      when(mockSnapshot.data()).thenReturn({'name': 'Test User'});

      final result = await repository.getById('user-1');
      expect(result, isNotNull);
    });

    test('returns null when document does not exist', () async {
      final mockSnapshot = MockDocumentSnapshot();
      when(mockCollection.doc('missing')).thenReturn(mockDoc);
      when(mockDoc.get()).thenAnswer((_) async => mockSnapshot);
      when(mockSnapshot.exists).thenReturn(false);

      final result = await repository.getById('missing');
      expect(result, isNull);
    });
  });
}

---

Troubleshooting

Test PR not appearing

If auto-test is enabled but no test PR is created:

1. Verify you are on a Pro or Max plan (auto-test is not available on Free)
2. Check that auto_test.enabled is true in your config or dashboard
3. Ensure you have not exhausted your monthly review quota
4. The diff may have been too small (fewer than 5 added lines in any file)
5. All generated tests may have failed validation (check your review history for the failure reason)

Generated tests do not compile

The validation pipeline catches most compilation issues, but some may slip through:

• Review the generated tests and fix any remaining issues before merging
• The tests are suggestions, not guaranteed to be production-ready
• Common issues: incorrect import paths, missing type definitions, framework version mismatches

Tests duplicate existing coverage

The current implementation does not read your existing test files. If you already have tests for the modified code, the generated tests may overlap. Review the test PR and remove duplicates before merging.

Wrong testing framework detected

If the system detects the wrong framework:

• The generated tests will use the wrong syntax (e.g., Jest instead of Vitest)
• Review and adjust the imports and assertion style
• Consider adding framework-specific files (like vitest.config.ts) to help future detection

Auto-test consuming reviews unexpectedly

Auto-test consumes 1 review from your quota each time it successfully generates tests. If you are on Pro with limited reviews, consider disabling auto-test when approaching your quota limit. You can check your usage with the check_credits MCP tool or on the billing page.

---

Frequently Asked Questions

Does auto-test run on every PR?

Yes, when enabled. Auto-test runs after every PR review that completes successfully, as long as the user is on a Pro or Max plan and has available quota. There is no way to trigger it selectively per PR -- it is either enabled or disabled globally.

Can I customize the testing framework used?

Not directly through configuration. The framework is auto-detected from the diff content. If detection is incorrect, the generated tests will use the wrong framework syntax. You can manually adjust the generated PR before merging.

Do generated tests run in CI?

That depends on your CI configuration. The test PR is a normal pull request on a branch. If your CI runs tests on all PRs, the generated tests will be executed. If they fail in CI, you will see the failure on the test PR and can fix them before merging.

What happens if test generation fails?

Nothing visible to the user. Test generation runs in the background. If it fails, no comment is posted and no PR is created. The main review is completely unaffected.

Can I use auto-test without auto-fix?

Yes. Auto-test and auto-fix are independent features. You can enable either one, both, or neither. They run in parallel and create separate PRs when both are enabled.

How many test files are generated per PR?

The number depends on how many source files were changed in the PR and how many pass validation. Typically, one test file is generated per changed source file that has at least 5 added lines. Very large PRs may have some files skipped due to the 35,000 character diff cap.

---

Related Documentation

• Auto-Fix -- Automatic fix PR generation (companion feature)
• Configuration -- .codepeel.yml reference including auto_test settings
• Billing -- Plan requirements and review consumption