How to Configure AI Engines and Run AI Testing Runs

Intro

BallersBook uses a smart chain of AI models from providers like Google Gemini, OpenAI, and Anthropic to generate drill diagrams and analyze them. As a super-admin, you can control exactly which models are active, in what order they are tried, and run structured tests to compare their output quality. This guide walks you through configuring the AI engine pool, executing a testing run on default or human-curated test corpuses, and verifying prompt behaviors.

Prerequisites

• A super-admin account (role with full system access)
• Familiarity with the concept of AI model fallback chains
• (Optional) A set of human test inputs or a desire to use the built-in default corpus
• The ai-admin-reviewer role assigned to your account (needed to view and manage testing features)

Step-by-step

1. Navigate to Global Preferences in your admin dashboard (or open the settings panel accessible only to super-admins).
2. Locate the AI Model Configuration section. This is where you manage the active engines for the two main pipelines:

• Drawing - Models used for generating new drill diagrams from text.
• Describing - Models used for analyzing existing diagrams and producing descriptions.

3. For each pipeline, review the list of available models. Each entry shows the provider (e.g., Gemini, OpenAI, Anthropic), model name, and its current status.
4. To enable or disable a model, toggle the Enabled checkbox next to it. Disabled models will be skipped during the fallback chain.
5. To change the priority (fallback order), enter a lower number in the Priority field for models you want tried first. Models with priority 1 are always consulted first; if they fail or return invalid JSON, the next model in numeric order is used.
6. Click Save Configuration to apply your changes. The new chain takes effect immediately for all future AI requests.
7. To test your configuration, navigate to the AI Testing panel (requires the ai-admin-reviewer role).

💡 Tip: If you do not see this panel, verify your active role with the Role Switcher in the navbar.

8. Click New Test Run. A dialog appears where you can choose:

• Test Corpus - Select Default (a fixed set of predefined prompts) or Human (custom prompts you provide).
• Models to Test - By default, all enabled models are tested. You can override this to test a subset.
• Pipeline - Choose Drawing, Describing, or both.

9. Click Start Run. The system executes each prompt against each selected model, recording outputs, errors, and timing. Progress is shown in real time.
10. Once the run completes, review the Results Table. It shows for each model:

• Number of successful responses
• Number of failures (invalid JSON, exceptions)
• Average response time
• (Drawing pipeline only) A thumbnail of the generated diagram

11. Click on any individual result to see the full model output versus the expected output from the corpus.
12. Use these insights to adjust the priority order or disable underperforming models. Then run a new test to verify improvements.

⚠️ Warning: Changes to the model chain take effect globally. Always run a test in a staging environment first if your club uses AI features in production.

Tips & Pitfalls

• Only models with supportsVision: true should be added to the Describing pipeline. If you add a text-only model there, it will always fail on image analysis tasks.
• The Priority field must be a positive integer. Avoid gaps in numbering; the chain is tried in strict numeric order.
• If you want to temporarily disable a model without removing it from the list, set its enabled flag to false in the configuration. It will remain available for future use.
• Default test corpuses are maintained by the development team and reflect typical coaching scenarios. You can create your own human corpus by uploading a CSV of prompts and expected outputs.
• After making configuration changes, wait a few seconds for Firestore to propagate before running a test - the Cloud Functions read the config from there.
• The ai-admin-reviewer role is distinct from ai-reviewer. Only ai-admin-reviewer can create and view testing runs. Regular reviewers can only see test results if shared.

Summary

You now know how to enable, order, and test the AI engines that power BallersBook’s drill generation and analysis. By managing the config/ai_models document via the Global Preferences UI and using the AI Testing panel, you can fine‑tune the fallback chain for reliability and quality. Next, consider running a human corpus test with actual coaching prompts from your club to validate real-world performance.