AutoBE
    Preparing search index...

    Interface AutoBeTestScenarioReviewEvent

    Event emitted when a single test scenario's review is completed.

    This event is triggered when the Test Scenario Review Agent completes analyzing one test scenario to validate authentication correctness, dependency completeness, execution order, and business logic coverage.

    The review process follows a three-tier priority system:

    PRIORITY 1: Validation Error Testing Detection (Can result in "erase"):

    • Detects scenarios testing HTTP 400 validation errors (invalid types, missing fields, invalid formats)
    • These scenarios violate absolute prohibitions from TEST_SCENARIO.md Section 2.1
    • AutoBE's three-tier compiler system (TypeScript + Typia + NestJS) already guarantees type safety
    • Testing framework guarantees wastes time and creates meaningless coverage
    • Such scenarios are marked for deletion via "erase" flag

    PRIORITY 2: Technical Correctness (Can result in improved scenario):

    • Missing authentication operations
    • Incomplete dependency chains
    • Wrong execution order (auth should precede business operations)
    • These are fixable issues that result in corrected scenarios

    PRIORITY 3: Quality Assessment (Can result in null):

    • Scenario passes all checks and needs no changes
    • Already tests business logic correctly with proper authentication and dependencies

    By extending multiple base interfaces, this event provides comprehensive tracking capabilities including progress monitoring for one-by-one scenario processing and token usage analytics for cost optimization.

    Michael

    Samchon

    interface AutoBeTestScenarioReviewEvent {
        completed: number;
        created_at: string & Format<"date-time">;
        endpoint: IEndpoint;
        id: string;
        improved: "erase" | AutoBeTestScenario | null;
        metric: AutoBeFunctionCallingMetric;
        original: AutoBeTestScenario;
        step: number;
        tokenUsage: IComponent;
        total: number;
        type: "testScenarioReview";
    }

    Hierarchy (View Summary)

    Index

    Properties

    completed created_at endpoint id improved metric original step tokenUsage total type

    Properties

    completed: number

    Number of items completed.

    Tracks how many items have been successfully processed so far in the current operation. This value increments as each item is completed, providing real-time progress indication.

    The ratio of completed to total gives the completion percentage: progress = (completed / total) * 100

    created_at: string & Format<"date-time">

    Timestamp when the event was created.

    ISO 8601 formatted date-time string indicating when this event was emitted by the system. This timestamp is crucial for event ordering, performance analysis, and debugging the agent workflow execution timeline.

    Format: "YYYY-MM-DDTHH:mm:ss.sssZ" (e.g., "2024-01-15T14:30:45.123Z")

    endpoint: IEndpoint

    The API endpoint being tested.

    Identifies the specific operation (method + path) that this test scenario targets. Used to correlate the review result with the corresponding API operation.

    id: string

    A unique identifier for the event.

    improved: "erase" | AutoBeTestScenario | null

    The review result: improved scenario, deletion flag, or null.

    Three possible outcomes reflecting the review priority system:

    1. "erase" - Scenario must be deleted (PRIORITY 1 violation):

    • Scenario tests HTTP 400 validation errors (invalid email, missing fields, wrong types)
    • Tests framework-level validations instead of business logic
    • Violates absolute prohibition from TEST_SCENARIO.md Section 2.1
    • AutoBE's compiler system already guarantees type safety - testing these guarantees is forbidden
    • Such scenarios are fundamentally wrong and must be completely removed
    • Examples: "test_api_user_registration_invalid_email", "test_api_article_creation_missing_title"

    2. AutoBeTestScenario - Scenario has been improved (PRIORITY 2 fixes):

    • Scenario tests business logic BUT had technical issues
    • Contains the improved version with corrections applied:
      • Corrected authentication operations
      • Complete dependency chains
      • Proper execution order (auth before business operations)
      • Refined test descriptions
    • Maintains the same endpoint and functionName as the original
    • Conceptually correct, just needed technical corrections

    3. null - No improvements needed (PRIORITY 3 - perfect scenario):

    • The original scenario was already correct with no issues found
    • Tests business logic correctly
    • Has proper authentication, complete dependencies, correct execution order
    • Ready for implementation as-is

    The review agent follows this decision tree:

    1. Does scenario test validation errors? → "erase"
    2. Does scenario have auth/dependency/order issues? → improved scenario
    3. Is scenario perfect? → null
    metric: AutoBeFunctionCallingMetric

    Function calling trial statistics for the operation.

    Records the complete trial history of function calling attempts, tracking total executions, successful completions, consent requests, validation failures, and invalid JSON responses. These metrics reveal the reliability and quality of AI agent autonomous operation with tool usage.

    Trial statistics are critical for identifying operations where agents struggle with tool interfaces, generate invalid outputs, or require multiple correction attempts through self-healing spiral loops. High failure rates indicate opportunities for system prompt optimization or tool interface improvements.

    original: AutoBeTestScenario

    The original test scenario before review.

    Contains the initial scenario as generated by the Test Scenario Agent, preserving the original for comparison and audit purposes.

    step: number

    Iteration number of the interface specification this review was performed for.

    Indicates which version of the API specification this review reflects. This step number ensures that the scenario review is aligned with the current interface structure.

    The step value enables proper synchronization between scenario review and the underlying interface definitions, ensuring that test scenarios remain valid as the API evolves through iterations.

    tokenUsage: IComponent

    Detailed token usage metrics for the operation.

    Contains comprehensive token consumption data including total usage, input token breakdown with cache hit rates, and output token categorization by generation type (reasoning, predictions). This component-level tracking enables precise cost analysis and identification of operations that benefit most from prompt caching or require optimization.

    Token usage directly translates to operational costs, making this metric essential for understanding the financial implications of different operation types and guiding resource allocation decisions.

    total: number

    Total number of items to process.

    Represents the complete count of operations, files, endpoints, or other entities that need to be processed in the current workflow step. This value is typically determined at the beginning of an operation and remains constant throughout the process.

    Used together with the completed field to calculate progress percentage and estimate time to completion.

    type: "testScenarioReview"

    Unique identifier for the event type.

    A literal string that discriminates between different event types in the AutoBE system. This field enables TypeScript's discriminated union feature, allowing type-safe event handling through switch statements or conditional checks.

    Examples: "analyzeWrite", "databaseSchema", "interfaceOperation", "testScenario"

    Settings

    Member Visibility

    On This Page

    Properties