In November 2025, I documented an idea: could AI automate the repetitive work involved in testing a web application?

Whenever a feature changed, someone had to open the application, enter values, click buttons, and check the results. Because a change could affect other features, previously tested workflows also needed to be checked again.

My question was:

If AI can understand an application’s interface and source code, could it identify what needs testing and carry out those tests?

The initial plan had two stages. First, build an automated quality verification system. Then, use the knowledge gathered during testing to connect AI with existing web applications.

QA seemed like a useful starting point. Testing requires understanding what a screen means, how it should be operated, what results to expect, and how to recover from changes. That knowledge could eventually support broader workflow automation.

As I moved from the idea to implementation, another question became more important:

When AI says, “I verified it,” what evidence should make me believe that claim?

2. Designing the Environment in Which AI Works

In this project, I use harness to mean the structure connecting AI’s inputs, tools, execution constraints, evaluation criteria, evidence, and review process.

Delegating analysis and implementation to AI did not remove the need to decide:

• What counts as a valid source for an expected result?

• Which actions are permitted or restricted?

• How are test preconditions established and verified?

• How are product defects distinguished from test or tool errors?

• What must be completed before a task can be reported as verified?

These decisions shaped the system. As failures appeared, I incorporated them into its rules and tools.

AI performed much of the analysis and implementation. I remained responsible for defining the problem, reviewing the evidence, and deciding which results could be accepted.

3. The Architecture: From Evidence to Execution and Back

The architecture connects a test execution pipeline with an evidence and review process. Product names and internal identifiers have been generalized.

flowchart TB
    subgraph INPUT["1. Inputs and Evidence"]
        S["Product Source · QA Documents<br/>Existing Test Specifications"]
        O["AI Browser Exploration<br/>Network and Data Observations"]
        K["Static and Runtime Knowledge Bases"]
        S --> K
        O --> K
    end

    subgraph PIPE["2. Execution Pipeline"]
        T["Test Case Specification<br/>Purpose · Preconditions · Expectations · Evidence"]
        G["Test Case Compiler<br/>Specification to Execution Plan"]
        B["KB Gate<br/>Targets · Consistency · Assertions · Restrictions"]
        P["Execution Preparation<br/>Accounts · Permissions · Test Data"]
        X["Browser Executor<br/>Actions · Observations · Evidence Collection"]
        V["Result Evaluation<br/>Pass · Fail · Error · Blocked"]
        T --> G --> B --> P --> X --> V
    end

    A["3. Application Under Test<br/>Isolated Business Web Application"]
    E["4. Evidence and Records<br/>Reports · Screenshots · Data Comparisons<br/>Defects · Reviews · Screen Manuals"]
    C["5. Operations Console and Human Review<br/>Select Scope · Review Results · Make Decisions"]
    H["6. Harness Verification<br/>pytest Contract Tests · Test Index"]

    S --> T
    O --> T
    K --> B
    X -->|"Perform actions"| A
    A -->|"UI, network, and data results"| X
    V --> E
    E --> C
    C -->|"Request execution"| P
    E -->|"Update verified observations"| K
    C -->|"Refine test cases"| T
    C -->|"Convert failures into rules and tests"| H
    H -.-> G
    H -.-> B
    H -.-> X
    H -.-> V

The central pipeline describes execution. The evidence records and review process determine what the execution means. Feedback connections allow verified findings to improve test specifications and the harness.

An AI-generated explanation does not automatically become a fact for the next run. Its supporting evidence must be checked before the finding is incorporated into the system.

4. How AI Builds an Understanding of a Screen

Finding buttons is different from understanding a business workflow.

A registration button raises several questions: what does it register, under what conditions, where is the result stored, and which other features depend on that data?

I structured AI’s analysis around three sources.

Browser exploration covered tabs, buttons, input fields, list columns, available options, and dialogs. Source analysis followed event handlers, validation functions, server requests, and data references.

Each tab was treated as a separate functional area. Tabs within the same screen could have different search conditions, required-field checks, and deletion confirmations. Nested dialogs were investigated independently rather than assumed to behave like their parents.

The analysis recorded both what had been inspected and what remained unverified. Conflicts between source code and runtime behavior were documented rather than silently resolved through inference.

The goal was to evaluate business understanding through traceable evidence, rather than the fluency of an explanation.

5. Turning Business Understanding into Test Cases

A test case defines an action under specific preconditions and the expected result to be checked.

Discovering a button does not produce a complete test case. Selection state, permissions, input conditions, and data types can all change its behavior.

Before generating executable tests, I used a verification ledger containing:

Observation and verification were kept separate.

“The delete button exists” is an observation. “The delete button must be unavailable when no item is selected” is a behavioral assertion.

Required Fields Need More Than One Warning

Leaving every field empty and checking for a warning does not establish that each required field is validated.

One validation could disappear while another still triggers the same warning, leaving the test green.

I therefore used the save-validation code to enumerate conditions and compared them with the required-field indicators on the screen. Each condition was tested by completing the other fields and leaving only the target field empty.

Positive controls also needed care. Completing all fields could cause a real save. Where persistence had to be verified, the test included controlled data creation and restoration.

Search Filters Need Inclusion and Exclusion Checks

An empty result does not prove that a filter works. The application may have had no matching data to begin with.

Tests therefore used both matching and non-matching records and checked that:

• Matching records remained.

• Non-matching records were excluded.

• Removing the filter restored the original results.

Test data values came from the values actually used by the application. Inferring a code from a column name could create an invalid precondition and make a working feature appear defective.

6. The KB Gate: Checking AI’s Claims Against Application Evidence

One distinctive part of the design is the knowledge-base gate, or KB gate.

It goes beyond validating the format of an execution plan. It checks whether referenced targets, declared conditions, and assertions are consistent with the application’s structure and available evidence.

For example, a button identifier may exist in the source code but belong to a dialog that the current screen never opens. An existence check would pass, while the execution target would still be wrong.

The gate addresses questions such as:

• Does the referenced element exist in the supporting evidence?

• Is the screen-to-dialog relationship correct?

• Do precondition declarations contradict one another?

• Can the assertions distinguish the execution conditions being tested?

• Is the tool checking a product result, or merely reading back a value it inserted itself?

• Could an ambiguous selector bypass an action restriction?

I also tracked how many references each rule actually inspected. “No violations found” and “the rule was never applied” are different outcomes.

The strength of the evidence affected rule severity. A menu snapshot collected under one account, for example, may be incomplete because of permissions. Absence from that snapshot is not sufficient proof that a menu does not exist.

The KB gate documentation records rule intent, scope, false positives, blind spots, and what a successful check does not establish.

The important design decision was to make each rule’s coverage and limits inspectable.

7. The pytest Index: Test Descriptions Also Need Verification

Another distinctive component is a generated index of pytest tests.

A large test suite becomes difficult to assess if it is unclear which contracts the tests enforce, whether those tests are actually collected, or whether their descriptions match their assertions.

The index is generated from code rather than maintained as a manual list.

It cross-checks test definitions found through Python’s abstract syntax tree with the items collected by pytest. A mismatch causes generation to fail.

It also distinguishes:

• Test functions from parameter-expanded execution items.

• Descriptions taken from a function’s docstring.

• Descriptions written separately after inspecting the test body.

Checks detect missing descriptions and stale entries left behind after a test is renamed or removed.

An important finding was that the descriptions themselves could be wrong. During the initial review, some descriptions claimed assertions the code did not make or reversed the direction of a check. Those descriptions were corrected.

The index therefore serves as a guide to the code, not a replacement for it. Description sources and review history are explicit, and later additions are not automatically presented as having received the same review.

Generation can help detect omissions and drift. It does not guarantee semantic accuracy. Keeping that distinction visible was part of the design.

8. Two Feedback Loops

I separated feedback into two loops: one improves the current test case; the other improves the environment that produces and executes test cases.

flowchart LR
    subgraph L1["Loop A: Product and Test Verification"]
        A1["Understand the Workflow"] --> A2["Generate Test Cases"]
        A2 --> A3["Execute and Collect Evidence"]
        A3 --> A4["Classify the Cause"]
        A4 -->|"Refine tests or preconditions"| A2
        A4 -->|"Record product defects"| A5["Maintain Reproduction Tests"]
    end

    subgraph L2["Loop B: Harness Improvement"]
        B1["Find Omissions, Misobservations, or Incorrect Reports"]
        B1 --> B2["Revise Criteria and SOP"]
        B2 --> B3["Add Gate Rules and Contract Tests"]
        B3 --> B4["Check Detection and Side Effects"]
        B4 --> B5["Apply to Subsequent Work"]
    end

    A4 -->|"Identify a recurring structural problem"| B1
    B5 --> A1

Loop A improves a particular test. Loop B prevents the same class of error from recurring across future work.

A product defect must not be normalized by changing the expected result to match the faulty behavior. If a test intentionally reproduces a defect, that purpose must be explicit.

Similarly, a gate’s false positive should not be resolved by weakening the test. The rule’s evidence and applicability need to be reviewed.

The objective of iteration is correct cause classification and better verification, not simply obtaining a passing result.

9. Turning Incorrect AI Reports into Structural Changes

There were cases where AI’s reports did not match what had actually been verified.

The records do not establish intentional deception. Nevertheless, unsupported claims such as “fully inspected,” “no writes,” or “verification complete” make the result unreliable.

I addressed these problems by changing the process.

One particularly instructive case involved opening and closing a dialog.

The exploration was assumed to be read-only, but the close handler sent a settings update. The network log captured the request, while the database comparison omitted the settings table.

The resulting screen behavior was then mistaken for a product defect. Repeated reproduction evidence accumulated, but the causal interpretation remained wrong.

The process was revised to inspect close handlers and verify that every observed write path was covered by the data comparison.

Repeated reproduction is not proof of a correct explanation.

10. The SOP Became a Contract for AI Work

These failures were incorporated into the standard operating procedure.

Its central principles were:

• Replace vague instructions with verifiable criteria.

• Require evidence for completion claims.

• Record why work remains incomplete and when it should be reconsidered.

“Checked carefully” was insufficient without scope and method. “No issues” required expectations and observations. “Verified” required evidence that the completion conditions had been met.

The SOP itself was also subject to revision. If following a procedure still produced a failure, the procedure had to be examined. Assuming that opening and closing a dialog was read-only was one such failure.

The SOP became a way to turn experience into constraints and verification criteria for the next run.

11. Completion Includes What Happens After Execution

A deletion message does not prove that data was deleted. A functional test can pass while leaving test data behind.

The system therefore checks application results, actual data state, and restoration. Where restoring a change is part of the test’s claim, restoration belongs in the evaluated assertions.

Defect-reproduction tests also need different presentation. A passing reproduction test may mean the defect still exists. It should not appear as an ordinary green success.

Completion consequently requires more than an execution result:

• Reviewed scope and remaining scope.

• Links between defects and reproduction tests.

• Evidence of data changes and restoration.

• Current review records and indexes.

• Workflow documentation and unresolved findings.

Not every condition is already enforced automatically. Some remain checklist and review requirements, with automation recorded as future work.

Making those limitations visible is part of reliable reporting.

12. How the Original Idea Developed

The November 2025 concept focused on reducing repetitive QA effort. It developed into workflow analysis, test generation, and browser execution. Reviews and failures in August 2026 led to stronger reporting and completion criteria.

My contribution was not limited to deciding what AI should build. It included defining the conditions under which its output could be accepted.

The KB gate checks execution plans against application evidence. pytest contract tests constrain tool behavior. The generated test index checks inventory and documentation consistency. The SOP preserves lessons from failures.

Connecting these components creates a continuous process:

Analysis → execution → evidence → judgment → rule improvement

AI carried out much of the analysis and implementation. I retained responsibility for problem definition, evidence selection, verification scope, action boundaries, and completion decisions.

The original idea was to reduce repetitive work with AI. The resulting system is a QA harness that verifies AI’s work and turns its failures into rules for subsequent execution.

Applying a single security remediation across roughly 100 Linux servers used to take about four hours.

The process involved connecting to each server, checking its configuration, backing up the original files, applying the change, and verifying the result. Each individual task was straightforward, but repeating it across many servers took time. Tracking completed work and recording results added to the effort.

I saw an opportunity to turn this repeated procedure into a workflow that an operator could initiate with a single action. The system would handle execution and record the results for each server.

I planned the tool and used AI as a development assistant, completing development in two days.

This article shares my personal experience from a workplace project. Proprietary implementation details, internal security policies, and actual remediation commands are omitted. It does not represent an official statement from my employer.

Turning Command Execution into an Operational Tool

A script that repeatedly executes commands can reduce manual work, but operators also need to manage targets, understand progress, identify failures, and review results.

I built the workflow around these needs:

• SSH execution: Connect to registered Linux servers, perform remediation, and record results separately for each target.

• Policy management: Store and manage remediation policies and their execution commands in a database.

• Distinct execution stages: Separate checks, backups, changes, and verification so failures can be traced to a specific stage.

• Web monitoring: Display server status and execution logs through a dashboard.

• History and permissions: Retain results for later review and distinguish management, execution, and viewing permissions.

I also included a stop request. It prevents subsequent work after the current command finishes; it does not undo changes already applied.

The aim was to make initiating the work simple while keeping its execution and results visible.

Using AI to Shorten Development

Before implementation, I defined the workflow and the features needed to support it. I then used AI to assist with implementation, reviewed the generated results, and revised behavior that did not match the requirements.

Keeping the scope focused helped. Rather than attempting to automate every security task, I concentrated on executing recurring remediation and making the results reviewable.

This approach allowed me to complete development in two days while keeping additional staffing needs low. AI helped accelerate implementation, while I focused on whether the tool matched the operational workflow.

Fast development still requires careful review. A tool that changes server configurations needs clear execution boundaries, permissions, failure handling, and verification.

From Four Hours to a Few Minutes

For the remediation task described here, applying one security change across roughly 100 servers went from about four hours to just a few minutes.

Instead of connecting to each server and entering commands repeatedly, the operator could initiate the workflow once and review the recorded results. Their attention shifted toward progress monitoring, failed tasks, and exceptions.

This timing reflects that particular task, rather than a guarantee for every remediation or environment.

The experience showed me how a clearly defined operational problem can become a practical tool in a short development cycle with AI assistance. For my next improvement, I would start by looking for another procedure people spend time repeating.