An AI’s reply is just as up-to-date because the documentation it makes use of

Earlier this 12 months, a service provider taking in-person funds requested our AI assist assistant easy methods to shortly settle for a fee with their card reader. It checked our documentation and confidently replied: “faucet ‘Easy Cost.’”

However there was no Easy Cost button. We had renamed it to “Add Customized Quantity” months earlier than, however the documentation hadn’t been up to date. The service provider couldn’t discover what the AI described and ultimately gave up.

Assist reported the problem, we up to date the short begin information and the product web page, and the documentation was fastened. However this incident caught with me as a result of the assistant did precisely what it was purported to do. It learn the documentation and shared what it discovered. The issue was that the documentation described a product that had already modified.

That is the principle level I need you to recollect from this: whenever you ask an AI easy methods to do one thing in WooCommerce — like altering a setting, discovering a function, or fixing an error — it solutions primarily based on the documentation, not on real-time information of the product. The product modifications with each launch, however the docs are written by individuals. Any hole between them can result in a flawed reply that sounds utterly assured.

So we created one thing to shut that hole.

It’s a WordPress plugin, constructed as an inside Automattic challenge, and it really works like a documentation author who’s at all times waiting for modifications.

It retains monitor of WooCommerce releases and any incoming requests to repair documentation. When one thing modifications, like a renamed button, a moved setting, or a brand new function, an AI mannequin drafts an replace. The draft then has to fulfill sure requirements earlier than it will get printed:

1. We rating it from 1 to five utilizing our high quality rubric, which appears to be like at accuracy, voice and tone, construction, grammar and elegance, findability, and visuals.
2. It additionally has to cross automated security checks. If a draft deletes greater than about 30% of a web page, breaks the web page’s formatting, leaves a placeholder, or provides a brand new picture hyperlink, it will get flagged for a human to assessment.
3. A author from our staff evaluations the draft in a preview and approves it earlier than it goes reside on woocommerce.com.

My favourite a part of the rubric is within the findability part: the primary paragraph of each doc should work as a standalone reply that an AI can quote. We’re not simply writing documentation for individuals scrolling by way of a web page anymore. We’re creating the supply materials for each AI reply about WooCommerce, and our rubric displays that.

We don’t simply rating new drafts. We additionally rating the reside documentation utilizing the identical rubric.

By late June, 39 out of the 117 WooCommerce docs we reviewed had been under our high quality requirements. That’s why we constructed this technique. WooCommerce updates on a regular basis, and even the most effective staff of writers can’t re-read each web page after every launch. However the plugin can. It checks pages as a author would with limitless time, and it flags those that want consideration.

One a part of the pipeline mechanically applies our terminology checklist to maintain product names constant, like utilizing “WooCommerce” as an alternative of “woocommerce.” It does this with a direct find-and-replace.

Some entries within the checklist had been written as writing recommendation as an alternative of actual phrase swaps. The system couldn’t inform the distinction, so it pasted the recommendation into reside pages. For a short while, a printed doc instructed readers “(descriptive hyperlink textual content describing the vacation spot)” the place a hyperlink ought to have been.

I observed this whereas engaged on one thing associated. We audited all 258 WooCommerce docs, discovered 9 that had been affected, fastened each, then eliminated the problematic entries and added a safeguard to stop this sort of mistake from occurring once more.

I’m sharing this as a result of it’s the identical lesson from a unique angle. Automation follows guidelines precisely as they’re written, shortly and with out judgment. That’s what makes it each useful and dangerous. Our system is constructed with this in thoughts: strict checks that fail safely, a human reviewing something essential, and audits for the automated steps.

What this implies whenever you’re troubleshooting

Should you ask an AI assistant a WooCommerce query and take the reply at face worth — and most of the people do, since that’s the purpose — the standard of the reply was set earlier than you even typed your query. It depends upon whether or not the documentation matches the product you’re utilizing.

Our job is to maintain the documentation correct, and the system above is how we try this. There’s a smaller half for you, nevertheless it’s nonetheless essential:

• If an AI’s directions don’t match what you see in your display, the documentation may be outdated. Verify the doc it’s utilizing earlier than considering you’ve made a mistake.
• Tell us when you spot a problem. When utilizing our AI assistant, charge a reply thumbs-down or faucet ‘Contact Assist’ to inform us what’s flawed. Should you’re on a doc that doesn’t appear present, inform our staff by way of the ‘Was this information useful?’ thumb on the backside of the web page.

The service provider with the cardboard reader didn’t do something flawed. Neither did the AI. The accountability was ours, and constructing a system that makes fixes shortly — and catches modifications earlier than you ever discover them — was one of the best ways we may study from that have.

David is an AI Engineer at Automattic who makes a speciality of transport manufacturing LLM infrastructure for buyer assist and documentation methods throughout WordPress.com and WooCommerce.