What we found checking AI credit tools' own demos

First, the good news. Most of the arithmetic tied out, often to the dollar. Ratios matched their inputs, subtotals footed, and the stated breaches were real breaches.

What did not always hold up was the reading around the arithmetic: which base a cap is measured on, which version of the agreement applies, which way an adjustment runs. Those are the choices that decide whether a borrower passes, and a demo built on the vendor's own example rarely shows how a tool makes them.

We have left out vendor names and changed every figure. The examples below use invented numbers that reproduce each pattern. None of them are a vendor's own figures.

1. The base of a percentage cap goes unstated, and it decides the answer

Concentration limits and add-back caps are percentages of something, and agreements differ on what: gross receivables or receivables after the eligibility tests, EBITDA before add-backs or after. More than one example we checked only reconciled on one reading, and the screen did not say which.

Worth asking: does the tool show the base it used, and can it be set to match each agreement?

2. Striking an add-back should move every test that uses EBITDA

A common demo removes an add-back the agreement does not permit and recomputes leverage. The question is whether fixed charge coverage moves with it. In one example we saw, a single coverage figure sat under a header that showed two different EBITDA figures, with no way to tell which one it used.

Worth asking: when one input changes, does the tool show every test it feeds?

3. The sign of an adjustment line

Debt service coverage often runs through adjustment lines. In one demo the ratios only reconciled if a line shown as a positive number was subtracted. The tool may well have computed it correctly, but a reviewer checking by hand could not tell.

Worth asking: are adjustments displayed with the sign they are applied with?

4. Headroom figures that do not match the ratios next to them

Headroom is quoted in more than one convention and rounded in different places. 4.40x against a 5.00x maximum is 12.0% headroom measured on the limit, which is also how far EBITDA can fall before a breach. Measured as how far debt can rise, it is 13.6%. We also saw cushions rounded to whole percentages: 3.30x against 3.50x is 5.7% headroom, which a summary card might show as 5%.

Rounding down on a maximum is the cautious direction, and none of the cases we found changed a pass or a fail. But a summary card and a covenant table that disagree by a few tenths of a percent is what makes a reviewer stop trusting the screen.

Worth asking: one convention, stated on the screen, to one decimal place.

5. Which version of the agreement applies

Amendments, step-downs and refinancings change the test. The best example we saw made exactly this point: correcting the add-backs made the ratio worse, but it was the date of the test that decided the breach.

Worth asking: does the tool test against the agreement as amended on the test date, and does it say which version it used?

6. Accuracy claims without a denominator

A 98% accuracy figure. Zero breaches missed. Claims like these appeared without a method: how many cases, how many contained a problem, and who decided the right answer. In one case a percentage on a homepage did not match the vendor's own case study. Zero misses means little without the number of breaches there were to find, and the number of false alarms raised along the way.

A related habit is validating a tool against a fund's own past monitoring work. That shows the tool agrees with the analysts. It does not show either of them was right, and past files contain errors too.

Worth asking: what was the test set, how many cases had a real problem, and how many had none?

What this means if you are choosing a tool

None of these are exotic. They are the ordinary judgment calls in every credit agreement. A demo built by the vendor on its own example cannot show how a tool handles them. A fixed test with a known answer key can.

Our suite has 40 synthetic private credit tasks covering covenant tests, add-back caps, borrowing bases and NAV facilities. Fifteen contain no problem at all, so a tool that flags everything is marked down too. Every answer key is computed in code and recomputed by a second, separately written checker. Each response is scored on whether the tool spotted the problem, got the number, escalated when it should and cited the governing clause, and every task runs three times.

Our tasks are synthetic and cleaner than real document sets, with no scans or OCR errors. They measure reasoning and calculation on stated terms, not document parsing.