Why an RFP completeness score should fall

An RFP response that reports itself most of the way complete is telling you how much the software matched, not how much is actually answered. The number is worth trusting only if a person checking a row can make it fall. That requires confidence to be visible, and partial to be a real state.

A completeness score is the output of a matching process. Something compares the solicitation's questions against the firm's precedent and the group's documents, and reports how much of the checklist is answered. The single figure folds together three different things:

• how much of the checklist the software found anything for at all;
• how much of what it found actually answers the question;
• how much of each requirement is genuinely satisfied.

Only the first is a property of the software. The other two are properties of the response. A score that only moves upward is usually measuring the first while presenting itself as the third, and the three come apart exactly where an RFP is hardest: the questions the group has not answered before.

That is not a hypothetical failure mode. It is the normal behaviour of a matcher that has to return something. The system retrieves the nearest precedent passage and reports a hit, and the hit is real in the sense that the passage exists. It is not real in the sense that the issuer would accept it as an answer.

Because a score produced by matching inherits the failure modes of matching, and matching is done by a system that has to choose.

If the only way to move a row is to attach a document or insert a paragraph, the number is tracking activity rather than readiness. The response grows, the number rises, and nothing in that sequence requires anyone to have checked whether the new material answers the question it was filed against. That is a progress bar, not a measurement.

The test is whether a person can lower it. A reviewer opens a row the system matched with low confidence, reads the passage, and finds it describes a different kind of financing or a different issuer. The reviewer downgrades the row. Complete loses a line, partial gains one, and a score weighted by criteria falls. Nothing was deleted. The response did not change. The understanding of the response changed, and the number moved to reflect that.

A score with no path downward cannot represent the correction. It can only accumulate, which means it can only be right by accident.

A wrong match is worse than no match, because a wrong match hides work and a missing one advertises it.

A system that has to return something will return the nearest passage available. The nearest passage to a question about refunding savings can be a strong paragraph about a new-money financing: real, well written, and not an answer. Nothing about it is fabricated, and nothing about it addresses the question. What makes the mistake hard to catch is that the passage is genuine, not that it is unusual.

When a system will not assert a match, it should say so and hand the line to a person. That is not a gap in the product. It is the product declining to manufacture certainty it does not have. A vendor whose system never returns "no match" has not built a better matcher. It has removed the only signal that would have told you the matcher was unsure.

Because answers cover questions partially, and a binary state forces you to choose which error to make.

Take a line asking for three years of audited financial statements for the issuer and its blended component units. A response arrives with the issuer's statements and one component unit's, and the second component unit's are outstanding. Mark the line done and the gap disappears from every view built on the checklist. Mark it not done and the work already delivered disappears with it. Neither is true, and both lose something the review needs.

Partial has to be a real state. A partially answered line earns partial credit, which is what criteria-weighted completeness means: the score reflects how much of each requirement is met rather than how many rows have a tick against them. Once partial exists, the chase list built on top of it can tell a line nobody has touched from a line that is one document short. Those are different conversations with different people.

That distinction is the point at which a response tracker stops being a checklist and starts being a record of the work. It is also the behaviour RFP automation has to have before its number tells you anything you could act on.

Ask what the number is made of, how it moves, and whether a person can move it.

• Can a reviewer lower it? Not "can I edit the checklist", which every tool allows. Can a person read a matched row, disagree with it, and watch the score fall as a result? And can they show you where that happened, and who changed the status and when?
• What happens when nothing matches? Does the system return its closest guess regardless, or nothing at all plus a note to a person? A system that always returns a match is telling you what it could reach, not what it found.
• Is partial a state? Take a line from your own checklist that a single document only partly answers, and ask how it gets scored. If every line is done or not done, the score is counting rather than weighing.

None of that is an unusual thing to ask. NIST's AI Risk Management Framework already sets the expectation that accuracy measurements "should always be paired with clearly defined and realistic test sets ... and details about test methodology". A completeness score is an accuracy measurement with a friendlier name, and the test set is your own solicitation.

A vendor who cannot answer these is not necessarily selling something bad. They may never have been asked, and a number nobody has interrogated is not the same as a number that is wrong. From the outside you cannot tell those apart, which is the reason to ask before you are relying on the number rather than after.

Put the same questions to us, and to every tool you are weighing us against. They belong in every public finance RFP evaluation, aimed at every tool in it, including the one already in the room. A number nobody is willing to have lowered is the one thing worth walking away from.

• NIST's AI Risk Management Framework airc.nist.gov