ACCURACY OVERVIEW

Accuracy assesses whether Generative AI is correctly answering the questions from consumers.  For each product or service, the LLMs are interrogated as to their knowledge of particular product/service features from the standpoint of the consumer. 

In this framework, LLM Accuracy for a travel brand transcends basic queries. It focuses on the model's ability to understand and consistently report on highly specific details that potential customers are asking. The methodology is designed to test "long-tail type questions" that go beyond general information.

 

Accuracy Module also is where you can find the hundreds of prompts that Bonafide uses to determine your accuracy score.

Examples of Specific Queries:

• For a hotel: Instead of asking if there is a pool, a more specific prompt would be, "Does your pool have a lifeguard specifically and what time does a pool open?"

• For a destination: Rather than asking if a festival exists in a certain month, a detailed query would be, "Do they have this festival in February and is it accessible to wheelchairs?"

The Scoring Methodology

The Role of LLM Consensus

The primary mechanism for scoring accuracy is the direct comparison of LLM answers to the system of record, not a consensus among the LLMs themselves. However, LLM consensus remains an important secondary metric, specifically to evaluate whether the models are aligned with each other when they are matching the System of Record (aka the source of truth).

Interpreting Blank Scores and Identifying Risks

 

The Anatomy of a Blank Score

A blank score, represented by two dashes (--), appears when two specific conditions are met simultaneously:

1. Absence in the System of Record: The crawler could not find an answer to the specific question anywhere on the brand's website or in its official content.

2. Lack of LLM Consensus: When the LLMs searched for an answer outside the system of record (on the broader web), there was no majority agreement among them on a single answer.

The 'Red Flag': Implications for Brands

A blank score is considered a "red flag" and a "threat" for a brand because it exposes a critical information gap that is being filled by unreliable sources.

• Unauthoritative Information: Real people are asking these questions, and since the brand does not provide the answer, LLMs pull information from external sources like Reddit or Trip Advisor.

• Conflicting Answers: Different LLMs provide varying answers to the same user query, leading to customer confusion and a lack of trust. This is described as a "complete misalignment" and can result in "hallucination" where the information provided is entirely incorrect.

• Brand Damage: This scenario represents a significant problem where user inquiries are being met with variable and unverified answers from sources outside the brand's control.

 

---

How to Navigate to View Details of the Prompts and Score:

Navigate: Accuracy → Property → Click on the Score

---

Examples in Accuracy Scoring

The following examples illustrate how the accuracy scoring is applied to specific prompts for both hotel and destination brands.

 

 

Example 1: High Accuracy (Hotel Accessibility)

 

LLM Performance Breakdown (5 Models Tested):

Conclusion: Four out of the five models tested successfully matched the verified answer from the system of record, resulting in a high accuracy score.

 

Example 2: Low Accuracy (Destination: Local Markets)

LLM Performance Breakdown (6 Models Tested):

Conclusion: Only one of the six models tested was able to replicate the correct list of markets found in the system of record, leading to a very low accuracy score.