Skip to content

25 questions to ask an AI vendor before your agency buys

A buyer's checklist for agencies evaluating AI software: logins, client data, accuracy, carriers, your AMS and price. Includes a scorecard.

Published · 10 min read

Most software demos for agencies look good. The vendor picks the client, the carrier and the line, and everything works. What you need to know is what happens with your clients, your carriers and your team on an ordinary Tuesday.

These 25 questions are the ones that separate a good demo from a tool your team will still be using in six months. They apply to any vendor, including us. Take them into every call.

In short: ask to see the work done on your own example, ask what the tool does when it is unsure, ask where your logins and client data live, and ask for the full cost in writing. Score each vendor on the same sheet so you compare answers, not presentations.

Download the scorecard to keep each vendor's answers side by side. It opens in Excel, Numbers or Google Sheets.

Before the first call: write down the task

A vendor can only be judged against a specific job. Before you talk to anyone, write three sentences:

  1. The task. "Quote a new homeowners client with five carriers" is a task. "Use AI to be more efficient" is not.
  2. What done looks like. For example: every carrier's premium and quote number, in one place, ready for a producer to compare.
  3. What it costs you today. Who does it, roughly how long it takes, and how many times a week. A rough count from one ordinary week is enough.

Bring one real example of that task to the demo. The first question on the list depends on it.

The work itself

1. Can you run our example, not yours? Ask to see the tool work a submission you brought, on a carrier you use. A vendor who can only show a prepared example is showing you a recording.

2. What exactly comes back? Ask to see the finished output and where it lands. A premium on a screen is different from a quote number, the carrier's own document and a note on what to check.

3. Which of our carriers and lines does it work with today? Give them your list. For each carrier, the honest answers are "yes, today", "not yet" and "we would need to set that up". Ask how that gets confirmed before you sign.

4. What still has to be done by hand? Every tool leaves something. You want to hear it from the vendor now, not find it in week three.

5. How long does one run take, start to finish? Ask for the hands-on minutes your team spends and the elapsed time until the result is usable. They are different numbers and both matter.

When it is unsure or wrong

6. What does it do when information is missing? The right answer is that it leaves the field blank or stops and asks. The wrong answer is that it fills in something reasonable.

7. How do we see what it did? You should be able to tell which values came from which document, and what the tool entered where. If you cannot trace a value to a source, you cannot review it.

8. What happens when a carrier's site changes? Carrier sites change often. Ask how the tool notices, what your team sees when it happens, and how quickly it is usually fixed.

9. What does a failure look like to my team? A clear "this carrier could not be completed, here is why" is workable. A wrong result that looks finished is not.

10. Who reviews the work before it reaches a client? The answer should be your licensed staff, with the tool making that review easy. Be cautious of any pitch that removes the review.

Logins and carrier access

11. Does it work under our own carrier logins? You hold the appointments. Work in a carrier system should happen under credentials the carrier issued to your agency.

12. Where are our credentials stored? Ask whether they are kept on your own computers or on the vendor's servers, who at the vendor can access them, and whether saving them is optional.

13. How does it handle MFA and CAPTCHAs? The usual sound answer is that it stops and hands that step to a person. Ask to see it happen.

14. Can each person use their own login? Shared logins make it hard to know who did what. Ask whether the tool supports the way your carriers expect access to work.

15. What do your terms say about carrier agreements? Your agency is responsible for following each carrier's terms of use. Ask the vendor what they know about this, and read your own agreements for the carriers you plan to use it with.

Client information

16. Is our clients' information used to train AI models? Ask for the place in the contract or documentation where this is written.

17. Where is client data stored, and for how long? You want a location, a retention period and a way to delete.

18. Who at your company can see it? Support staff sometimes need access to fix a problem. Ask when, how it is approved, and whether it is logged.

19. What happens to our data if we leave? Ask how you export it and how deletion is confirmed.

20. What should we tell our E&O carrier? A good vendor has been asked this before and can describe plainly what the tool does and does not do. Use their answer in your own AI use policy.

Your systems and your team

21. How does the result get into our AMS? "Integrates with" can mean many things. Ask to see the exact steps for your management system, including what is manual.

22. What does setup involve, and who does it? Ask how many hours your team spends, over how many days, and what the vendor does for you.

23. How long until a new person can use it alone? The best evidence is a story about someone specific. Ask how training works after the kickoff, when you hire your next person.

Price and commitment

24. What is the full cost for an agency our size? Ask for everything in writing: the subscription, setup, per-user or per-quote charges, and any separate AI or usage bill. A price that depends on volume is fine as long as you can estimate your volume.

25. What are we committing to? Ask about contract length, how you cancel, and what happens if the tool does not work with a carrier you were counting on.

How to score the answers

Use the same sheet for every vendor. For each question, write the answer in the vendor's own words, then mark it:

MarkMeaning
ShownYou saw it work on your example, or saw it in writing
SaidThe vendor told you, but you have not seen it
UnclearThe answer was vague, or changed the subject
NoThe tool does not do this

Count the "Shown" marks. A vendor with fewer features and more "Shown" is usually the safer choice than one with a long list of "Said".

Then go back to your three sentences from before the call. Does the tool do that task, to that finish line, for less than it costs you today once you include review time? That is the decision.

Five answers that should slow you down

  • "It handles everything automatically." Something always needs a person. You want to know what.
  • "We work with every carrier, no setup needed." Ask about yours, by name, and ask what setup each one takes.
  • "The AI is very accurate." Accuracy claims without a description of what happens on a miss do not help you.
  • "You won't need to review it." You hold the license.
  • "We can't share pricing until later in the process." A range, at least, should be possible on the first call.

Common questions

How many vendors should we evaluate?

Two or three is enough for most agencies. More than that and the calls blur together. The scorecard matters more than the count.

Should we run a trial?

If a vendor offers a way to run your own work before committing, take it, and decide in advance what result would make you say yes. If they do not, ask to run one real submission together on a call. Either way, judge a real task rather than a demo.

Who from the agency should be on the call?

The person who does the task today and the person who signs. The first one knows where the work actually gets stuck. The second one hears the price.

How does Relay answer these?

We will answer all 25 on a call. The short version is on our how it works, security and control and pricing pages. Where the answer is "not yet" or "your team still does that", we will say so.

What to do next

Write your three sentences, download the scorecard, and book your calls. If you are still deciding which task to start with, how agencies use AI agents walks through the common first choices.

From Relay, by email

Guides your agency can use, with or without Relay.

Templates and checklists for independent insurance teams. Read these now, and leave your work email to hear when we publish a new one.

We’ll email you when we publish a new guide. Unsubscribe anytime. Privacy

See it quote your carriers.

Thirty minutes. Bring the carriers you quote most and one submission that took too long.

Built for the independent agency.

Piper, the Relay fox, standing with a red cape