Ask to use it, not just watch it
A recorded demo can be rehearsed. A live build can't. Ask to drive the thing yourself, or at minimum to see it run on an input you pick in the moment, not one from a script.
If every example you're shown is a polished video and nobody will let you touch anything, that's the answer to a different question than the one you asked.
Ask where your data actually goes
Does the system run on your own accounts and infrastructure, or does it route through someone else's server? Is your data used to train a public model, or is it excluded by contract or provider setting? Who can see it, and can you revoke their access without their involvement?
A vendor who can answer this precisely, without hedging, has built systems where it mattered. One who waves at 'enterprise-grade security' without specifics usually hasn't had to answer the question for real.
Ask who's still there after launch
Automations don't fail on day one — they fail in week six, when an upstream API changes a field name or a volume spike exposes an edge case nobody tested. Ask who owns that: is it included, billed hourly, or silently nobody's job once the invoice is paid?
Get this in writing before you start, not after something breaks.
Ask for evidence, not just a portfolio page
Case studies are easy to write and hard to verify. Ask for something closer to proof: a reference you can actually contact, a specific measured result (not a round number), or a public artifact — a live product, an open-source repo, a client review on a platform you don't control.
The gap between 'we delivered X for a client' and 'here's the client, here's the number, here's where you can check it' is the whole signal.
Match the hire to the job, not the other way around
A freelancer is fast to start and you talk directly to whoever builds it, but one person is a capacity ceiling. An agency has bench depth and project management, at the cost of layers between you and the actual work. In-house makes sense once there's a steady stream of this kind of work, not for one project. None of these is universally right — the honest version of this question depends on your volume, your timeline, and how much you need to talk to the person actually writing the workflow.
Related