A couple of days ago I wrote about the four filters I use before paying for a tool. That post answers one question: should I buy this. It does not answer the one that comes right after it, which is the one that has actually cost me money — the tool is in front of me now, so how do I find out whether it works before it is holding up something I care about?
I run five things at once and I have broken every one of them at least once by trusting a tool too early. What follows is the procedure I use now. It takes two days.
The test project is never the real project
This is the whole thing. Everything after it is detail.
When I evaluated Expo and EAS Build for the Play Store app, I did not start with the app. I built a throwaway project that did nothing except show one screen and get itself signed and uploaded. It had no features to protect, so I could be honest about the build pipeline. I found the signing and review-queue problems in a sandbox where they cost me a weekend instead of a launch date.
The moment you test a tool inside real work, you stop evaluating it. You start defending the hours you already put in. That is not a test, that is a sunk cost wearing a test’s clothes.
Day one: build the smallest thing that can fail
My scope rule is one round trip. Data goes in, the tool does its job, something comes out the other end that I can look at. For a scraping tool that is one scrape of one page into one CSV. For a broker API it is pulling a single instrument’s price and printing it. For a print-on-demand backend it is one product and one test order, all the way to what the customer actually receives.
Anything bigger and you are not testing the tool, you are building with it. One round trip is enough to surface the two things that matter: where the documentation is lying, and where the tool needs something you do not have yet — a card, a registration, a domain, a verification period, a device you do not own.
Day two: break it on purpose
Day one only tells you the tool works when everything is correct. Nobody ships in that condition. Day two is where I feed it bad input, kill the network mid-run, push past the free-tier limit, and read what the error message says.
The error message is the real product. A tool that fails with a clear reason is a tool you can debug at 11pm three months from now. A tool that fails silently, or returns an empty success, will eventually cost you a full day — and it will be a day you did not schedule.
The same day I check what happens when I want to leave. Can I export? In what format? If the answer is a screenshot or a copy-paste, I already know how this ends.
Four things I write down, in the same file every time
Time to first working output, measured rather than estimated. This is the number that tells me what onboarding a second person would cost, and it is always larger than the landing page implies.
The thing that surprised me. There is always one. It is usually a limit that is not on the pricing page.
What it replaces, named specifically. Not ‘saves time’ — which existing step disappears. If nothing disappears, the tool is an addition, and additions have to earn far more than replacements do.
The exit cost, in hours. What it takes to get my data out and go somewhere else.
Six months later I remember none of this. The file does. A good part of what I now think of as judgement is just old notes.
The procedure part, not the tool part
Tools are the easy half. The process around a new project is what I get wrong more often, so I now write one page before anything starts: the single question this project answers, who it answers it for, what has to be true for it to be worth continuing after thirty days, and the date I will check.
The check date is the part everyone skips, myself included. Without it a project never fails — it just goes quiet. And quiet projects keep taking your attention forever.
What I Test Before I Trust a Tool
The two-day test is deliberately small, but it is not casual.
I am trying to answer a few specific questions before the tool gets anywhere near a client project, production website, customer data, or money.
Does it work?
Does it work consistently?
Can I understand what went wrong when it doesn’t?
Can I get my data back?
And, most importantly, does it make the workflow better enough to justify another dependency?
That last question matters because every new tool creates something I have to maintain.
A subscription has to be paid.
An integration has to be maintained.
Credentials have to be managed.
Someone has to learn the interface.
Someone has to troubleshoot it when the API changes.
The tool therefore has to earn its place twice: once by solving the original problem and again by being worth the operational overhead it creates.
Test the Failure Path, Not Just the Happy Path
The easiest demo for almost any tool is the happy path.
Everything is configured.
The input is clean.
The account is verified.
The network works.
The file is in the expected format.
The API responds immediately.
Real projects are not like that.
So I deliberately create a few bad conditions during the second day.
For a data tool, I might provide an incomplete record.
For an automation tool, I might send duplicate input.
For an API, I might use an invalid parameter.
For a file-processing service, I might upload the wrong file type.
For an AI tool, I might give it ambiguous instructions and see whether it asks for clarification or confidently produces something unusable.
The point is not to make the tool fail for entertainment.
The point is to learn what failure looks like before failure happens to a client.
NIST’s AI Risk Management Framework takes a similar lifecycle-oriented approach: systems should be evaluated before deployment and monitored as they operate, with testing and measurement documented rather than treated as a one-time activity.
What a Good Failure Looks Like
A useful failure tells me:
- What went wrong
- Why it went wrong
- Whether the original data was preserved
- Whether the operation can be retried
- Whether I can correct the problem myself
- Whether support documentation explains the problem
- Whether the tool leaves a useful log
A bad failure looks like:
“Something went wrong.”
Then nothing else.
That sentence is not an error message.
It is a future support ticket.
I Test Duplicate Inputs
This is one of the simplest tests and one of the most useful.
Suppose an automation receives the same form submission twice.
What happens?
Does it create two customer records?
Two invoices?
Two tasks?
Two orders?
Two emails?
A workflow that works perfectly once can become dangerous when the same event arrives twice.
So I deliberately repeat an input during the sandbox test.
If the tool creates duplicates, I document that immediately.
It does not necessarily mean I reject the tool.
It means I know that the production workflow needs an idempotency check, unique identifier, deduplication rule, or human approval step.
The important thing is discovering this before the workflow touches real data.
I Test What Happens When the Connection Disappears
A tool should not only be tested while everything is online.
I want to know what happens when the network fails, a service times out, an API returns an error, or authentication expires.
The question is simple:
Does the workflow fail safely?
For example, imagine an automation that creates a task and then sends an email.
If the task is successfully created but the email fails, I need to know whether the workflow:
- retries automatically
- reports the failure
- creates a duplicate task during retry
- stops completely
- records the failed step
- allows me to resume from the correct point
This is where a two-day test can reveal more than a polished product demo.
I Check the Data Before Checking the Features
A tool can have fifty impressive features and still be wrong for a project because it handles data poorly.
Before committing to it, I check:
Input: What can I put into the system?
Processing: What happens to that information?
Output: What can I retrieve?
Export: Can I take it somewhere else?
Retention: How long does the service keep it?
Deletion: Can I remove it?
Access: Who can access it?
For client work, these questions are more important than whether the dashboard looks beautiful.
If the tool will process sensitive or confidential information, I also read its privacy and security documentation instead of relying on a marketing page.
I Check Permissions Before Connecting Anything Important
This is especially important with AI tools and integrations.
A tool asking for access to a service is not automatically suspicious.
But I want to know exactly what I am authorizing.
Does it need read access?
Write access?
Administrative access?
Access to all files?
Access to a single project?
The principle I use is simple:
Give a tool the smallest amount of access required to complete the test.
If a tool only needs to read one test document, I do not connect my entire business workspace just because the setup wizard makes that easier.
For software and web applications, the OWASP Top 10 is a useful security reference. The current 2025 edition includes risks such as broken access control, security misconfiguration, software supply-chain failures, authentication failures, and insecure design.
You do not need to become a security engineer to use the principle.
Just stop treating permissions as a checkbox.
I Test the Tool With Someone Who Did Not Build the Test
This is my optional test when the tool is going to be used by another person.
I give the sandbox project to someone else and ask them to complete the basic workflow without me sitting beside them.
Then I watch where they get stuck.
I do not immediately help.
If they cannot figure out what to do, that is useful information.
The problem may not be the tool itself.
It may be the documentation.
It may be the setup.
It may be the terminology.
It may be that the workflow makes sense only to the person who originally configured it.
That matters enormously when a freelancer eventually has to hand a project to a client or subcontractor.
A workflow that only one person understands is not really automated.
It is dependent on one person.
I Measure Time Instead of Asking Whether It Feels Faster
This is where I became much stricter.
I used to finish a test and think:
“That felt much faster.”
That is not a measurement.
Now I record the old process and the new process.
For example:
Old process: 42 minutes
New process: 17 minutes
Time saved: 25 minutes
Then I ask a second question:
How often will I actually perform this task?
If it happens twice a month, saving 25 minutes may not matter much.
If it happens five times a day, the calculation changes completely.
The useful number is therefore not:
Minutes saved
It is:
Minutes saved × realistic frequency
That gives me an estimate of the actual operational value.
I Calculate the Real Cost
The subscription price is only one part of the cost.
I also consider:
- Setup time
- Learning time
- Migration time
- Integration costs
- API usage
- Extra user seats
- Storage
- Support
- Maintenance
- Switching cost
- Exit cost
A $20 tool that saves ten hours every month can be valuable.
A free tool that creates three hours of maintenance every week is not automatically free.
This is why I keep the “time to first working output” number from day one.
It tells me how expensive the tool is in human time before I even look at the invoice.
I Look for Vendor Lock-In
The exit test deserves its own section because it is easy to ignore when everything is working.
I ask:
Can I export my data?
Is the export complete?
Is it in a standard format?
Can another tool import it?
Can I cancel without losing access immediately?
Can I retrieve my historical data after cancellation?
Can I replace the integration without rebuilding the entire workflow?
The answers do not have to be perfect.
But I want to know what I am buying into.
The most dangerous tool is not necessarily the expensive one.
It is the one that quietly becomes impossible to leave.
AI Tools Get One Extra Test
When the tool contains AI, I add a fifth category to the normal reliability test:
Does the output remain useful when the input changes?
I create several variations of the same task.
One is clean.
One is incomplete.
One is unusually worded.
One contains irrelevant information.
One is deliberately ambiguous.
Then I compare the outputs.
I am not looking for identical answers.
I am looking for predictable behavior.
If the AI produces excellent results only when I phrase the prompt exactly the way the demo did, that is important information.
It means the workflow may require more structure before I can trust it.
NIST’s current work on AI evaluation similarly emphasizes structured test and evaluation methods for assessing whether AI systems meet their intended goals and understanding their limitations.
I Never Put Client Data Into the First Test
This rule has saved me more trouble than any other.
The first test uses dummy data.
Not:
“Let’s connect the real customer database and see what happens.”
Instead:
“Let’s create five fake customers and see what happens.”
Use:
- Fake names
- Fake email addresses
- Test orders
- Dummy documents
- Sample invoices
- Non-sensitive images
- Test API keys where supported
Once the workflow behaves correctly, I can decide whether it is appropriate for real information.
The sandbox exists precisely so the first failure does not become a real-world incident.
My Two-Day Decision Has Four Possible Outcomes
I used to think the result of testing a tool was either yes or no.
Now I use four outcomes.
Green: Use It
The tool works reliably, the workflow is clear, the cost is reasonable, and I understand the failure modes.
It can move toward real work.
Yellow: Use It With Conditions
The tool works, but it has a limitation.
Maybe it has a low API quota.
Maybe it needs a manual approval step.
Maybe exports are awkward.
Maybe an important action cannot be automated.
That does not automatically make it unusable.
It means the workflow needs to be designed around the limitation.
Orange: Keep Testing
The tool looks promising, but I have not tested an important part yet.
For example, I may need to test it with a larger dataset or on another device.
I do not put it into production until that uncertainty is resolved.
Red: Walk Away
The tool fails a requirement that matters.
It may be unreliable.
The exit path may be unacceptable.
The permissions may be excessive.
The cost may make no sense.
Or the workflow may simply be more complicated than the problem it was supposed to solve.
Walking away is a successful test result.
The Two-Day Test File
I now keep the same structure for every new tool.
Day One
Tool:
Problem:
Test project:
Time started:
Time to first working output:
Expected result:
Actual result:
Setup surprises:
Dependencies discovered:
Day Two
Bad input tested:
Duplicate input tested:
Connection failure tested:
Limit tested:
Export tested:
Permissions reviewed:
Error messages reviewed:
AI variation tested, if applicable:
Final Decision
What does it replace?
What does it cost?
What does it save?
What can go wrong?
What happens if I leave?
Green, Yellow, Orange or Red?
That is enough.
I do not need a 30-page evaluation document for a $15 monthly tool.
The point is to create a repeatable decision process without turning tool evaluation into another full-time job.
The Test Also Protects Against Shiny-Tool Syndrome
There is another benefit I did not expect.
The procedure makes new tools less exciting.
That is a good thing.
A new AI product can look incredible in a three-minute demonstration.
But after two days of testing, the important questions become boring:
Does it export?
Does it fail clearly?
Does it duplicate records?
Does the API work?
Can I revoke access?
Does it actually save time?
What happens when the network disappears?
Boring questions are exactly what I want before something touches production.
The goal is not to find the most impressive tool.
The goal is to find the tool that remains useful after the demo is over.
My Rule After the Two Days
I no longer ask:
“Is this tool good?”
That question is too vague.
I ask:
“Is this tool reliable enough for this particular job, at this particular cost, with these particular risks?”
A tool can be excellent and still be wrong for my workflow.
A tool can be simple and still be exactly what I need.
And a tool can be free and still be expensive in time.
That is why the two-day test exists.
It separates the tool from the advertisement.
It separates the demo from the workflow.
And most importantly, it gives me permission to walk away before a small experiment becomes a real project.
The Test Is the Product Decision
The biggest lesson from this process is that evaluating a tool is not about predicting the future perfectly.
It is about making uncertainty cheap.
I cannot know exactly how a service will behave six months from now.
I cannot know whether its pricing will change.
I cannot know whether a better alternative will appear next month.
But I can find out whether it works for the job I have today.
I can test its failure modes.
I can measure its actual time savings.
I can understand its data and permission requirements.
I can check whether I can leave.
And I can do all of that before a client project depends on it.
That is enough.
The best tool is not the one that wins the comparison table.
It is the one that survives a small, boring, deliberately inconvenient test and still earns its place in the workflow.
Why two days and not two weeks
Because two weeks of evaluation is itself a project. It builds its own momentum and its own sunk cost, and at the end of it you will choose the tool you spent two weeks on. You will call that a decision.
Two days is short enough that throwing the result away costs nothing. That is the entire point. The test has to be cheap enough to fail, or you will not let it fail.
Most of the tools I tested this year did not make it into anything. That is not wasted time. That is the two days doing its job.
