Why your AI assistant keeps calling the wrong tool: a 36-server audit

A static-analysis scan of 36 popular AI tool servers found that 11 of them, including official integrations from MongoDB, Notion, and Airtable, got a D or F for agent usability, though all of them passed the protocol's compliance checks.

The scanner, mcpgrade, is a Lighthouse-style linter for Model Context Protocol (MCP) servers, the open standard that lets AI assistants like Claude and ChatGPT call external tools such as calendars, databases, and code repositories. The author, a developer who runs a production AI-agent integration practice, built mcpgrade to grade how usable those tools actually are to a language model. It runs as a single command with no API key required.

Out of 36 servers scanned, 15 received an A grade and 11 received a D or F. The bottom of the leaderboard reads like a roll call of enterprise software: MongoDB's official server scored 66, Notion's official server 62, Airtable 69, Todoist 67, and Firecrawl dead last at 57. Two more, Stripe and Supabase, were excluded entirely because they couldn't be scanned with dummy credentials rather than graded.

The dominant rule the linter enforces is D004: a parameter has no description. Firecrawl's 134 errors are 132 of this kind. Todoist has 110. MongoDB and Airtable have 66 each. The schema itself is fine. The names are fine. The token count is fine. The model gets a name and a type, and nothing else.

When an AI assistant tries to use one of these tools, it ends up calling the wrong one, hallucinating arguments it has to invent, ignoring the tool entirely, or burning roughly 8,000 tokens parsing schemas on every request. None of that is a protocol bug. The MCP spec covers transport, JSON-RPC framing, capability negotiation, and schema shapes, and it says nothing about whether a model can actually use the tools a server exposes. A server can be 100% spec-compliant and still be unusable.

The grade reflects description quality, not observed task success against each server, so the leaderboard is a documentation audit dressed as a usability one. That's a real distinction. A model might still flounder on a well-described tool, and a model might limp through a poorly described one. What mcpgrade measures is the part the vendor controls with a one-line edit: a string that tells the model what each parameter is for.

Any vendor can run npx mcpgrade --stdio 'npx -y their-mcp-server' against their own server, get a report in seconds, and ship a fix. The full sortable leaderboard is public. Several of the top scorers, brave-search, google-maps, slack, and gitlab, are explicitly marked as archived reference implementations still widely installed and copied, which means the A list is partly a roster of templates other servers should be learning from, not a list of current production deployments.

Hacker News commenters took up the thread with the obvious question: if the fix is a missing description string, why isn't the ecosystem shipping one? The answer is buried in the rule list: the protocol doesn't require it. A documentation lint is exactly the kind of thing the community can add on top of the spec, and the author has already shipped one.

The audit is a snapshot, and the panel is not exhaustive: Stripe and Supabase couldn't be scanned, and the server landscape is moving fast. A server that scored 57 in July may well ship a fix in August. But the pattern is clear: the brand names on the F list are the ones enterprise buyers are most likely to plug into their AI assistants, and the gap between "passes the protocol" and "an agent can use it" is one missing string per parameter.