NYC's official MyCity chatbot tells businesses to break the law

What happened

The Markup found that New York City's MyCity business chatbot was confidently giving illegal advice: that employers could take workers' tips, landlords could refuse Section 8 voucher holders, and that there were no rent restrictions — all contrary to New York law.

Impact

Business owners received official-looking guidance that would have exposed them to legal liability; the city faced sustained criticism for keeping the bot online.

How this could have been prevented

Retrieval grounding against authoritative sources, refusal behavior on legal-compliance questions, and pre-deployment red-teaming catch systematic errors in government chatbots.

Sources

More hallucination incidents

Sixth Circuit imposed $116K sanctions over AI-hallucinated appellate briefs Air Canada held liable for chatbot's invented bereavement-fare policy

Would runtime governance have caught this?

Guardion enforces policy on every agent action inline — with visibility, tamper-evident evidence, and DLP for agents and MCPs.

Request a demo