When the Lawyer Stopped Thinking and the Machine Started Hallucinating
A judge in Mississippi recently did something remarkable. She did not punish lawyers on both sides of a case for using artificial intelligence. She cancelled a trial and removed counsel because they had blindly trusted AI by submitting citations to non-existent cases, referencing judgments that had never been written, and presenting legal authority that existed nowhere except inside the confident, fluent, and entirely fabricated output of a generative AI model.
The incident was widely reported as a story about technology gone wrong. However, it is not. It is a story about professional failure dressed in the language of innovation. And for India, a country with one of the largest and most overburdened legal systems in the world, and a legal community increasingly turning to AI tools to manage that burden, is a warning that deserves far more serious attention than it has received.
The most important thing to understand about the Mississippi case is what it was actually about. Artificial intelligence did not file fabricated citations. It did not sign a legal brief or make a representation before a court. A lawyer did all of that. The AI was a tool. The professional holding that tool was the one who failed.
This distinction matters enormously. Legal accountability is built on the principle that the advocate who signs a document owns that document completely, with no exceptions. Whether the error originates from a sleepless paralegal, an overworked junior associate, a faulty legal database, or a generative AI model that invented cases with the confidence of a seasoned professor, the responsibility belongs to the person who submitted it to the court.
In India, this question has immediate and urgent implications. If an advocate files a petition before the Supreme Court or a High Court containing AI-generated citations to judgments that do not exist, what happens? Who is liable, the advocate, the firm, the AI vendor? The Bar Council of India has no specific framework to answer this question yet. It needs one urgently. The principle must be clear and unambiguous: AI does not create new excuses for professional negligence. It only creates new forms of it.
Hallucinated Justice: The Threat No One Predicted
Generative AI systems such as ChatGPT, Google Gemini, and their legal-facing counterparts are trained to produce fluent, confident, and contextually appropriate text. They are not trained to be truthful in the way a law library is truthful. When asked to provide case citations, legal statutes, or judicial quotations, these systems will sometimes produce material that sounds authoritative but does not exist — a phenomenon that researchers and technologists call “hallucination.”
In most contexts, a hallucination is inconvenient. A chatbot inventing a restaurant recommendation or a fictional historical date is mildly irritating and easily corrected. In a court of law, a hallucination is something qualitatively different. It is the insertion of fabricated authority into a system that depends entirely on verified authority. Courts decide cases on precedent. Lawyers build arguments on judgments. Judges reason by reference to established legal principles. If any part of that chain of authority is invented, the entire structure of the argument is corrupt.
The danger compounds because these hallucinations are not obviously wrong. They do not appear in red text with a warning label. They are written in the precise, formal, citation-heavy style of real legal documents. A lawyer rushing to meet a filing deadline, or one who simply did not think it necessary to verify an AI output, may submit them without realising what has happened, until a judge notices, or worse, until the fabricated authority goes unchallenged and influences a decision.
Courts in the United States, Canada, and India have already begun flagging this concern. The Supreme Court of India has recently expressed discomfort with AI-generated petitions containing inaccurate or fictitious legal material. The Bombay High Court has dealt with similar citation-related issues. These are not isolated incidents. They are early signals of a systemic vulnerability. A legal system can survive honest human disagreement. It can survive imperfect arguments, insufficient evidence, and genuine ambiguity. What it cannot survive is fabricated authority that no one questions.
Where AI Genuinely Belongs in Legal Practice
None of this means AI has no place in legal work. It means it has a specific, bounded, and clearly defined place, and the profession must be precise about where that place is.
Artificial intelligence is genuinely excellent at information-intensive, repetitive, and volume-heavy tasks. In a legal context, this includes summarising long judgments to help a lawyer quickly understand the core finding, translating documents across languages, organising and categorising large volumes of evidence, creating chronological timelines from case records, conducting preliminary contract reviews to flag non-standard clauses, and assisting in due diligence by scanning hundreds of documents for specific provisions.
For Indian law firms working with corporate clients, private equity transactions, or compliance-heavy sectors, these capabilities are not trivial. The amount of documentation involved in a large infrastructure project or a merger filing can run into thousands of pages. AI can reduce weeks of preliminary review to days. For junior lawyers entering the profession, AI can accelerate learning by helping them navigate case law and statutory frameworks more efficiently than a manual search would allow.
These are legitimate, valuable, and appropriate uses of the technology. The argument is not against AI in law. The argument is about precision knowing exactly where the tool’s strengths end and where human judgment must begin. Studies on legal AI systems consistently find that the technology performs best when handling information management rather than legal reasoning. The distinction is not semantic. It is foundational.
Where AI Must Never Replace the Lawyer’s Judgment
There is a category of legal functions that cannot be automated, not because the technology is insufficiently advanced yet, but because the functions themselves require something that no statistical model can replicate: human moral and contextual judgment.
Determining whether someone is guilty of a crime involves assessing not just facts but the believability of witnesses, the credibility of confessions, the weight of circumstantial evidence, and the proportionality of punishment. These are not calculations. They are judgments. Deciding whether to grant bail requires evaluating the likelihood of flight, the nature of the alleged offence, the personal circumstances of the accused, and the potential harm to society, none of which reduces to a data pattern.
Interpreting legislative intent, evaluating the sincerity of remorse, assessing the credibility of expert testimony, understanding the human consequences of a contractual dispute, these are all domains in which AI cannot operate responsibly. The Andhra Pradesh High Court has explicitly cautioned against using AI for evaluative and judgmental functions in legal proceedings, recognising that legal reasoning is an inherently human act involving moral understanding that cannot be delegated to a machine.
The point is not that AI will get these things wrong. The point is that even if it were right, the legitimacy of a legal outcome depends on a human being exercising accountable judgment. Justice is not the output of an optimised function. It is a social and moral act.
The Verification Paradox That No One Is Talking About
There is an intellectual tension at the heart of the AI-in-law conversation that most advocates for the technology are reluctant to address directly. Lawyers adopt AI because it promises efficiency. It will research faster, draft faster, organise faster. Time saved means cost saved, and for large law firms handling enormous caseloads, this is a genuinely compelling proposition. But the Mississippi case and others like it have introduced a requirement that disrupts this promise: every AI-generated output now requires human verification.
Every citation must be checked against the original database. Every quotation must be cross-verified with the source judgment. Every legal proposition must be confirmed against current statute and binding authority. If any of these verification steps is skipped, the lawyer is exposed to exactly the kind of professional and legal liability that the Mississippi case demonstrated.
The question this raises is uncomfortable but necessary: if thorough verification requires as much time and expertise as the original research, is AI actually saving time? Or is it simply shifting effort, from research and drafting toward review and error-detection, while introducing new categories of risk along the way?
For overworked lawyers in Indian district courts and high courts, this is not an abstract concern. It is a practical one. The technology that promised to ease their burden may, if used carelessly, add to it. The efficiency dividend of AI in legal practice is real, but it is conditional: it depends entirely on a lawyer having the expertise to recognise when the AI has gone wrong.
The Ethics Question: Does AI Endanger a Lawyer’s Duty to the Court?
Legal ethics in India, as codified under the Advocates Act and the Bar Council of India Rules, rests on a hierarchy of duties. A lawyer’s foremost professional obligation is not to a client, a firm, or a billable hour target. It is to the court.
This duty encompasses the obligation of candour — never misleading the court. It encompasses the duty of competence — bringing adequate knowledge and preparation to every matter. It encompasses the duty of diligence — taking reasonable care to ensure that submissions are accurate and properly supported.
When an advocate submits an AI-generated document containing hallucinated citations without independent verification, each of these duties is potentially compromised simultaneously. The AI did not breach the duties. The lawyer did, by treating a machine’s output as a substitute for professional judgment.
The broader principle the article earlier examined bears repeating here: AI does not create new ethical problems in the legal profession. It magnifies old ones. The lawyer who relied uncritically on AI is the spiritual successor of the lawyer who relied uncritically on what a junior told them, or what they remembered from a lecture twenty years ago, without checking. The tool is different. The failure is identical.
What Indian Courts and the Bar Must Do Now
India needs a formal, enforceable framework for AI use in legal practice before the Mississippi scenario plays out in Patiala House or Bombay High Court. Several elements of such a framework are already being debated internationally and can be adapted for the Indian context.
Mandatory disclosure of AI-assisted filings would require advocates to indicate when AI tools were used in preparing documents submitted to the court, creating a record and reinforcing the expectation of verification. Verification certificates, appended to AI-assisted filings, would require the signing advocate to formally attest that all AI-generated citations and legal propositions have been independently confirmed against primary sources. Penalties for fabricated citations, separate from and in addition to existing contempt provisions would ensure that AI-related misconduct is treated with the seriousness it deserves.
Law schools must begin integrating AI literacy into their curricula: not courses about AI in the abstract, but training in how to use legal AI tools responsibly, how to identify hallucinations, and how to understand the limits of what the technology can and cannot be trusted to do. Bar Council guidelines specific to AI use would give practitioners clear professional standards to operate within. And the Supreme Court of India, in its role as the apex supervisory body of the judicial system, is well-positioned to issue practice directions that establish baseline standards across all courts.
The future of legal practice in India will involve AI. There is no responsible argument for excluding it entirely. The question is whether the profession will shape the terms of its integration or simply allow the technology to dictate them by default.
Justice Is Not an Autocomplete Function
The Mississippi case will be remembered as a turning point, which is a moment when the legal profession was forced to confront the consequences of confusing convenience with competence. The most successful lawyers of the coming decade will not be those who refuse to use AI, nor those who use it uncritically. They will be those who understand it precisely: what it can do reliably, where it fails predictably, and how to extract its efficiency benefits while maintaining the professional judgment, ethical accountability, and intellectual rigour that the practice of law has always demanded.
Judges are not search engines. Courts are not content platforms. Legal arguments are not autocomplete suggestions. AI can accelerate legal work. It can reduce drudgery. It can make the law more accessible to practitioners who lack the resources of large firms. But the responsibility for truth, accuracy, and fairness — the three pillars on which every legitimate legal outcome rests — must remain entirely and unconditionally human. When lawyers stop thinking and machines start guessing, it is not just the lawyers who suffer. It is everyone who comes to court expecting justice.
