AI Chatbot Passes the Bar Exam - Lawyers Are Finished
After GPT-4 scored in the 90th percentile on the Uniform Bar Exam, numerous commentators declared that AI would replace lawyers and make law school unnecessary.
AI-generated
When OpenAI published GPT-4's benchmark results showing a 90th percentile score on the Uniform Bar Exam, a wave of commentary declared that AI was about to replace lawyers. Social media posts, YouTube videos, and news articles proclaimed that law school was now a waste of money.
Passing a multiple-choice and essay exam is very different from practicing law. The bar exam tests knowledge recall and basic legal reasoning in a controlled setting. Actual legal practice requires client management, courtroom presence, negotiation skills, ethical judgment, deep understanding of precedent in specific jurisdictions, and the ability to handle ambiguous situations where the law is unclear.
AI tools have become valuable for legal research, document review, and contract analysis. But they supplement lawyers rather than replace them. In fact, most legal AI products are marketed to lawyers as productivity tools, not to clients as lawyer replacements.
Benchmark performance is a poor predictor of real-world job replacement. The bar exam is a standardized test with known patterns, which is exactly the kind of task AI excels at. The messy, relational, judgment-heavy reality of legal practice is where AI struggles. AI legal tools also have a hallucination problem: fabricating case citations (which has already led to sanctions for lawyers who did not verify AI output).
OpenAI: GPT-4 bar exam results - https://arxiv.org/abs/2303.08774
ABA Journal: "AI and the legal profession" - https://www.abajournal.com/
Stanford CodeX: AI + Law research - https://law.stanford.edu/codex-the-stanford-center-for-legal-informatics/