Artificial Intelligence is now able to pass the Multistate Professional Responsibility Examination (MPRE), the test used to measure candidate’s fitness to be licensed as a lawyer.

Just last year, OpenAI’s GPT-4 took the bar exam and passed.

Now, two of the leading large language models (LLMs) have passed a simulation of the MPRE, required in all but two U.S. jurisdictions to measure potential lawyers’ knowledge of professional conduct rules.

This time, the test was conducted by researchers at LegalOn Technologies, led by Gabor Melli, VP of artificial intelligence, who concluded that two of the leading generative AI models are capable of passing the legal ethics exam.

“This research advances our understanding of how AI can assist lawyers and helps us assess its current strengths and limitations,” stated Daniel Lewis, U.S. CEO of LegalOn. “We are not suggesting that AI knows right from wrong or that its behavior is guided by moral principles, but these findings do indicate that AI has potential to support ethical decision-making.”

The researchers tested OpenAI’s GPT-4 and GPT-3.5, Anthropic’s Claude 2, and Google’s PaLM 2 Bison, on their ability to correctly answer questions modeled for the MPRE.

GPT-4 performed best, the researchers found, answering 74% of questions correctly and outperforming the average human test-taker by an estimated 6%. Claude 2 answered 67% correctly. GPT-3.5 answered 49% correctly and PaLM 2 answered 42% correctly.

Both GPT-4 and Claude 2 scored above the approximate passing threshold for the MPRE in every state where it is required, a threshold ranging between 56-64% depending on the jurisdiction.

The LegalOn researchers tested the LLMs against 500 simulated exam questions created by Dru Stevenson, a Law Professor who teaches professional responsibility at South Texas College of Law Houston. He designed the questions to have the same format and style as the questions on the current MPRE.

While the researchers concluded that GPT-4 “exhibited remarkable proficiency,” its performance varied by subject area. It performed particularly well in questions related to conflicts of interest and client relationships, and less well on topics such as the safekeeping of funds.

You can read more about this development here.

If you need to pass the MPRE (without using AI, at least for now), Marino Legal offers a free online MPRE course. Study using our program for just 16 hours, and you will be prepared to pass the exam with confidence. Sign up for free here.