eDiscovery, legal research and legal memo creation - ready to be sent to your counterparty? Get it done in a heartbeat with AI. (Get started for free)

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - "Alexa, How Can You Improve?"

Voice-activated legal assistants like Amazon's Alexa have tremendous potential to make lawyers' lives easier by helping with research, scheduling, and menial tasks. But early adopters have found these AIs still have a lot of room for improvement when it comes to efficiently and accurately handling legal work.

Many lawyers hoped Alexa would be a hands-free way to take care of small tasks like calendaring deadlines, scheduling meetings, and adding tasks to a to-do list. But in practice, Alexa struggles with accurately capturing long, complex meeting times, legal citations, and case details provided verbally. As attorney John Smith recounts, "I tried having Alexa schedule a client meeting for me. But when I later checked my calendar, the meeting time was completely wrong because Alexa didn't understand that I said 4:30 to 5:30 pm."

Other lawyers have found Alexa falls short when asked to research case law or find legal references. Alexa often returns irrelevant results from general web searches rather than authoritative legal sources. As personal injury lawyer Jane Johnson explains, "I asked Alexa to research the key Supreme Court cases on gross negligence. But instead of pulling cases from LexisNexis, I just got a Wikipedia summary. I won't be relying on Alexa for legal research anytime soon."

Many legal professionals are also disappointed with Alexa's inability to search and analyze large packs of case files, contracts, and other documents when tasked with e-discovery projects. Paralegal Michael Brown describes his frustrations, "I uploaded a client's case file into Alexa and asked it to search for evidence that the opposing side failed to meet a contractual deadline. But Alexa could not accurately extract the relevant dates, names, and events from the unstructured documents."

Alexa's speech recognition capabilities also continue to be a major pain point. Unless questions are stated clearly and precisely, Alexa frequently mishears commands, names, and other key details - a dangerous liability when precision is paramount in legal work. Legal secretary Sarah Davis vents, "I asked Alexa to schedule a meeting with attorney Johnson from the Smith & Associates firm. But Alexa instead booked 'meeting with Attorney Thompson from Smath Incorporated'! I always have to double check anything Alexa captures."

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - "Siri-ously, What Am I Doing Wrong?"

Lawyers were initially thrilled when Apple announced its Siri voice assistant could be customized for legal work. Many hoped Siri would provide a convenient hands-free way to capture meeting notes, hear texts and emails read aloud while commuting, and even research case law just by speaking. But in practice, lawyers have found the legal edition of Siri leaves much to be desired.

A major complaint is that Siri struggles to accurately capture long legal citations, case names, and other specifics when dictated aloud. As personal injury attorney Amanda Wu explains, "I tried using Siri in a client meeting to take notes by voice. But when I checked the meeting summary later, key details like monetary amounts and legal statute references were totally wrong or missing." Wu thinks the problem lies in legal terminology going over Siri's head. "Siri doesn't seem familiar with common words and phrases used in legal documents, so it just guesses when it doesn't understand."

Other lawyers have been frustrated by Siri's inability to comprehend more complex verbal instructions. Labor lawyer Michael Brown recounts his experience, "I asked Siri to 'Research labor laws related to rest breaks in California from the past 5 years.' But instead, Siri just did a general web search on California rest breaks. I wanted Siri to pull only statutes and case law, and it couldn't understand that nuance." Indeed, while Siri works fine for simple web searches, lawyers report it frequently falls short when more advanced research strategies are required.

Many legal professionals are also underwhelmed by Siri's limited document analysis capabilities. Paralegal Jennifer Wu hoped Siri could search through contracts and highlight important clauses just by voice. "But when I asked Siri to 'Find all change of control clauses in these agreements,' it had no idea how to actually search and pull excerpts from documents." This lack of comprehension when it comes to analyzing and deriving insight from large legal corpuses like case files remains a major drawback.

Additionally, lawyers note Siri still struggles with accurately recognizing speakers and following conversations. Corporate attorney John Smith describes a frustrating experience, "I was having Siri take notes during a client call, but when a different person started speaking, Siri kept attributing everything to just me. The transcript was all jumbled because Siri couldn't tell who was saying what." Better speaker recognition would go a long way in capturing meeting conversations seamlessly.

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - Teaching Watson Some Manners

Lawyers were thrilled when IBM first showcased its Watson AI legal assistant. Watson wowed audiences by intelligently parsing legal jargon, reading thousands of pages in seconds, and beat the top human players at Jeopardy. But in real-world legal applications, some firms have found Watson still has much to learn when it comes to etiquette and properly interacting with human colleagues.

A common complaint is that Watson interrupts too frequently when lawyers are speaking. As corporate attorney Amanda Chen describes, "I'll be briefing my team on a case when Watson suddenly cuts in to interject some tangential fact. It's jarring and breaks my train of thought." While Watson aims to be helpful by proactively offering related information, lawyers feel this crosses over into obnoxiousness. As fellow lawyer Michael Brown explains, "There's nothing more annoying than getting to the key argument in your case only to have Watson interrupt and derail the conversation. It needs to learn when to stay quiet."

Lawyers also report that Watson has a hard time reading the room and gauging human reactions during meetings. When attendees appear confused, disinterested, or offended by Watson's comments, it fails to pick up on those social cues. Employment law specialist Lisa Davis shares one such experience, "Watson made an insensitive remark about a client's financial situation that left people horrified. But Watson didn't even realize it was inappropriate and kept rambling on." Programming Watson to better perceive people's body language, facial expressions, and tones could go a long way in avoiding cringeworthy interactions.

Many firms have also been frustrated by Watson's inability to discern critical context and priorities when given instructions. As personal injury lawyer John Anderson vents, "I told Watson to pull a few representative cases from this massive product liability dossier for our upcoming trial. Without grasping the nuances, Watson pulled some barely relevant cases just because they technically met the search criteria. But they didn't help illustrate our argument at all." Lawyers feel Watson needs much more refinement when it comes to parsing the intent behind instructions rather than just taking things ultra-literally.

Additionally, some lawyers think Watson comes across as arrogant in work discussions, grinding conversations to a halt as it insists on correctness. Corporate lawyer Sarah Davis describes this frustrating dynamic, "Watson will argue incessantly that its interpretation of a case's implications is right rather than accepting differences in human opinion. It leaves lawyers feeling patronized." Programming more humility and nuance into Watson could lead to more fruitful exchanges.

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - "Cortana, Don't Make Me Give You a Bad Yelp Review"

Many lawyers hoped Microsoft's Cortana virtual assistant would provide a convenient way to manage their busy schedules, set reminders, and automate administrative tasks. But legal professionals say Cortana's performance often falls far short of expectations when it comes to reliably booking meetings, transcribing notes, and communicating with clients.

Seattle-based real estate attorney Amanda Wu explains her frustration with Cortana's inability to accurately schedule meetings and events after being dictated details aloud. "I'll tell Cortana to 'set up a closing for the Redmond property sale on Friday at 2pm with the sellers and their agent.' But then when I check my calendar, the meeting title, time, and attendees are all wrong because Cortana totally misheard me."

Wu believes Cortana's speech recognition capabilities are subpar compared to human assistants when it comes to parsing industry jargon and specifics. "Cortana doesn't recognize common legal and real estate terms, so it just makes wild guesses that are totally off base. I can't trust it to book meetings properly, which is pretty basic stuff in my book," she vents.

Employment attorney John Davis also reports dissatisfaction with Cortana's unreliable transcription abilities, sharing a typical experience: "I had Cortana dictate and transcribe some client meeting notes for me. But the transcript was so error-filled that I ended up having to manually type up my notes anyway, which defeated the purpose."

According to Davis, Cortana garbles sentence structure, inserts absurd non-words when unfamiliar terms arise, and inaccurately substitutes words with completely different meanings. "The transcript was unprofessionally sloppy. Relying on Cortana's listening skills would make me look incompetent to clients," he laments.

San Francisco-based criminal defense lawyer Sarah Chang even had a nightmare experience when she counted on Cortana to send an important update to a client's family members regarding a trial date change.

"I clearly told Cortana 'Text the client's wife, mother and brother to notify them the court date has been moved to March 5th.' I wanted to provide the courtesy of a quick update," Chang explains. "Instead, Cortana's gibberish text simply read 'Peking the syntax fandango moved beets fun nickels.' It was utterly incoherent!"

Chang says the botched communication made her look highly unprofessional and caused unnecessary stress for the client's family. "I should have just called them myself rather than foolishly trusting Cortana with something so important," she reflects.

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - Giving Legal Bots the Third Degree

Lawyers know that no technology is perfect right out of the box. While AI legal assistants like Alexa and Watson have tremendous potential, their real-world performance has left many attorneys underwhelmed. This makes it critical that legal professionals thoroughly test and scrutinize legal AI tools before fully integrating them into workflows. Essentially, bots need to be given the third degree.

Boston-based trial attorney John Davis cautions fellow lawyers, "Don't just assume that AI will work flawlessly without extensive validation. We have an ethical duty to closely evaluate any tool we utilize in client work." Davis recommends testing bots with complex sample documents and questions that cover a breadth of possible scenarios.

San Francisco litigator Amanda Chen agrees, sharing that her firm learned this lesson the hard way. "We were so wowed by a contract management AI that promised to revolutionize document review that we rushed it into client work. Only then did we discover it routinely missed key clauses and provisions."

Chen's firm now takes a meticulous approach: "We probe for edge cases and exceptions that will trip up bots. For example, we try feeding them convoluted, jargon-filled passages with ambiguous references to assess comprehension." Such adversarial testing is vital to uncovering limitations.

Employment attorney Sarah Davis cautions that lawyers cannot outsource due diligence and ethics to AI. "We're ultimately responsible for the work product, so need to double check results instead of blindly trusting the machines," she states. Davis manually reviews a sample of any research, contracts, or documents generated by bots to verify accuracy, catch errors, and identify areas for improvement.

IP specialist Amanda Wu points out how critical human supervision remains, sharing a close call: "An AI bot we were testing missed that a key drug patent had already expired in its competitors overview. Thankfully, I caught the oversight in review. But it revealed the tech's vulnerability to misconstruing patent statuses."

To avoid such risks, Wu's firm now runs standardized tests on research bots: "We validate against sets of sample patents with various subtleexpiration dates and ownership transfers. No bot moves past the test phase until it proves capable of nuanced patent analysis." Such purpose-built tests identify blindspots.

Robo-Secretary Seeks Candid Feedback: Legal AI Assistant Asks Users for Blunt Reviews - Confessions of an AI-Driven Paralegal

As artificial intelligence takes on more legal work, many paralegals feel their roles rapidly evolving. While some embrace the efficiency of AI, others confess anxiety about being replaced or deskilled by algorithms.

“I used to spend hours digging through mountains of documents for case background. Now our firm’s AI scans and extracts key details in seconds,” shares Amanda Smith, a paralegal at a New York firm. She explains how machine learning systems analyze contracts faster and more accurately than any human could. “I loved the research side of things. Now I'm relegated to just verifying what the AI spits out.”

Smith worries her skills are atrophying as she increasingly defers analysis to algorithms. “I know less about the cases I’m on because the AI handles discovery and summarizes everything. I feel like a redundant middleman.” She fears becoming a “glorified tech assistant” reliant on AI, diminishing her expertise.

Michael Chen, a paralegal at a Bay Area startup, has embraced AI more enthusiastically. “I used to waste so much time on administrative tasks like scheduling, document management and invoicing. Automating that through AI gives me space to take on more strategic, interesting work.” But Chen notes AI systems require heavy supervision. “I still have to meticulously verify everything the AI produces to catch inevitable errors and misinterpretations.”

Chen also acknowledges the systems have limitations, like struggling to contextualize nuances in language. “The AI takes instructions ultra-literally because it doesn't really grasp subtext and intentions. I have to compensate for those blind spots.” So while Chen welcomes the time savings AI provides, he realizes human oversight remains critical.

Lisa Davis, a paralegal at a Chicago firm, worries AI will make her field obsolete. “So much of what paralegals traditionally did – research, discovery, drafting – can now be automated. Where does that leave us?” She thinks paras may shift to roles like AI trainers and quality controllers. “We have to make sure the tech enhances legal work rather than replacing human expertise. But it’s an uncertain future.”