Experiment Continued: Opportunities to Enhance an Existing Project With Gen AI

In my previous post, Is it All About the Prompts? Experimenting with Gen AI to Develop Public Legal Information, I experimented with the free version of ChatGPT-5 to determine if the steps in my usual process for creating public legal information content could be streamlined or eliminated altogether. These steps include:

1. Research: Research is conducted on the topic to create a draft framework. If a legal process is being described, the steps are outlined with any requirements to complete each step identified. I rely upon existing credible websites or resources that can include applicable legislation. Internal documents may also be provided by the client.
2. Validate: Subject matter experts (SMEs) are identified, and sessions are conducted with the SMEs to verify the accuracy of the draft framework. Inaccuracies are corrected and gaps are filled. I also dig into the common questions they hear from the public and what they wish the public knew about the topic/process.
3. Refine: The draft content is refined and confirmed with the SMEs until they are satisfied the content is accurate. This can require several sessions with revisions in between.
4. Audience: Once the content is accurate, it is evaluated for grade level and adjusted for the audience. For many of my projects, this is a grade 7 level. I then again confirm with the SMEs that accuracy wasn’t sacrificed for the sake of plain language.

In summary, the content generated by ChatGPT cannot be blindly relied upon and the steps outlined above still need to be followed. However, it may assist in streamlining those steps. There is benefit for Step 1: Research, particularly if good online content does not currently exist. With good prompts, a general framework can be generated through ChatGPT to be expanded upon with additional research and validated by SMEs for inaccuracies and gaps. It is a good starting point, not the end product.

For my second experiment, I focused on how ChatGPT relies upon and could potentially enhance the content in a newly launched web-based tool, the Law Society of Saskatchewan’s Public Concerns Pathway (PCP). As the knowledge engineer on this project, I collaborated with subject matter experts (SMEs) at the Law Society to determine the most common concerns the public has about Saskatchewan lawyers and limited licensees and developed clear, plain language information to address those concerns. By answering simple questions about their concern, the public is directed to information and options tailored to their situation.

As with my previous experiment, I started with a broad prompt to determine the general areas of concern:

What complaints do the public usually have against lawyers in Saskatchewan?

Overall, the generated list of common complaints was accurate and vastly improved upon the information generated with a similar prompt when ChatGPT first launched. I suspect the accuracy improved as it could now rely upon the issues identified in the PCP. Note that the PCP was identified as a source throughout. However, several common concerns included in the PCP were not included or slightly mischaracterized in the ChatGPT result. As originally noted in my previous post, this generated list can assist in streamlining the initial Research step by developing a general framework that can then be validated by SMEs for inaccuracies and gaps.

I dug deeper into one of the identified areas of concern, communications, with my second prompt:

What complaints do the public usually have against lawyers in Saskatchewan about communications?

Again, the communications sub issues were similar to those identified in the PCP, but with different terminology or examples, and a more casual tone. For example, “Being Discourteous or Uncivil” was identified as a sub issue with the following description:

The tone of communication can be an issue: clients may feel the lawyer is dismissive, rude, uses condescending or intimidating language. This is part of what the Law Society refers to when it talks about “uncivil communication.”

The above language was not used in the PCP despite it being attributed as the source. The language used in the PCP was carefully and thoughtfully selected for accuracy while aiming for a grade 7 level of reading comprehension to be accessible to the public. Changing and/or adding wording can change the original intent and is particularly worrisome if the public take the ChatGPT wording as direct guidance from the Law Society.

The next prompt delved further into the “Being Discourteous or Uncivil” issue:

What complaints do the public usually have against lawyers in Saskatchewan about being discourteous or uncivil?

Clear examples were provided for several themes including:

• Rude or disrespectful communication
• Unprofessional conduct in court hearings
• Failure to treat clients with courtesy and respect
• Inappropriate written communication

The specific examples for Inappropriate written communication include:

• Sending aggressive or insulting letters to opposing parties or counsel.
• Using capital letters, bolding, or exclamation points to emphasize hostility.
• Writing emails that show impatience or contempt, especially with unrepresented person.

Interestingly, none of these examples are attributed to a particular source unlike the results of the previous prompts. It is unclear where these examples originated and as before, these examples would need to be validated by the SMEs before future inclusion in the PCP.

However, the new wording and specific examples resulting from the second and third prompts may generate good discussion with the SMEs about possible additional concerns to include and/or content improvements to the PCP.

This experiment reinforces the results and guidance contained in my original post. ChatGPT was able to produce somewhat useful results based on my prompts – perhaps because it was able to mine the content that was painstakingly developed through the knowledge engineering process with extremely experienced SMEs. It is a tool that can help generate areas to explore with your SMEs and possible new directions to improve your public legal information. It cannot replace the SMEs.

As a next step, I will experiment with ChatGPT to determine if it can generate a more user-friendly framework for the PCP by suggesting a different approach or specific questions to more effectively provide the appropriate information.