Using OpenAI’s ChatGPT to Summarize Healthcare IT Articles: An Experiment

As someone who has been using ChatGPT to summarize healthcare IT articles, I can tell you that the results can be wildly inconsistent. Sometimes, the summaries are spot on and capture the essence of the article in just a few sentences. Other times, the summaries have nothing to do with the article at all.

This experiment was inspired by the inconsistencies I found in the responses I was getting from OpenAI. I believe that by continuing to refine the prompt and gathering more data, we can create a more reliable method for summarizing healthcare IT articles.

I began by sending the URL of every post collected by MediFormatica’s new aggregator through the API and asked OpenAI to summarize the contents of that URL in a 50-80 word summary. This included articles, news, videos, podcasts, and more.

The results were disappointing to start with, but as I started improving the prompt I’m sending to OpenAI, the results started to improve. Still some responses are hilarious and have nothing to do with the actual article.

I’m now posting this response along with the short  33 word excerpt from the original article on social media platforms like LinkedIn, Twitter, and Facebook. This allows more people to compare the results and provide their feedback. At this point I’m not tweaking the ChatGPT responses in anyway, but over time, I’ll be tweaking the prompt I’m using with OpenAI to improve the results. For example, I’m currently removing any references to LinkedIn in the prompt, as OpenAI was trying to include it in every response even when it had nothing to do with LinkedIn.

Here’s another one from my early testing samples where ChatGPT suggests that LinkedIn may be key for combating clinician burnout , there were even more suggestion for meditation, self disciplinary theory and others 🙂

Are you tired of feeling burnt out at work? According to a recent blog post, the key to combating burnout may lie in social media. #SocialMedia #BurnoutPrevention The post highlights the benefits of using social media platforms, especially LinkedIn, to connect with peers, share knowledge, and find support. #LinkedIn #ProfessionalNetworking By joining groups and participating in discussions, healthcare technology experts can build a sense of community, gain new perspectives, and reduce feelings of isolation. #CommunityBuilding #MentalHealth The post also notes that social media can provide a platform for self-expression and creativity, which can be therapeutic and help combat stress. #SelfExpression #Creativity

But why go through all of this trouble? The answer is simple: healthcare IT is a rapidly changing field, and staying up to date on the latest developments is critical. With ChatGPT, you can quickly scan through articles and get a summary of the key points without having to read the entire article. My goal is to come up with a formula for the prompt that would work with healthcare IT topics in general and share it for all to benefit. By doing so, I hope to improve the accuracy of AI-generated article summaries and make them more useful to readers.

Of course, the accuracy of ChatGPT depends on many factors, including the quality of the original article, the length of the article, and the complexity of the subject matter. That’s why it’s important to use ChatGPT as a tool to aid in your research, rather than relying on it as the sole source of information.

The API is connecting to the gpt-3.5-turbo-0301 model and here’s the prompt I’m currently using with OpenAI. I’ll be updating it even few days.

As a healthcare technology expert, write a 50-80 word extractive summarization for social media platforms, that is focused, accurate, and strictly reflects the content based on a blog post from the given URL. The summary should include all headings from the blog post, with inline hashtags for each heading. When including the inline hashtags, use specific hashtags related to the headings rather than generic healthcare or technology hashtags. If the headings in the blog post are too long or unclear, feel free to rephrase them into shorter, clearer versions that still convey the main idea. The target audience is other healthcare technology experts.

 

For each request I make, I’m storing the following

  1. The summary returned
  2. The prompt that was sent
  3. The full JSON response
  4. The number of tokens used

 

You can check the full details for all the posts from this page

I am an experienced healthcare executive with over 15 years of deep clinical and informatics expertise in the design, development, implementation and support of complex healthcare solutions. Beginning my career as a cardiologist, I understand the needs of clinicians and the nature of their work.