Skip to content

Medical professionals often summarize patient encounters in transcripts written in natural language, which include details about symptoms, diagnosis, and treatments. These transcripts can be used for other medical documentation, such as for insurance purposes, but as they are densely packed with medical information, extracting the key data accurately can be challenging.

You and your team at Lakeside Healthcare Network have decided to leverage the OpenAI API to automatically extract medical information from these transcripts and automate the matching with the appropriate ICD-10 codes. ICD-10 codes are a standardized system used worldwide for diagnosing and billing purposes, such as insurance claims processing.

The Data

The dataset contains anonymized medical transcriptions categorized by specialty.

transcriptions.csv

ColumnDescription
"medical_specialty"The medical specialty associated with each transcription.
"transcription"Detailed medical transcription texts, with insights into the medical case.
# Import the necessary libraries
import pandas as pd
from openai import OpenAI
import json
# Load the data
df = pd.read_csv("data/transcriptions.csv")
df.head()
# Initialize the OpenAI client
client = OpenAI()

## Start coding here, use as many cells as you need
function_definition = [{
    "type": "function",
    "function": {
    "name": "extract_medical_data",
    "description": "Extract age, recommended treatment or procedure, medical specialty, and ICD code from the transcript that will be passed.",
    "parameters": {
        "type": "object",
        "properties": {
            "age": {
                "type": "string",
                "description": "The age of the patient in the transcript."
            },
            "recommended_treatment": {
                "type": "string",
                "description": "The recommended treatment or procedure for the patient."
            },
            "medical_specialty": {
                "type": "string",
                "description": "The medical specialty the patient needs."
            },
            "ICD_code": {
                "type": "string",
                "description": "The corresponding International Classification of Diseases (ICD) code for the treatment."
            }
        },
        "required": ["age", "recommended_treatment", "medical_specialty", "ICD_code"]
    }
        }
}]
def process_transcript(transcript):

    messages = [{"role": "system", "content": "Your job is to analyze the medical transcripts and extract the required data in a structured format using the extract_medical_data function"},
               {"role": "user", "content": transcript}]

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        tools=function_definition
    )
    processed_data = response.choices[0].message.tool_calls[0].function.arguments
    processed_data = json.loads(processed_data)
    return pd.Series([processed_data.get('age'), processed_data.get('recommended_treatment'), processed_data.get('medical_specialty'), processed_data.get('ICD_code')], index=['age', 'recommended treatment', 'medical_specialty', 'ICD_code'])
new_columns = df['transcription'].apply(process_transcript)
df_structured = pd.concat([df, new_columns], axis=1)
df_structured.head()