Clinical Text Summarization Using Large Language Models
Replicated a research study on hospital course summarization using Clinical-T5, LLaMA2-13B, and GPT-4 with the MIMIC-IV dataset, implementing advanced fine-tuning techniques and comprehensive evaluation.
Project Overview
Successfully replicated a cutting-edge research paper on hospital course summarization using Large Language Models (LLMs) with the MIMIC-IV dataset. Implemented a comprehensive clinical summarization pipeline exploring Clinical-T5, LLaMA2-13B, and GPT-4 for generating accurate Brief Hospital Course (BHC) summaries. The project involved rigorous data preprocessing, including regex-based BHC extraction, clinical text normalization, and dataset segmentation into context bins (short ≤1024, medium 1025-2048, long >2048 tokens). Utilized advanced technical optimizations including 4-bit quantization, Unsloth framework integration, and QLoRA fine-tuning achieving significant performance improvements with LLaMA2-13B reaching 0.683 BERT F1-Score.
Key Features
- ✓Clinical text summarization pipeline
- ✓Multi-model comparison (Clinical-T5, LLaMA2-13B, GPT-4)
- ✓MIMIC-IV dataset processing and preprocessing
- ✓Regex-based Brief Hospital Course extraction
- ✓Clinical text normalization and validation
- ✓Context-based dataset binning (short/medium/long)
- ✓QLoRA fine-tuning implementation
- ✓4-bit quantization for memory efficiency
- ✓Unsloth framework integration
- ✓Comprehensive hyperparameter tuning
- ✓BERT Score evaluation metrics
- ✓Zero-shot and prefix prompting strategies
- ✓HPC infrastructure deployment
- ✓WandB integration for monitoring
- ✓Memory optimization techniques
- ✓Production-ready clinical NLP pipeline
Technical Challenges
- ⚡Handling confidential healthcare data with proper certification
- ⚡Managing large-scale clinical text preprocessing
- ⚡Optimizing memory usage for 13B parameter models
- ⚡Balancing performance across different context lengths
- ⚡Implementing efficient fine-tuning strategies
- ⚡Managing 8+ hours of intensive computation
- ⚡Ensuring clinical relevance and accuracy
- ⚡Deploying on HPC infrastructure with SLURM
Technologies Used
Project Info
Screenshots
