How to Actually Use Foundation Models: From Classification to Agents Learning Lab

Foundation Models—including Vision-Language Models and Large Language Models—are rapidly reshaping what is possible in medical imaging. Yet many imaging informaticists, researchers, and developers find a disconnect between the promise described in the literature and what can realistically be implemented in clinical and research environments. Questions around practical use, efficient adaptation, and system-level integration remain major barriers. This learning lab is designed to close that gap by focusing on applied, hands-on use of Foundation Models in real-world imaging informatics workflows. 

This highly interactive, 120-minute learning lab emphasizes practical takeaways participants can apply immediately. Attendees work directly in pre-configured, cloud-based Jupyter environments and progress through three structured modules:  

• Zero-Shot Classification with Foundation Models
  Attendees learn how to apply multimodal Foundation Models using zero-shot inference, enabling medical image classification without task-specific training. Participants gain practical experience with prompt design and understand when zero-shot approaches are effective—and when they are not. 
• Efficient Adaptation Under Resource Constraints
  When zero-shot performance is insufficient, participants learn how to adapt large models efficiently. Attendees implement lightweight adapter techniques to tailor Foundation Models to specific medical imaging tasks, gaining insight into how meaningful customization can be achieved without large computational investments. 
• From Models to Agents: Orchestrating Workflows
  Participants move beyond single-model use cases to explore agentic AI. Attendees learn how to orchestrate Foundation Models, ground outputs in domain-specific knowledge, and build a simple agent that executes a coherent imaging informatics workflow using models developed earlier in the lab.

The lab follows a learn-by-doing format designed to reinforce understanding through action. Participants actively code along with guided demonstrations, complete targeted challenges that require modifying and extending example code, and receive real-time troubleshooting support. Short knowledge checks help confirm understanding of key concepts. By the end of the session, attendees leave with practical skills, reusable patterns, and a clear framework for applying Foundation Models and agentic workflows within their own imaging informatics environments. 

Objectives 

• Identify appropriate use cases for Foundation Models in medical imaging, including when zero-shot approaches are sufficient and when model adaptation is required. 
• Implement efficient adaptation techniques to tailor Foundation Models for specific medical imaging tasks using resource-conscious methods. 
• Use Foundation Models within a simple agentic workflow to integrate models, knowledge sources, and tools into a coherent imaging informatics application. 
• Build a zero-shot image classification workflow using a multimodal Foundation Model within a cloud-based Jupyter environment. 
• Modify prompts and adapter parameters to improve model performance for a targeted medical imaging task. 
• Demonstrate an agentic workflow by orchestrating Foundation Models and knowledge sources to execute a simple imaging informatics use case.