Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge

M Maximilian Zenk U Ujjwal Baid S Sarthak Pati A Akis Linardos B Brandon Edwards M Micah Sheller P Patrick Foley A Alejandro Aristizabal D David Zimmerer A Alexey Gruzdev J Jason Martin R Russell T. Shinohara (Center for AI and Data Science for Integrated Diagnostics, Perelman School of Medicine, University of Pennsylvania) A Annika Reinke F Fabian Isensee S Santhosh Parampottupadam K Kaushal Parekh R Ralf Floca H Hasan Kassem B Bhakti Baheti S Siddhesh Thakur V Verena Chung K Kaisar Kushibar K Karim Lekadir M Meirui Jiang Y Youtan Yin H Hongzheng Yang Q Quande Liu C Cheng Chen Q Qi Dou P Pheng-Ann Heng X Xiaofan Zhang S Shaoting Zhang (Department of Chemistry, Chemical Theory Center, University of Minnesota) M Muhammad Irfan Khan M Mohammad Ayyaz Azeem M Mojtaba Jafaritadi E Esa Alhoniemi E Elina Kontio S Suleiman A. Khan L Leon Mächler I Ivan Ezhov F Florian Kofler S Suprosanna Shit J Johannes C. Paetzold T Timo Loehr B Benedikt Wiestler H Himashi Peiris K Kamlesh Pawar S Shenjun Zhong Z Zhaolin Chen M Munawar Hayat G Gary Egan M Mehrtash Harandi E Ece Isik Polat G Gorkem Polat A Altan Kocyigit A Alptekin Temizel A Anup Tuladhar L Lakshay Tyagi R Raissa Souza N Nils D. Forkert P Pauline Mouches M Matthias Wilms V Vishruth Shambhat A Akansh Maurya S Shubham Subhas Danannavar R Rohit Kalla V Vikas Kumar Anand G Ganapathy Krishnamurthi S Sahil Nalawade C Chandan Ganesh B Ben Wagner D Divya Reddy Y Yudhajit Das F Fang F. Yu B Baowei Fei A Ananth J. Madhuranthakam J Joseph Maldjian G Gaurav Singh (Institute for Stem Cell Science and Regenerative Medicine) J Jianxun Ren W Wei Zhang N Ning An Q Qingyu Hu Y Youjia Zhang Y Ying Zhou V Vasilis Siomos G Giacomo Tarroni J Jonathan Passerrat-Palmbach A Ambrish Rawat G Giulio Zizzo S Swanand Ravindra Kadhe J Jonathan P. Epperlein S Stefano Braghin Y Yuan Wang R Renuga Kanagavelu Q Qingsong Wei Y Yechao Yang Y Yong Liu K Krzysztof Kotowski S Szymon Adamski B Bartosz Machura W Wojciech Malara L Lukasz Zarudzki J Jakub Nalepa Y Yaying Shi H Hongjian Gao S Salman Avestimehr Y Yonghong Yan (School of Chemistry and Chemical Engineering, Key Laboratory of Magnetic Molecules and Magnetic Information Materials, The Ministry of Education, Shanxi Normal University 2 , No. 339, Taiyu Road, Taiyuan, Shanxi 030031,) A Agus S. Akbar E Ekaterina Kondrateva H Hua Yang (State Key Laboratory of Natural Medicines, School of Pharmacy, China Pharmaceutical University, 24 Tong Jia Xiang, Nanjing 210009, China) Z Zhaopei Li H Hung-Yu Wu J Johannes Roth C Camillo Saueressig A Alexandre Milesi Q Quoc D. Nguyen N Nathan J. Gruenhagen T Tsung-Ming Huang (Department of Mathematics, National Taiwan Normal University) J Jun Ma H Har Shwinder H. Singh N Nai-Yu Pan D Dingwen Zhang R Ramy A. Zeineldin M Michal Futrega Y Yading Yuan G Gian Marco Conte X Xue Feng Q Quan D. Pham Y Yong Xia (School of Medical Engineering) Z Zhifan Jiang H Huan Minh Luu M Mariia Dobko A Alexandre Carré B Bair Tuchinov H Hassan Mohy-ud-Din S Saruar Alam A Anup Singh N Nameeta Shah W Weichung Wang C Chiharu Sako M Michel Bilello S Satyam Ghodasara S Suyash Mohan C Christos Davatzikos E Evan Calabrese J Jeffrey Rudie J Javier Villanueva-Meyer S Soonmee Cha C Christopher Hess J John Mongan M Madhura Ingalhalikar M Manali Jadhav U Umang Pandey J Jitender Saini R Raymond Y. Huang K Ken Chang M Minh-Son To S Sargam Bhardwaj C Chee Chong M Marc Agzarian M Michal Kozubek F Filip Lux J Jan Michálek P Petr Matula M Miloš Ker^kovský T Tereza Kopr^ivová M Marek Dostál V Vaclav Vybihal M Marco C. Pinho J James Holcomb M Marie Metz R Rajan Jain (Departments of Medicine and Cell and Developmental Biology, Penn Cardiovascular Institute, Penn Epigenetics Institute, Perelman School of Medicine, University of Pennsylvania) M Matthew D. Lee Y Yvonne W. Lui P Pallavi Tiwari R Ruchika Verma R Rohan Bareja I Ipsa Yadav J Jonathan Chen N Neeraj Kumar Y Yuriy Gusev K Krithika Bhuvaneshwar A Anousheh Sayah C Camelia Bencheqroun A Anas Belouali S Subha Madhavan R Rivka R. Colen A Aikaterini Kotrotsou P Philipp Vollmuth G Gianluca Brugnara C Chandrakanth J. Preetha F Felix Sahm M Martin Bendszus W Wolfgang Wick A Abhishek Mahajan C Carmen Balana J Jaume Capellades J Josep Puig Y Yoon Seong Choi S Seung-Koo Lee J Jong Hee Chang S Sung Soo Ahn H Hassan F. Shaykh A Alejandro Herrera-Trujillo M Maria Trujillo W William Escobar A Ana Abello J Jose Bernal J Jhon Gómez P Pamela LaMontagne D Daniel S. Marcus M Mikhail Milchenko A Arash Nazeri B Bennett Landman K Karthik Ramadass K Kaiwen Xu S Silky Chotai L Lola B. Chambless A Akshitkumar Mistry R Reid C. Thompson A Ashok Srinivasan J J. Rajiv Bapuraj A Arvind Rao (Michigan Center for Translational Pathology, University of Michigan) N Nicholas Wang O Ota Yoshiaki T Toshio Moritani S Sevcan Turk J Joonsang Lee S Snehal Prabhudesai J John Garrett M Matthew Larson R Robert Jeraj H Hongwei Li (Peking University , , ,) T Tobias Weiss M Michael Weller A Andrea Bink B Bertrand Pouymayou S Sonam Sharma T Tzu-Chi Tseng S Saba Adabi A Alexandre Xavier Falcão S Samuel B. Martins B Bernardo C. A. Teixeira F Flávia Sprenger D David Menotti D Diego R. Lucio S Simone P. Niclou O Olivier Keunen A Ann-Christin Hau E Enrique Pelaez H Heydy Franco-Maldonado F Francis Loayza S Sebastian Quevedo R Richard McKinley J Johannes Slotboom P Piotr Radojewski R Raphael Meier R Roland Wiest J Johannes Trenkler J Josef Pichler G Georg Necker A Andreas Haunschmidt S Stephan Meckel P Pamela Guevara E Esteban Torche C Cristobal Mendoza F Franco Vera E Elvis Ríos E Eduardo Lopez S Sergio A. Velastin J Joseph Choi S Stephen Baek Y Yusung Kim H Heba Ismael B Bryan Allen J John M. Buatti P Peter Zampakis V Vasileios Panagiotopoulos P Panagiotis Tsiganos S Sotiris Alexiou I Ilias Haliassos E Evangelia I. Zacharaki K Konstantinos Moustakas C Christina Kalogeropoulou D Dimitrios M. Kardamakis B Bing Luo L Laila M. Poisson N Ning Wen (National Engineering Research Center for Colloidal Materials, School of Chemistry and Chemical Engineering) M Martin Vallières M Mahdi Ait Lhaj Loutfi D David Fortin M Martin Lepage F Fanny Morón J Jacob Mandel G Gaurav Shukla S Spencer Liem G Gregory S. Alexandre J Joseph Lombardo J Joshua D. Palmer A Adam E. Flanders A Adam P. Dicker G Godwin Ogbole D Dotun Oyekunle O Olubunmi Odafe-Oyibotha B Babatunde Osobu M Mustapha Shu’aibu Hikima M Mayowa Soneye F Farouk Dako A Adeleye Dorcas D Derrick Murcia E Eric Fu R Rourke Haas J John A. Thompson D David Ryan Ormond S Stuart Currie K Kavi Fatania R Russell Frood A Amber L. Simpson J Jacob J. Peoples R Ricky Hu D Danielle Cutler F Fabio Y. Moraes A Anh Tran M Mohammad Hamghalam M Michael A. Boss J James Gimpel D Deepak Kattil Veettil K Kendall Schmidt L Lisa Cimino C Cynthia Price B Brian Bialecki S Sailaja Marella C Charles Apgar A Andras Jakab M Marc-André Weber E Errol Colak J Jens Kleesiek J John B. Freymann J Justin S. Kirby L Lena Maier-Hein J Jake Albrecht P Peter Mattson A Alexandros Karargyris P Prashant Shah B Bjoern Menze K Klaus Maier-Hein S Spyridon Bakas

Abstract

Abstract Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.

Article Details

Volume / Issue Vol. 16, Issue 1
Published July 08, 2025
ISSN 2041-1723
Publisher Nature Portfolio

Journal Info

Nature Communications

Nature Portfolio

ISSN: 2041-1723 Open Access Life Sciences

Authors (347)

M

Maximilian Zenk

U

Ujjwal Baid

S

Sarthak Pati

A

Akis Linardos

B

Brandon Edwards

M

Micah Sheller

P

Patrick Foley

A

Alejandro Aristizabal

D

David Zimmerer

A

Alexey Gruzdev

J

Jason Martin

R

Russell T. Shinohara

Center for AI and Data Science for Integrated Diagnostics, Perelman School of Medicine, University of Pennsylvania

A

Annika Reinke

F

Fabian Isensee

S

Santhosh Parampottupadam

K

Kaushal Parekh

R

Ralf Floca

H

Hasan Kassem

B

Bhakti Baheti

S

Siddhesh Thakur

V

Verena Chung

K

Kaisar Kushibar

K

Karim Lekadir

M

Meirui Jiang

Y

Youtan Yin

H

Hongzheng Yang

Q

Quande Liu

C

Cheng Chen

Q

Qi Dou

P

Pheng-Ann Heng

X

Xiaofan Zhang

S

Shaoting Zhang

Department of Chemistry, Chemical Theory Center, University of Minnesota

M

Muhammad Irfan Khan

M

Mohammad Ayyaz Azeem

M

Mojtaba Jafaritadi

E

Esa Alhoniemi

E

Elina Kontio

S

Suleiman A. Khan

L

Leon Mächler

I

Ivan Ezhov

F

Florian Kofler

S

Suprosanna Shit

J

Johannes C. Paetzold

T

Timo Loehr

B

Benedikt Wiestler

H

Himashi Peiris

K

Kamlesh Pawar

S

Shenjun Zhong

Z

Zhaolin Chen

M

Munawar Hayat

G

Gary Egan

M

Mehrtash Harandi

E

Ece Isik Polat

G

Gorkem Polat

A

Altan Kocyigit

A

Alptekin Temizel

A

Anup Tuladhar

L

Lakshay Tyagi

R

Raissa Souza

N

Nils D. Forkert

P

Pauline Mouches

M

Matthias Wilms

V

Vishruth Shambhat

A

Akansh Maurya

S

Shubham Subhas Danannavar

R

Rohit Kalla

V

Vikas Kumar Anand

G

Ganapathy Krishnamurthi

S

Sahil Nalawade

C

Chandan Ganesh

B

Ben Wagner

D

Divya Reddy

Y

Yudhajit Das

F

Fang F. Yu

B

Baowei Fei

A

Ananth J. Madhuranthakam

J

Joseph Maldjian

G

Gaurav Singh

Institute for Stem Cell Science and Regenerative Medicine

J

Jianxun Ren

W

Wei Zhang

N

Ning An

Q

Qingyu Hu

Y

Youjia Zhang

Y

Ying Zhou

V

Vasilis Siomos

G

Giacomo Tarroni

J

Jonathan Passerrat-Palmbach

A

Ambrish Rawat

G

Giulio Zizzo

S

Swanand Ravindra Kadhe

J

Jonathan P. Epperlein

S

Stefano Braghin

Y

Yuan Wang

R

Renuga Kanagavelu

Q

Qingsong Wei

Y

Yechao Yang

Y

Yong Liu

K

Krzysztof Kotowski

S

Szymon Adamski

B

Bartosz Machura

W

Wojciech Malara

L

Lukasz Zarudzki

J

Jakub Nalepa

Y

Yaying Shi

H

Hongjian Gao

S

Salman Avestimehr

Y

Yonghong Yan

School of Chemistry and Chemical Engineering, Key Laboratory of Magnetic Molecules and Magnetic Information Materials, The Ministry of Education, Shanxi Normal University 2 , No. 339, Taiyu Road, Taiyuan, Shanxi 030031,

A

Agus S. Akbar

E

Ekaterina Kondrateva

H

Hua Yang

State Key Laboratory of Natural Medicines, School of Pharmacy, China Pharmaceutical University, 24 Tong Jia Xiang, Nanjing 210009, China

Z

Zhaopei Li

H

Hung-Yu Wu

J

Johannes Roth

C

Camillo Saueressig

A

Alexandre Milesi

Q

Quoc D. Nguyen

N

Nathan J. Gruenhagen

T

Tsung-Ming Huang

Department of Mathematics, National Taiwan Normal University

J

Jun Ma

H

Har Shwinder H. Singh

N

Nai-Yu Pan

D

Dingwen Zhang

R

Ramy A. Zeineldin

M

Michal Futrega

Y

Yading Yuan

G

Gian Marco Conte

X

Xue Feng

Q

Quan D. Pham

Y

Yong Xia

School of Medical Engineering

Z

Zhifan Jiang

H

Huan Minh Luu

M

Mariia Dobko

A

Alexandre Carré

B

Bair Tuchinov

H

Hassan Mohy-ud-Din

S

Saruar Alam

A

Anup Singh

N

Nameeta Shah

W

Weichung Wang

C

Chiharu Sako

M

Michel Bilello

S

Satyam Ghodasara

S

Suyash Mohan

C

Christos Davatzikos

E

Evan Calabrese

J

Jeffrey Rudie

J

Javier Villanueva-Meyer

S

Soonmee Cha

C

Christopher Hess

J

John Mongan

M

Madhura Ingalhalikar

M

Manali Jadhav

U

Umang Pandey

J

Jitender Saini

R

Raymond Y. Huang

K

Ken Chang

M

Minh-Son To

S

Sargam Bhardwaj

C

Chee Chong

M

Marc Agzarian

M

Michal Kozubek

F

Filip Lux

J

Jan Michálek

P

Petr Matula

M

Miloš Ker^kovský

T

Tereza Kopr^ivová

M

Marek Dostál

V

Vaclav Vybihal

M

Marco C. Pinho

J

James Holcomb

M

Marie Metz

R

Rajan Jain

Departments of Medicine and Cell and Developmental Biology, Penn Cardiovascular Institute, Penn Epigenetics Institute, Perelman School of Medicine, University of Pennsylvania

M

Matthew D. Lee

Y

Yvonne W. Lui

P

Pallavi Tiwari

R

Ruchika Verma

R

Rohan Bareja

I

Ipsa Yadav

J

Jonathan Chen

N

Neeraj Kumar

Y

Yuriy Gusev

K

Krithika Bhuvaneshwar

A

Anousheh Sayah

C

Camelia Bencheqroun

A

Anas Belouali

S

Subha Madhavan

R

Rivka R. Colen

A

Aikaterini Kotrotsou

P

Philipp Vollmuth

G

Gianluca Brugnara

C

Chandrakanth J. Preetha

F

Felix Sahm

M

Martin Bendszus

W

Wolfgang Wick

A

Abhishek Mahajan

C

Carmen Balana

J

Jaume Capellades

J

Josep Puig

Y

Yoon Seong Choi

S

Seung-Koo Lee

J

Jong Hee Chang

S

Sung Soo Ahn

H

Hassan F. Shaykh

A

Alejandro Herrera-Trujillo

M

Maria Trujillo

W

William Escobar

A

Ana Abello

J

Jose Bernal

J

Jhon Gómez

P

Pamela LaMontagne

D

Daniel S. Marcus

M

Mikhail Milchenko

A

Arash Nazeri

B

Bennett Landman

K

Karthik Ramadass

K

Kaiwen Xu

S

Silky Chotai

L

Lola B. Chambless

A

Akshitkumar Mistry

R

Reid C. Thompson

A

Ashok Srinivasan

J

J. Rajiv Bapuraj

A

Arvind Rao

Michigan Center for Translational Pathology, University of Michigan

N

Nicholas Wang

O

Ota Yoshiaki

T

Toshio Moritani

S

Sevcan Turk

J

Joonsang Lee

S

Snehal Prabhudesai

J

John Garrett

M

Matthew Larson

R

Robert Jeraj

H

Hongwei Li

Peking University , , ,

T

Tobias Weiss

M

Michael Weller

A

Andrea Bink

B

Bertrand Pouymayou

S

Sonam Sharma

T

Tzu-Chi Tseng

S

Saba Adabi

A

Alexandre Xavier Falcão

S

Samuel B. Martins

B

Bernardo C. A. Teixeira

F

Flávia Sprenger

D

David Menotti

D

Diego R. Lucio

S

Simone P. Niclou

O

Olivier Keunen

A

Ann-Christin Hau

E

Enrique Pelaez

H

Heydy Franco-Maldonado

F

Francis Loayza

S

Sebastian Quevedo

R

Richard McKinley

J

Johannes Slotboom

P

Piotr Radojewski

R

Raphael Meier

R

Roland Wiest

J

Johannes Trenkler

J

Josef Pichler

G

Georg Necker

A

Andreas Haunschmidt

S

Stephan Meckel

P

Pamela Guevara

E

Esteban Torche

C

Cristobal Mendoza

F

Franco Vera

E

Elvis Ríos

E

Eduardo Lopez

S

Sergio A. Velastin

J

Joseph Choi

S

Stephen Baek

Y

Yusung Kim

H

Heba Ismael

B

Bryan Allen

J

John M. Buatti

P

Peter Zampakis

V

Vasileios Panagiotopoulos

P

Panagiotis Tsiganos

S

Sotiris Alexiou

I

Ilias Haliassos

E

Evangelia I. Zacharaki

K

Konstantinos Moustakas

C

Christina Kalogeropoulou

D

Dimitrios M. Kardamakis

B

Bing Luo

L

Laila M. Poisson

N

Ning Wen

National Engineering Research Center for Colloidal Materials, School of Chemistry and Chemical Engineering

M

Martin Vallières

M

Mahdi Ait Lhaj Loutfi

D

David Fortin

M

Martin Lepage

F

Fanny Morón

J

Jacob Mandel

G

Gaurav Shukla

S

Spencer Liem

G

Gregory S. Alexandre

J

Joseph Lombardo

J

Joshua D. Palmer

A

Adam E. Flanders

A

Adam P. Dicker

G

Godwin Ogbole

D

Dotun Oyekunle

O

Olubunmi Odafe-Oyibotha

B

Babatunde Osobu

M

Mustapha Shu’aibu Hikima

M

Mayowa Soneye

F

Farouk Dako

A

Adeleye Dorcas

D

Derrick Murcia

E

Eric Fu

R

Rourke Haas

J

John A. Thompson

D

David Ryan Ormond

S

Stuart Currie

K

Kavi Fatania

R

Russell Frood

A

Amber L. Simpson

J

Jacob J. Peoples

R

Ricky Hu

D

Danielle Cutler

F

Fabio Y. Moraes

A

Anh Tran

M

Mohammad Hamghalam

M

Michael A. Boss

J

James Gimpel

D

Deepak Kattil Veettil

K

Kendall Schmidt

L

Lisa Cimino

C

Cynthia Price

B

Brian Bialecki

S

Sailaja Marella

C

Charles Apgar

A

Andras Jakab

M

Marc-André Weber

E

Errol Colak

J

Jens Kleesiek

J

John B. Freymann

J

Justin S. Kirby

L

Lena Maier-Hein

J

Jake Albrecht

P

Peter Mattson

A

Alexandros Karargyris

P

Prashant Shah

B

Bjoern Menze

K

Klaus Maier-Hein

S

Spyridon Bakas