The largest AI language models are being trained with object storage, not file storage.

HPC expert Glenn Lockwood says that the largest AI language models are being trained with object storage, not file storage.

AI model training needs unstructured data. Most unstructured data is stored in files. Ergo, large language model (LLM) training needs access to file data and parallel file systems, as used in high-performance computing (HPC), to deliver file data faster to processors than serial file systems. Therefore, parallel file systems are needed for LLM training. Actually, no, says Lockwood, because the characteristics of LLM model training phases actually favor object storage more than parallel file systems.

Lockwood is an AI Infrastructure Architect at Microsoft who has worked on one of the world’s largest supercomputers. He writes: “I guess supercomputers and parallel file systems are like peas and carrots in so many people’s minds that the idea of being able to run a massive parallel compute job without a massive parallel file system is so unintuitive that it is unbelievable.”