can someone who understands this stuff explain to me why crowdsourced ml training is not a thing? afaiu when training you _already_ have to use many gpus across many machines potentially in different data centers. would the same techniques not work for distributing the training process across many "peer" machines?
Apr 29, 2026, 10:08 PM@nicebyte "Figure 5 shows the overall training pipeline of GLM-5. Our Base Model training began with a massive 27 trillion token corpus"
With average 4 bytes per token and counting only the raw data that's 108 TB, just for the base training corpus.
The full F16 model can be 1.5 TB (GLM-5), I would guess the layers are split between multiple GPUs. A single data center GPU will have e.g. 80, 144, 288 GB of VRAM. The speed of such GPU is basically what you get with a gaming GPU though.
Apr 29, 2026, 10:12 PM@nicebyte To train the model, you go through all the training data you have, multiple times. For each sample you do backpropagation through the entire model.
The actual computation speed is not the problem. But you need to update the whole model, and it's on multiple GPUs, and uh-oh, there's actually a lot of data to transfer between in and out to these GPUs.
The data center GPUs are very close physically and have very efficient memory data interconnects.
Apr 29, 2026, 10:19 PM@nicebyte The typical crowdsourced computation like SETI@Home, or the protein folding thing is basically: get a small chunk of data, then crunch heavily on it for a long time (hours, days), then send the results.
Machine learning is the opposite, the main bottleneck is raw data throughput.
It's also wrong to assume that training itself is the primary problem. It isn't. Dataset preparation is. You spend the majority of your time (like 90+%) combing through the data.