Massive Computing and Data

     We are moving at a breakneck pace into an era defined by unprecedented amounts of computing power and an explosion of data. I believe this trend will reshape all aspects of both our personal and professional lives. Another explosion is happening with the Internet of Things (IoT). Devices range from smart home systems to wearable health monitors. It is not unusual for homeowners to have more than 100 smart devices in their home: door locks, ceiling fan controls, wall keypads, motion sensors, and smart lights. These have created interconnected ecosystems which generate huge amounts of real-time data streams. Some analysts put the revenue from IoT as heading to a trillion dollars. This data deluge is amplified by advancements in AI and quantum computing, enabling organizations to analyze patterns of data at previously unimaginable scales. For example is Spotify.

Spotify has reached 675 million monthly active users as of Q4 2024. The music provider employs AI-driven analytics to curate its hyper-personalized music experience. The use collaborative filtering to analyze 500+ million user interactions (plays, skips, playlist adds) to group listeners with similar tastes and recommend tracks. They create Discover Weekly, which generates 30-song playlists updated weekly using very sophisticated algorithms. They use a specialized AI technology called convolutional neural networks (CNNs) to process raw audio data. They examine beats per minute, musical key and timbre from 100+ million music tracks to classify songs into 1,500+ micro-genres. There is much more of what they do. While we are enjoying Spotify music, there is a lot of computing going on. Despite some privacy concerns, Spotify’s blend of analytics and cultural insights keeps 68% of users engaged for 2.5+ hours daily. Its data-centric approach remains central to maintaining dominance in the $80B+ streaming industry.

In a professional setting, the “big data” is transforming decision-making through predictive supply chain optimization, automated HR processes, and hyper-targeted marketing strategies. Quantum computing breakthroughs promise to solve complex problems 100 million times faster than classical computers. Various AI tools will soon make real-time data analysis as commonplace as electricity, fundamentally altering how we work, enjoy, learn, and interact. Where does all the computing power come from to make all this possible. AI data centers are not the only source of the growth in computing power.

The World Community Grid (WCG) was founded by IBM in 2004. It was an ambitious project to harness the world’s spare computing power for scientific research to benefit humanity. I was impressed with the vision and was an early volunteer.

I have used WideglideJRP as a handle for many years. Wideglide was the name of a Harley-Davison bike I had at the time.

What is the World Community Grid (WCG) and how does it work. WCG operates as a global volunteer computing platform which harnesses unused processing power from personal computers. Modern PCs use a lot of processing power when playing games or editing videos, but for daily web browsing or office applications, the average PC only consumes 2-10% of its power. That is when the PC is doing something. PCs remain inactive 76% of the day on average, leaving 18 hours daily of untapped processing potential. All the world’s idle processing capacity could be connected over the Internet and act as a giant supercomputer. That is exactly what the WCG does. Many of us have old PCs which we store in the basement or in a closet. If instead, these idle PCs join the WCG, they can contribute to scientific research. I have had up to six computers in the WCG at one time. During the 20 years I have contributed the equivalent of 58 years of computing time to the WCG.

The way it works is volunteers install WCG lightweight software which runs in the background when the PC is not busy. During that idle time, servers at WCG send the PC small computational tasks called workunits. These tasks, such as protein structure predictions or disease treatment simulations, are fragments of larger research projects broken into parallelizable units. For example, cancer research which would take 162 years on conventional systems was completed in under two years using WCG’s distributed network of volunteer contributed processing power. The research projects span health (e.g., HIV, cancer, COVID-19), sustainability, and climate science, with WCG’s cumulative power exceeding that of supercomputers processing 1.3 million environmental simulations in one year instead of 90. WCG was managed initially by IBM and now by the Krembil Research Institute in Toronto, Canada. Since 2004, its 2 million+ devices from 600,000 volunteers have donated over 300,000 years of computing time, enabling peer-reviewed breakthroughs like Zika virus inhibitors and clean energy material discoveries.

After its launch, WCG quickly gained support from notable organizations, including National Institutes of Health, Mayo Clinic, Oxford University, and many other distinguished research organizations. The initial goals of World Community Grid were to create the world’s largest volunteer computing platform for humanitarian research, tackle society’s challenges by harnessing unused processing power from personal computers, enable scientists to compress years of computations into months or weeks, and advance research in complex global areas. What about AI? Will it do away with needing WCG? I will explain the differences and challenges in a future blog post.

More about healthcare, science, and technology at johnpatrick.com.