Optimal Illusions: The False Promise of Optimization by Coco Krumme is a fascinating exploration into the complex interplay between human behavior, decision-making, and the algorithms that increasingly govern our lives. In her book, Krumme delves into the nuanced ways in which technology shapes our choices, often in ways we scarcely understand. Through a combination of rigorous research, engaging storytelling, and insightful analysis, Krumme reveals how the pursuit of efficiency and optimization can lead to unintended consequences, challenging our perceptions of free will and autonomy in the digital age.
This Q&A session with Coco Krumme offers a deeper look into the themes and questions raised in Optimal Illusions, providing readers with a unique opportunity to engage with the author’s thought-provoking ideas on how to navigate a world where reality and algorithmically-crafted illusions increasingly blur.
How did Optimal Illusions come together? What made you choose to tackle the notion of optimization?
Initially, I’ve been contemplating the concept of the book for about a decade, which coincides with the completion of my PhD at MIT. This period, around 2012-2013, is when I would say my skepticism towards the unquestioned optimism in optimization and Silicon Valley’s promises was at its peak. Emerging from that phase, I planned my return to where I grew up, the San Francisco Bay Area, to immerse myself in that very world. Having worked there briefly before my time at MIT, I had already experienced the genuine optimism surrounding technology’s potential to address many of our challenges. Furthermore, my PhD work focused on optimization, exploring how models can represent the world to accelerate improvements and enhance efficiency. This led me to ponder the assumptions underlying this optimistic worldview.
As I discuss in the book, my family background is somewhat intertwined with the study of optimization, yet it deviates from the typical narratives of success associated with Silicon Valley. The region’s economic boom and the pervasive way of thinking it represents caught me off guard. I began to question the limitations of this approach, especially given the widespread enthusiasm for it. However, I harbored a more skeptical view, which, over time, seemed to be validated.
I pinpoint 2017-2018 as the period when my disillusionment with technology began to crystallize. At that time, there was a prevailing belief that platforms like Twitter, Facebook, and Google would be the saviors of the world and democracy. However, the ensuing years have seen a growing disenchantment with technology, a sentiment that, while initially felt in isolation, has become more widely acknowledged. This book aims to articulate the disconnection between working within the tech industry and recognizing some of its failures.
How do you define optimization?
That’s an excellent question. Essentially, I adhere to two distinct definitions of optimization. The first is technical: it refers to a program or an algorithm designed to either maximize or minimize a specific value within a given system, based on certain parameters and constraints. This definition, although a simplified version, encapsulates the essence of years of refinement in the field.
The second definition leans more towards a cultural perspective. It embodies the belief that through human ingenuity and engineering, we can enhance the world, striving towards an ideal or the best possible version of it. This belief underpins the conviction that part of our human endeavor is to work towards such an optimal state. Consequently, this viewpoint has significantly influenced various cultural, economic, and political systems, shaping the way we approach and interpret optimization in a broader sense.
The sports world is an example of how people are turning to data as a means of improving efficiency and optimizing their performances. How tightly intertwined is the notion of quantification with optimization?
Indeed, I believe there is a deep connection between the concepts mentioned. Taking sports as an example, while I may be among the few Americans who don’t closely follow the Super Bowl or sports in general, the era of “Moneyball,” as popularized by Michael Lewis, is a case in point. This period, which in my view is still ongoing, showcases how sports analytics have become integral to team strategies. This encompasses decision-making regarding players, calculation of odds, and even the pricing of advertisements.
Then there’s the personal aspect you touched upon, related to the quantified self. This involves tracking one’s fitness and nutrition, which I identify in the book as a foundational element of optimization. It revolves around the atomization of the world into measurable components. After all, there’s a business adage that states, “You can’t manage what you can’t measure.” Essentially, without breaking down aspects of our lives into discrete, measurable units, optimization remains elusive.
The question of causality—whether the desire to optimize in areas such as the quantified self and fitness preceded the technology that enables such measurement, or vice versa—is intriguing. While I lean towards the idea that the impulse to optimize predated these technologies, there’s a compelling argument to be made for the reverse.
In Optimal Illusions, you sort of tell the story of Claude Shannon, the Father of Optimization. Can you discuss his role?
Indeed, while he is featured among many narratives in my work, I wouldn’t categorize him as the “father of optimization.” Rather, he’s more accurately recognized as the father of information theory. His contributions primarily lay in the development of methodologies for discretizing information, a process essential for capturing the inherent noise or entropy in communication systems. This act of discretization, as we’ve discussed, is central to the essence of optimization. It enables us to comprehend a sentence or a stream of information as entities that can be distilled into a single metric. This approach, I believe, marks a significant milestone in the mindset surrounding optimization. It suggests that we can simplify the complexities of our world into one or more measurable metrics, which can then be improved upon over time.
Would I be correct in saying that it’s something to do with efficiency?
Certainly, optimization and efficiency are closely linked. Efficiency, in essence, may seem like a matter of semantics, which might not be profoundly intriguing. However, striving for efficiency is undoubtedly a form of optimization.
Can you give an example of how we’re primed for optimization in the digital world, we’re living in right now.
The concept of optimization is deeply ingrained in our belief system, a point I delve into within my book by exploring its historical roots. This intrinsic value drives us to embrace technologies and digital solutions that promise to enhance efficiency or improve our lives. For example, we often gravitate towards apps that save us time, such as calendar management applications, or those that embody the quantified self concept, which was previously mentioned.
Conversely, our digital environment continuously exposes us to information and processes designed to optimize from another perspective—namely, that of corporations. The algorithms curating our experiences, from the advertisements we see to the news feeds we scroll through, are optimized to increase clicks or generate revenue for businesses. Thus, our interaction with the digital world is profoundly shaped by technologies founded on principles of optimization. This trend extends beyond our digital interactions into the physical realm, affecting everything from supply chains to the electrical grid, indicating how pervasive the influence of optimization has become in both our digital and material worlds.
Would you say that the notion of optimization in that idea, is there sort of a naive utopianism to it? Does it promise a lot more than it can actually deliver in reality and that it does deliver in wherever we are at this point?
Indeed, my view isn’t universally accepted, particularly among the utopian thinkers in Silicon Valley who might strongly disagree with my perspective. However, I believe that the drive towards optimization not only overpromises but also overlooks significant drawbacks. This oversight results from our somewhat uncritical embrace of optimization as an unalloyed good. I’ve detailed these losses in my work and can elaborate on them if desired.
How do you categorize what is lost as a result of optimization?
Indeed, I categorize the losses due to optimization into three main areas. The first is what I term “slack,” referring to the reduction of downtime or opportunities for rest and recovery. Our systems, such as transportation and supply chains, are so finely tuned for efficiency that any disruption—like the container ship incident in the Suez Canal—can have catastrophic consequences. This fragility results from the lack of buffer or slack in these optimized systems.
The second area is the loss of specificity, or a sense of place. This is most evident in the uniformity of American strip malls and cities, where the presence of identical hotel chains, rental car services, and fast-food restaurants streamlines efficiency for travelers but at the cost of local uniqueness and diversity. While convenient, this homogenization diminishes the richness of local culture and diversity, resulting in a bland, uniform landscape that prioritizes efficiency over character.
The third category addresses the loss of integrity between scales, emphasizing the interconnectedness of parts to the whole. This concept, which suggests that small and large elements are inherently linked, is increasingly overlooked in Western thought. Other cultures and belief systems are more attuned to this interconnectivity, recognizing the importance of maintaining a balance across different scales of existence. Our relentless pursuit of optimization, however, has led us to neglect this balance, forgetting the crucial relationship between the micro and macro in our ecosystems and societies.
Is there any way you can opt out without actually just dropping out?
It’s a valid question, and I want to clarify that my book doesn’t suggest that everyone should relocate to a remote island to escape the system. Despite my own less connected lifestyle, I’m not completely disconnected—I still use a smartphone and the internet, and I’m part of the global supply chain, albeit perhaps slightly less so than someone in New York City. In today’s world, fully opting out is challenging and rare. There are minor ways to reduce connectivity, like using a basic phone or limiting smartphone use, but completely disconnecting from all utilities and information channels is neither feasible for most people nor something I desire.
There are aspects of our optimized world that I appreciate, such as the unprecedented connectivity and the wealth of information and culture it brings. However, I believe in finding a balance. While my book avoids giving prescriptive advice, recognizing that everyone’s approach to resisting over-optimization will differ, I do highlight emerging trends that suggest a counter-movement. This nascent counterculture aims to reclaim what we’ve lost: a more restful existence, a sense of place and specificity, and the connection between the micro and macro aspects of life. These themes suggest a path towards balancing the benefits of connectivity with the value of what we risk losing in relentless pursuit of optimization.
How does optimization alter the sense of how we see reality itself?
That’s an insightful question. Indeed, our belief systems significantly shape our perception of reality, often to such an extent that we might not fully grasp their influence. This subtle embedding of beliefs is possibly why the book only hints at these biases rather than providing a definitive guide. My experience, particularly within the Silicon Valley ethos, illustrates one way these biases manifest: the world begins to appear as a collection of problems awaiting solutions. This engineering mindset, once adopted, frames our interactions with the world, leading us to prioritize solving over understanding or experiencing. In essence, the pervasive drive for optimization has profoundly influenced how we interpret and engage with our surroundings.
I understand the point you’re making and recognize the challenge in explaining it. While reading your book, I found myself engaged in an internal dialogue, at times arguing against your ideas. However, upon reflection, I realized this reaction embodies precisely what you discuss: we are ensnared by a mindset that perceives optimization as the sole approach to life. I would catch myself and remind myself that the purpose of reading is to engage with the text openly, rather than to dispute it. Arguing, after all, stems from our current default mode where everything is geared towards efficiency and optimization. This even extends to mindfulness apps, which are ironically designed to help us relax yet end up pressuring us to do so at specific times. It’s a paradox that genuinely frustrates me.
It’s a reaction I often encounter regarding the book. Indeed, I could have approached the subject with a staunchly anti-optimization stance, presenting arguments in a format reminiscent of optimization itself—perhaps a list of reasons outlining the concept’s failures, such as economic losses or diminished quality of life compared to 50 years ago. However, there are two main reasons I chose not to do this.
Firstly, despite what some readers might infer, I do not view optimization as inherently negative. Instead, I see it as a significant phenomenon that warrants examination. My aim was not to denounce optimization but to explore its essence and origins.
Secondly, I wanted to present my analysis from a perspective that diverges from merely proposing an ‘optimized’ critique of optimization. My approach, while acknowledging the pervasive influence of optimization, strives for a nuanced understanding rather than adopting the very framework I’m scrutinizing. This method allows for a broader, more reflective discussion on the subject, avoiding the pitfalls of arguing against optimization using its own language and metrics.
How does ChatGPT fit into the optimization equation?
That indeed opens up an entirely different conversation. Firstly, it’s important to acknowledge that the development of AI technologies has been ongoing for much longer than just the past year. Moreover, my view is that the hype surrounding these technologies is significantly overblown, for various reasons, by both optimists and pessimists alike.
AI technologies are intrinsically linked to optimization, utilizing algorithms designed to optimize outcomes while incorporating a certain degree of calculated randomness. This close relationship with optimization is precisely why I believe they are overhyped. The common assertion that AI will supplant a wide range of human functions, including creative and intuitive capacities, overlooks a critical limitation. The foundational mechanism of AI, rooted in optimization, inherently lacks the ability to replicate the nuanced and diverse capabilities that humans excel at. This distinction underscores why AI, despite its advancements, cannot fully replace human creativity and ingenuity.
Sign up for the Daily Dose Newsletter and get the morning’s best science news from around the web delivered straight to your inbox? It’s easy like Sunday morning.
Leave a Reply