(Note: This post will be continuously updated as I learn more about the world.)

This project started as a final essay submission for my first semester in school, under the Module “Big Questions: Happiness & Suffering”. I am still considered somewhat young and have much more to learn, so my opinions on this matter will likely change over time. Re-writing this in first-person, not to be taken as an objective paper.

Introduction #

In recent studies on ‘hot streaks’, periods of peak performance, of over 20,000 artists, film directors, and scientists, researchers found that ‘hot streaks’ seem to occur randomly within a career (Liu et al., 2017).

In my own life, in hindsight, most of the best things that happened to me were unplanned. Heck, I never even knew they existed.

We may find these random events (good or bad) changing the entire trajectory of our lives, leading us to states of greater happiness or success. We rationalise this in different ways. Some of us attribute its non-deterministic nature to God and higher beings, while some call it ‘luck’ or ‘serendipity’. However, that explanation might be not be good enough for some of us. What if there was a more quantitative way to understand how randomness contributes to human flourishing?

From Faith to Formulas #

The field of computer science offers a promising model for explanation. Several authors proposed that the way humans learn and make life decisions, such as careers, is an iterative optimisation problem that is similar to the Hill-Climbing algorithm (Career as a Gradient Descent Problem, 2023; Climbing the Wrong Hill, 2009; Hill Climbing, 2021; Langley et al., 1987; Qu, 2019; Tang, 2021).

Imagine standing in a foggy mountain range, trying to reach the highest peak. Since it’s foggy, we can only see our immediate surroundings. The rational strategy would be to keep walking uphill until we can’t go any higher. This mirrors how we typically make decisions too. With limited cognitive resources and information, we rely on heuristics to make these decisions. We evaluate our next available options and make incremental improvements based on what we can observe over time, much like the Hill-Climbing algorithm.

Limits of Simple Hill-Climbing #

However, this approach to decision-making has a crucial limitation – the problem of local optima.

Consider how we approach career decisions. We may look at our current situation and nearby options - perhaps a promotion at our company or a similar role at another firm. We evaluate these options based on factors like salary, work-life balance, and personal interest. This is like climbing a hill by always taking steps upward. But just as a hiker might reach a small peak while missing a much higher mountain hidden in the fog, humans can become trapped in locally optimal but globally suboptimal situations.

We might settle into a comfortable job while missing more fulfilling opportunities or maintain habits that provide immediate satisfaction while preventing greater long-term well-being. This is where randomness plays its crucial role, and the stochastic hill-climbing algorithm - an upgrade to simple hill-climbing - provides us with insights into how randomness can help us better flourish in life.

Escaping from Local Peaks #

To solve this problem, computer scientists suggest that we may need to temporarily worsen our solution if we want to continue searching for improvements, and randomness offers a few strategies for doing so: “Jitter”, Random-Restart Hill-Climbing, and the Metropolis Algorithm (Christian & Griffiths, 2016).

“Jitter” suggests for us to make random small changes to our lives when we are stuck, even if they temporarily worsen our level of well-being, and then continue hill-climbing. Random-Restart Hill-Climbing, as its name suggests, is to completely scramble our combinations of choices when we reach a local optima and restart again. Lastly, the Metropolis Algorithm suggests applying a small amount of randomness to every decision we make. This probability of making a random decision is controlled by a variable called temperature.

Empirical observations (maybe) #

Interestingly, these stochastic hill-climbing strategies can be somewhat observed empirically and affect well-being on multiple levels: individually and evolutionary.

Individual Well-being #

Random-Restart. Research on over 20,000 artists, directors, and scientists reveals that “hot streaks” - periods of exceptional performance - follow patterns consistent with stochastic optimisation. Peak performance emerges from a specific sequence: extensive exploration across diverse fields followed by focused exploitation of promising areas (Liu et al., 2021). This mirrors Random Restart Hill-Climbing, where individuals sample multiple “hills” before concentrating on the most promising peak.

Metropolis Algorithm. The Metropolis Algorithm parallels the entropy patterns surrounding these hot streaks too. Before peak periods, individuals exhibit high unpredictability in their work, which decreases significantly during the streak as they focus narrowly - resembling the temperature reduction in simulated annealing optimisation (Wegener, 2005).

Evolutionary Well-being #

Jitters. Stochastic hill-climbing finds parallel support in biological evolution through adaptive landscapes. Organisms navigate fitness landscapes where height represents survival advantage, climbing via natural selection driven by genetic recombination and random mutations - analogous to algorithmic “jitters” (McGhee, 2006). When trapped at local fitness peaks, organisms may also escape through peak jumping or environmental changes that reshape the landscape itself.

It does seem that this pattern - in biological evolution to human careers - suggests a fundamental need for stochastic optimisation in complex, uncertain environments.

We can’t jump alone! #

If exploration through randomness is beneficial, why don’t we simply choose to be more random in our decisions? The answer might lie in behavioral barriers that prevent self-directed exploration.

Temporal discounting - our tendency to overvalue immediate rewards compared to future benefits - creates a systematic bias against voluntary exploration (Reed & Luiselli, 2011). Research shows this phenomenon correlates significantly with reduced directed exploration (Sadeghiyeh et al., 2020). Even when we intellectually understand exploration’s value, we struggle to voluntarily leave comfortable situations. Chris Dixon’s example illustrates this: his acquaintance knows joining a startup aligns with his dreams yet remains trapped in an unfulfilling finance job due to immediate security concerns (Climbing the Wrong Hill, 2009).

This explains why externally imposed randomness often proves crucial for escaping local peaks. Environmental disruptions - whether evolutionary pressures, global pandemics, or personal upheavals like job losses - force exploration that behavioral biases would otherwise prevent. These external forces compel us out of suboptimal situations, enabling the discovery of higher peaks we might never have voluntarily pursued.

Limitations #

Defining the goal #

This whole time, I used ‘well-being’ as a general term to describe the outcome humans are optimising through our life choices. However, this very idea of ‘well-being’ itself is subject to individual goals and values. While we have explored the optimisation of ‘well-being’ through the lenses of individuals and evolutionary biology using the concepts of ‘hot streaks’ and organisms’ fitness to survive ecologically, for one to effectively apply this model to their individual lives, we need to consider how success or well-being is defined for ourselves.

Assumption of motivation #

This model of stochastic hill-climbing assumes the desire to climb the hill. It assumes the individual making decisions has the motivation to optimise their well-being. ‘Hot streaks’ would not be possible if the artist, film director, or scientist had no motivation to even produce works. On the other hand, adaptive landscapes assume that the organism is influenced by natural selection and an innate or biological desire to survive.

Notably, the idea of randomness contributing to flourishing also depends on our ability to adapt and climb the hill from new locations. If random events were to throw us off our current hills, are we capable of climbing back up? Similarly, as the environment changes, which it rapidly does as we often observe in the world, are we capable of identifying these changes, the positions we are in, and subsequently learning to adapt to the new environment? Behavioural traits like learning agility (Critical Core Skills That Employers Want | MySkillsFuture. Gov. Sg, 2024), trait curiosity, and resilience will be necessary for individuals to maintain or improve their well-being in these dynamic environments. Perhaps it hints at how education systems around the world, like Singapore (Ng, 2020), should adjust curriculums to better facilitate the development of these traits amongst students.

Complexity of dimensionality #

In adaptive landscapes, a criticism is raised by R.A. Fisher on the complexity of dimensionality. The probability of local optima decreases with increasing dimensions in a high-dimensional genotype space (Obolski et al., 2017). As individuals, it might be difficult to mentally evaluate our well-being simply because there are so many dimensions that influence one another. Similarly, our choices in life are highly interdependent, be it due to opportunity costs or other relationships. A simple choice of going to the gym may increase our physical well-being but lead to opportunity costs of time and result in lower capacities in other life commitments.

A possible step would be to operationalise the dimensions and metrics of well-being clearly. For instance, narrowing the dimensions down only to the possible career options and salary. While such models can be narrower in scope, they may be more practical for evaluation.

Conclusion #

Stochastic hill-climbing offers a unique computational lens for understanding how randomness contributes to human flourishing. While it may not capture every aspect of well-being yet, it provides a promising framework for understanding how unexpected events and enforced changes, despite their initial challenges, can lead to improved life circumstances.

From empirical observations of stochastic hill-climbing, we may come to realise that rather than viewing life’s uncertainties as pure noise to be minimised, we could see them as essential components of a natural optimisation process operating at multiple scales - from biological evolution to individual development. In our quest for well-being or ‘success’, perhaps we should view life’s randomness not as an obstacle to overcome but as a feature that helps us escape the limitations of our bounded rationality and discover new peaks of flourishing.

References (this is not APA! just a list) #

Here are all the references from your essay in a simple list format:

• Aguilar-Rodríguez, J., Payne, J. L., & Wagner, A. (2017). A thousand empirical adaptive landscapes and their navigability. Nature Ecology & Evolution, 1(2), 0045.
• Career as a gradient descent problem. (2023, May 30). https://daily.stoa.com/newsletter/career-as-a-gradient-descent-problem
• Christian, B., & Griffiths, T. (2016). Algorithms to live by: The computer science of human decisions (First U.S. Edition). Henry Holt and Company.
• Climbing the wrong hill. (2009, September 19). https://cdixon.org/2009/09/19/climbing-the-wrong-hill/
• Critical core skills that employers want | MySkillsFuture. Gov. Sg. (2024, April 30). https://www.myskillsfuture.gov.sg/content/portal/en/career-resources career-resources/education-career-personal-development/2022_Critical_Core_Skills.html
• Crona, K. (2018). Recombination and peak jumping. PLOS ONE, 13(3), e0193123.
• Giron, A. P., Ciranka, S., Schulz, E., Van Den Bos, W., Ruggeri, A., Meder, B., & Wu, C. M. (2023). Developmental changes in exploration resemble stochastic optimisation. Nature Human Behaviour, 7(11), 1955–1967.
• Hill climbing. (2021, November 7). Byron Grealy. https://www.byrongrealy.com/blog/hill-climbing
• Langley, P., Gennari, J. H., & Iba, W. (1987). Hill-climbing theories of learning. In Proceedings of the Fourth International Workshop on MACHINE LEARNING (pp. 312–323). Elsevier.
• Little, D. (2018). Rational life plans? In G. Bronner & F. Di Iorio (Eds.), The Mystery of Rationality (pp. 131–145). Springer International Publishing.
• Liu, L., Dehmamy, N., Chown, J., Giles, C. L., & Wang, D. (2021). Understanding the onset of hot streaks across artistic, cultural, and scientific careers.
• Liu, L., Wang, Y., Sinatra, R., Giles, C. L., Song, C., & Wang, D. (2017). Hot streaks in artistic, cultural, and scientific careers.
• McMahon, D. (2013). The pursuit of happiness in history. In I. Boniwell, S. David, & A.C. Ayers (Eds.), The Oxford Handbook of Happiness (pp. 252-261). Oxford University Press.
• McGhee, G. R. (2006). The geometry of evolution: Adaptive landscapes and theoretical morphospaces (1st ed.). Cambridge University Press.
• Ng, P. T. (2020). The paradoxes of student well-being in Singapore. ECNU Review of Education, 3(3), 437–451.
• Obolski, U., Ram, Y., & Hadany, L. (2017). Key Issues Review: Evolution on rugged adaptive landscapes.
• Qu, V. (2019, July 12). Simulated annealing: A life framework. https://vivqu.com/blog/2019/07/12/simulated-annealing/
• Reed, D. D., & Luiselli, J. K. (2011). Temporal discounting. In S. Goldstein & J. A. Naglieri (Eds.), Encyclopedia of Child Behavior and Development (pp. 1474–1474). Springer US.
• Sadeghiyeh, H., Wang, S., Alberhasky, M. R., Kyllo, H. M., Shenhav, A., & Wilson, R. C. (2020). Temporal discounting correlates with directed exploration but not with random exploration. Scientific Reports, 10(1), 4020.
• Tang, L. (2021, September 25). Life is gradient descent: A plea against short-term overoptimisation. Leonard Tang. https://leonardtang.me/posts/Life-Gradient-Descent/
• Thorstad, D. (2024). Bounded rationality. In D. Thorstad, Inquiry Under Bounds (1st ed., pp. 21–36). Oxford University Press.
• Wegener, I. (2005). Simulated annealing beats metropolis in combinatorial optimisation. In L. Caires, G. F. Italiano, L. Monteiro, C. Palamidessi, & M. Yung (Eds.), Automata, Languages and Programming (Vol. 3580, pp. 589–601). Springer Berlin Heidelberg.