aises_5_2

<h1 id="introduction-to-complex-systems">5.2 Introduction to Complex
Systems</h1>
<h2 id="the-reductionist-paradigm">5.2.1 The Reductionist Paradigm</h2>
<p>Before we describe complex systems, we will first look at non-complex
systems and the methods of analysis that can be used to understand them.
This discussion sits under the <em>reductionist paradigm</em>. According
to this paradigm, systems are just the sum of their parts, and can be
fully understood and described with relatively simple mathematical
equations or logical relations.</p>
<p><strong>The mechanistic approach analyzes a system by studying each
component separately.</strong> A common technique for understanding a
system is to identify its components, study each one separately, and
then mentally “reassemble” it. Once we know what each part does, we can
try to place them all in a simple mechanism, where one acts on another
in a traceable sequence of steps, like cogs and wheels. This style of
analysis is called the <em>mechanistic approach</em>, which often
assumes that a system is like a line of dominos or a Rube Goldberg
machine; if we set one component in motion, we can accurately predict
the linear sequence of events it will trigger and, thus, the end
result.</p>
<figure id="rube_goldberg">
<p><img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/Rubes-diagram.png"
        alt="image" class="tb-img-full" style="width: 70%"/>
<span id="rube_goldberg" label="rube_goldberg"></span></p>
<p class="tb-caption">Figure 5.1: A Rube Goldberg machine with many parts that each feed directly into the next can be
well explained by the mechanistic approach.</p>
<!--<figcaption>Example of mechanistic approach - Rube Goldberg-->
<!--machine</figcaption>-->
</figure>
<p><strong>Many human artifacts can be understood
mechanistically.</strong> Devices like bicycles, clocks, and sewing
machines are designed with specific mechanisms in mind, where one
component directly acts on another in a cause-and-effect way to perform
an intended function. For example, we can look at a bicycle and
understand that turning the pedals will pull on a chain, which will turn
the wheels, which will move the bicycle forward.</p>
<p><strong>We can often derive mathematical equations that govern
mechanistic systems.</strong> If we can successfully model a system’s
behavior mechanistically, then we can usually find mathematical
equations that describe its behavior. We can use these equations to
calculate how the system will respond to different inputs. With this
knowledge, we can control what the system does by controlling the
inputs. For example, if we know how quickly the pedals on a bicycle are
rotating then we can calculate the speed at which it is traveling.
Conversely, we can control the bicycle’s speed by controlling how
quickly the pedals rotate.</p>
<p><strong>Many conventional computer programs can also be understood
mechanistically.</strong> Simple algorithmic computer programs involving
for-loops and “if... else...” constructions can be understood in this
way too. Given any input, we can trace through the program’s operations
to predict the output. Similarly, for any given output, we can trace the
steps backward and deduce information about the input.<p>
Functions in computer programs can also be understood mechanistically.
We can create functions within programs and give them names that are
readable and intuitive to humans. For instance, we can name a function
“add<span class="math inline">(<em>x</em>,<em>y</em>)</span>” and define
it to return the sum of <span class="math inline"><em>x</em></span> and
<span class="math inline"><em>y</em></span>. We can then write a
computer program using various functions like this, and we can analyze
it by understanding how each function works on its own and then looking
at the sequence of functions the program follows. This enables us to
predict reliably what output the program will give for any input.</p>
<p><strong>If there are a large number of components, we can sometimes
use statistics.</strong> Suppose we are trying to predict the behavior
of a gas in a box, which contains on the order of <span
class="math inline">10<sup>23</sup></span> particles (that is, 1
followed by 23 zeros). We clearly cannot follow each one and keep track
of its effects on the others, as if it were a giant mechanism.<p>
However, in the case of a system like a gas in a box, the broader system
properties of pressure and temperature can be related to averages over
the particle motions. This allows us to use statistical descriptions to
derive simple equations governing the gas’s coarse-grained behavior at
the macroscopic level. For example, we can derive an equation to
calculate how much the gas pressure will increase for a given rise in
temperature.</p>
<p><strong>The mechanistic and statistical approaches fall within the
reductionist paradigm.</strong> Both mechanistic and statistical styles
of analysis seek to understand and describe systems as combinations or
collections of well-understood components. Under the mechanistic
approach, we account for interactions by placing the components in a
mechanism, assuming they only affect one another in a neat series of
direct one-to-one interactions. Under the statistical approach, we
assume that we do not need to know the precise details of how each
interaction plays out because we can simply take an average of them to
calculate the overall outcome.</p>
<p><strong>Summary.</strong> Reductionist styles of analysis assume that
a system is no more than the sum of its parts. For a reductionist
analysis to work, one of the following assumptions should often apply:
There either needs to be a simple, traceable mechanism governing the
system’s behavior, or we need to be able to relate the broader system
properties to statistical averages over the components.</p>
<h3 id="limitations-of-the-reductionist-paradigm">Limitations of the
Reductionist Paradigm</h3>
<p>Having discussed simple systems and how they can be understood
through reductionism, we will now look at the limitations of this
paradigm and the types of systems that it cannot be usefully applied to.
We will look at the problems this presents for understanding systems and
predicting their behaviors.</p>
<p><strong>Many real-world systems defy reductionist
explanation.</strong> Imagine that, instead of looking at a bicycle or a
gas in a box, we are trying to understand and predict the behavior of an
ecosystem, weather patterns, or a human society. In these cases, there
are clearly far too many components for us to keep track of what each
one is doing individually, meaning that we cannot apply the mechanistic
approach. Additionally, there are also many complex interdependencies
between the components, such that any given component might behave
differently in the context of the system than it does in isolation. We
cannot, therefore, use statistics to treat the system’s behavior as a
simple aggregate of the components’ individual behaviors.</p>
<p><strong>In complex systems, the whole is more than the sum of its
parts.</strong> The problem is that reductionist-style analysis is
poorly suited to capturing the diversity of interdependencies within
complex systems. Reductionism only works well if the interactions follow
a rigid and predictable mechanism or if they are random and independent
enough to be modeled by statistics. In complex systems, neither of these
assumptions hold.<p>
In complex systems, interactions do not follow a rigid, structured
pattern, but components are still sufficiently interconnected that they
cannot be treated as independent. These interactions are the source of
many novel behaviors that make complex systems interesting. To get a
better grasp of these systems, we need to go beyond reductionism and
adopt an alternative, more holistic framework for thinking about
them.<p>
<strong>We can sometimes predict general short-term trends in complex
systems.</strong> Note that we may be able to predict high-level
patterns of behavior in some complex systems, particularly if we are
familiar with them and have many observations of their past behavior.
For example, we can predict with a high degree of confidence that, in
the northern hemisphere, a day in January next year will be colder than
a day in June.<p>
However, it is much more difficult to predict specific details, such as
the exact temperature or whether it will rain on a given day. It is also
much more challenging to predict the longer-term trajectory of the
system, such as what the climate will look like in several centuries or
millennia. This is because complex systems often develop in a more
open-ended way than simple systems and have the potential to evolve into
a wider range of states, with numerous factors influencing the path they
take.</p>
<p><strong>New or unfamiliar complex systems are even more difficult to
predict.</strong> The challenges in predicting how complex systems will
behave are compounded when we face newly emerging ones, such as those
involving AI. While we have plenty of historical information and
experience to help us predict weather patterns, we have little past data
to inform us on how AI systems and their use in society will develop.
Nevertheless, studying other complex systems and paying attention to
their shared properties can give us insights into how AI might evolve.
This might offer clues as to how we can avoid potential negative
consequences of using AI.</p>
<figure id="approaches">
<img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/approaches.png" class="tb-img-full" style="width: 80%"/>
<p class="tb-caption">Figure 5.2: Often, we use mechanistic or statistical approaches to analyzing systems. When there
are many components with strong interdependence, these are insufficient, and we need a complex
systems approach.</p>
<!--<figcaption>Three approaches to analysing systems</figcaption>-->
</figure>
<p><strong>Summary.</strong> Reductionist styles of analysis cannot give
us a full understanding of complex systems, whose components neither
function like a deterministic mechanism nor behave randomly and
independently enough to use statistics—shown in Figure 5.2. This lack of a full understanding
presents challenges for predicting the system’s behavior on two levels:
which component will perform which action at what time, and how the
whole system might change over the long term.<p>
</p>
<h2 id="the-complex-systems-paradigm">5.2.2 The Complex Systems Paradigm</h2>
<p>Now that we have seen that many systems of interest are inscrutable
to the reductionist paradigm, we need an alternative lens through which
to understand them. To this end, we will discuss the <em>complex systems
paradigm</em>, which takes a more holistic view, placing emphasis on the
most salient features shared across various real-world complex systems
that the reductionist paradigm fails to capture. The benefit of this
paradigm is that it provides “a way of seeing and talking about reality
that helps us better understand and work with systems to influence the
quality of our lives.”</p>
<p><strong>Complex systems exhibit emergent properties that are not
found in their components.</strong> As discussed above, some systems
cannot be usefully understood in a reductionist way. Studying a complex
system’s components in isolation and doing mental reassembly does not
amount to what we observe in reality. One primary reason for this is the
phenomenon of <em>emergence</em>: the appearance of striking,
system-wide features that cannot be found in any of the system’s
components.<p>
The presence of emergent features provides one sense in which complex
systems are “more than the sum of their parts.” For example, we do not
find atmospheric currents in any of the molecules of nitrogen and oxygen
that make up the atmosphere, and the flexible intelligence of a human
being does not exist in any single neuron. Many biological concepts such
as adaptation, ecological niche, sexuality, and fitness are not simply
reduced to statements about molecules. Moreover, “wetness” is not found
in individual water molecules. Emergence is so essential that we will
use it to construct a working definition of complex systems.</p>
<p><strong>Working definition.</strong> Complex systems are systems of
many interconnected components that collectively exhibit emergent
features, which cannot, in practice, be derived from a reductive
analysis of the system in terms of its isolated components.</p>
<p><strong>Ant colonies are a classic example of a complex
system.</strong> An ant colony can grow to a size of several million
individuals. Each ant is a fairly simple creature with a short memory,
moving around in response to chemical and tactile cues. The individuals
interact by randomly bumping into each other and exchanging pheromones.
Out of this mess of uncoordinated interactions emerge many fascinating
collective behaviors. These include identifying and selecting
high-quality food sources or nest sites, forming ant trails, and even
constructing bridges over gaps in these trails (formed by the stringing
together of hundreds of the ants’ bodies). Ant colonies have also been
observed to “remember” the locations of food sources or the paths of
previous trails for months, years, or decades, even though the memory of
any individual ant only lasts for a few days at most.</p>
<p><strong>Ant colonies satisfy both aspects of the working definition
of complex systems.</strong> First, the emergent features of the colony
include the collective decision-making process that enables it to choose
a food source or nest site, the physical ability to cross over gaps many
times wider than any ant, and even capabilities of a cognitive nature
such as extended memory. We could not predict all of these behaviors and
abilities from observing any individual ant, even if each ant displays
some smaller analogs of some of these abilities.<p>
Second, these emergent features cannot be derived from a reductive
analysis of the system focused on the properties of the components. Even
given a highly detailed study of the behavior of an individual ant
considered in isolation, we could not derive the emergence of all of
these remarkable features. Nor are all of these features simple
statistical aggregates of individual ant behaviors in any practical
sense, although some features like the distribution of ants between
tasks such as foraging, nest maintenance, and patrolling have been
observed as decisions on the level of an individual ant as well.</p>
<p>This distinguishes a more complex system like an ant colony from a
simpler one such as a gas in a box. Although the gas also has emergent
properties (like its temperature and pressure), it does not qualify as
complex. The gas’s higher-level properties can be straightforwardly
reduced to the statistics of the lower-level properties of the component
particles. However, this was not always the case: it took many decades
of work to uncover the statistical mechanics of gases from the
properties of individual molecules. Complexity can be a feature of our
understanding of the system rather than the system itself.</p>
<p><strong>Complex systems are ubiquitous in nature and
society.</strong> From cells, organisms, and ecosystems, to weather
systems, cities, and the World Wide Web, complex systems are everywhere.
We will now describe two further examples, referred to throughout this
chapter.</p>
<p><strong>Economies are complex systems.</strong> The components of an
economic system are the individual persons, companies, and firms
participating in the economy. These economic agents interact via various
kinds of financial transactions, such as lending, borrowing, investing,
and purchasing and selling goods. Out of these interactions emerge
complex economic phenomena such as inflation, stock-market indexes, and
interest rates. These economic phenomena are not manifested by any
individual agent and cannot be derived by studying the behavior of these
agents considered separately; rather, they arise from the complex
network of interactions between them.</p>
<p><strong>The human brain is a complex system.</strong> The human brain
consists of around 86 billion neurons, each one having, on average,
thousands of connections to the others. They interact via chemical and
electrical signals. Out of this emerge all our impressive cognitive
abilities, including our ability to use language, perceive the world
around us, and control the movements of our body. Again, these cognitive
abilities are not found in any individual neuron, arising primarily from
the rich structure of neuronal connections; even if we understood
individual neurons very well, this would not amount to an understanding
of (or enable a derivation of) all these impressive feats accomplished
by the brain.</p>
<p><strong>Interactions matter for complex systems.</strong> As these
examples illustrate, the interesting emergent features of complex
systems are a product of the interactions (or interconnections) between
their components. This is the core reason why these systems are not
amenable to a reductive analysis, which tries to gain insight by
breaking the system into its parts. As the philosopher Paul Cilliers
writes: “In ‘cutting up’ a system, the analytic method destroys what it
seeks to understand” <span class="citation"
data-cites="cilliers2014complexity">[1]</span>.</p>
<p><strong>Summary.</strong> Complex systems are characterized by
emergent features that arise from the complex interactions between
components, but do not exist in any of the individual components, and
cannot be understood through or derived from a reductive analysis of
them. Complex systems are ubiquitous, from ant colonies to economies to
the human brain.</p>
<h2 id="deep-learning-systems-as-complex-systems">5.2.3 Deep Learning Systems
as Complex Systems</h2>
<p><strong>An essential claim of this chapter is that deep learning
models are complex systems.</strong> Here, we will briefly discuss what
a reductionist approach to understanding deep learning systems would
look like and why it is inadequate.<p>
Consider a deep learning system that correctly classifies an image of a
cat. How does it do this? The reductionist approach to this question
would first try to break down the classification into a sequence of
smaller steps and then find parts of the neural network responsible for
executing each of them. For instance, we might decompose the problem
into the identification of cat ears + whiskers + paws and then look for
individual neurons (or small clusters of neurons) responsible for each
of these elements.</p>
<p><strong>The reductionist approach cannot fully describe neural
networks.</strong> In some cases, it seems possible to find parts of a
neural network responsible for different elements of such a task.
Researchers have discovered that progressively later layers of deep
neural networks are generally involved in recognizing progressively
higher-level features of the images they have been trained to classify.
For example, close to the input layer, the neural network might be doing
simple edge detection; a little further into the hidden layers, it might
be identifying different shapes; and close to the output, it might be
combining these shapes into composites.<p>
However, there is no clear association between an individual node in a
given layer and a particular feature at the corresponding level of
complexity. Instead, all the nodes in a given layer are partially
involved in detecting any given feature at that level. That is to say,
we cannot neatly attribute the detection of each feature to a specific
node, and treat the output as the sum of all the nodes detecting their
specific features. Although there have been instances of researchers
identifying components of neural networks that are responsible for
certain tasks, there have been few successes, and they have required
huge efforts to achieve. In general, this approach has not so far worked
well for explaining higher-level behaviors.</p>
<p><strong>The complex systems paradigm is more helpful for deep
learning systems.</strong> As these problems suggest, we cannot
generally expect to find a simple, human-interpretable set of features
that a neural network identifies in each example and “adds together” to
reach its predictions. Deep learning systems are too complex to reduce
to the behavior of a few well-understood parts; consequently, the
reductionist paradigm is of limited use in helping us think about them.
As we will discuss later in this chapter, the complex systems paradigm
cannot entirely make up for this or enable a complete understanding of
these systems. Nonetheless, it does give us a vocabulary for thinking
about them that captures more of their complexity and can teach us some
general lessons about interacting with them and avoiding hazards.</p>
<p><strong>Summary.</strong> The difficulties involved in explaining
neural networks’ activity through simple mechanisms are one piece of
evidence that they are best understood as complex systems. We will
substantiate this claim throughout the next section, where we run
through some of the hallmark features of complex systems and discuss how
they apply to deep learning models.</p>
<h2 id="complexity-is-not-a-dichotomy">5.2.4 Complexity is Not a
Dichotomy</h2>
<p>In the previous section, we proposed a working definition of complex
systems that suffices for an informal discussion, though it is not
completely precise. In fact, there is no standard definition of
complexity used by all complex-systems scientists. In part, this is
because complexity is not a dichotomy.</p>
<p><strong>Understanding system complexity.</strong> While we have
described a distinction between a “simple” and “complex” system,
labeling a system as inherently simple or complex can be misleading.
Complexity is not always intrinsic to a system. Instead, it depends on
our understanding. Certain phenomena in physics, for instance, have
transitioned from being poorly understood “complex” concepts to
well-explained “simple” mechanics through advanced analysis of the
properties of the system. Superconductivity—the property of a material
to conduct electricity without resistance when cooled below a certain
critical temperature—is an example of this transition in
understanding.<p>
Superconductivity was originally perceived as a complex phenomenon due
to the emergent behavior arising from electron interactions in metals.
However, with the discovery of the Bardeen-Cooper-Schrieffer (BCS)
theory, it became clear that superconductivity could be explained
through the pairing of electrons. By considering these pairs as the
components of interest rather than individual electrons,
superconductivity was reclassified as a conceptually “simple” system
that can be described by reductionist models.</p>
<p><strong>Complexity, information, and reductionism.</strong> Current
research in complex systems acknowledges the importance of interactions
in determining emergent behavior but doesn’t abandon the search for
mechanistic explanations. Often, mechanistic explanations of systems can
be found when considering a larger basic building block, such as
pairs of electrons for superconductivity. This choice of scale is
important for creating effective models of possibly complex
phenomena.<p>
Thus, rather than a binary classification, systems might be better
understood as existing on a spectrum based on the scale and amount of
information required to predict their behavior accurately. Complex
systems are those that, at a certain scale, require a vast amount of
information for prediction, indicating their relative incompressibility.
However, they could still be explained mechanistically, if we understood
them sufficiently well.</p>
<h2 id="the-hallmarks-of-complex-systems">5.2.5 The Hallmarks of Complex
Systems</h2>
<p>Since complexity is not a dichotomy, it is difficult to pin down when
exactly we can consider systems complex. In place of a precisely
demarcated domain, complex-systems scientists study numerous salient
features that are generally shared by the systems of interest. While
disciplines like physics seek fundamental mechanisms that can explain
observations, the study of complex systems looks for salient
higher-level patterns that appear across a wide variety of
systems.<p>
We consider seven key characteristics of complex systems. Chief among
these is emergence, but several others also receive attention:
self-organization, feedback and nonlinearity, criticality, adaptive
behavior, distributed functionality, and scalable structure. We will now
describe each of these hallmarks and explain their implications. Along
the way, we will show that deep learning systems share many similarities
with other complex systems, strengthening the case for treating them
under this paradigm.</p>
<h3 id="emergence">Emergence</h3>
<p>We have already discussed emergence, the appearance of striking
system-wide features that cannot be found in any of the components of
the system. Ant colonies swarm over prey and build bridges over gaps in
their trail; economies set prices and can crash; human brains think,
feel, and sense. These remarkable behaviors are inconceivable for any
individual component—ant, dollar, or neuron—existing in isolation.</p>
<p><strong>Emergent features often spontaneously “turn on” as we scale
up the system in size.</strong> A group of 100 army ants placed on the
ground behaves not like an enfeebled colony but rather like no colony at
all; the ants just walk around in circles until they starve or die of
exhaustion. If the system is scaled up to tens of thousands of ants,
however, a qualitative shift in behavior occurs as the colony starts
behaving like an intelligent superorganism.</p>
<p><strong>Emergent abilities have been observed in deep learning
systems.</strong> Large language models (LLMs) are trained to predict
the next token in a string of words. Smaller LLMs display a variable
ability to output coherent sentences, as might be expected based on this
training. Larger LLMs, however, spontaneously gain qualitatively new
capabilities, such as translating text or performing three-digit
arithmetic. These abilities can emerge without any task-specific
training.</p>
<p><strong>Summary.</strong> Emergent properties arise collectively from
interactions between components, and are a defining feature of complex
systems. These features often appear spontaneously as a system is scaled
up. Emergent capabilities have already been observed in deep learning
systems.</p>
<h3 id="feedback-and-nonlinearity">Feedback and Nonlinearity</h3>
<p>Two closely related hallmarks of complexity are <em>feedback</em> and
<em>nonlinearity</em>. Feedback refers to circular processes in which a
system and its environment affect one another. There are multiple types
of nonlinearity, but the term generally describes systems and processes
where a change in the input does not necessarily translate to a
proportional change in the output. We will now discuss some mechanisms
behind nonlinearity, including feedback loops, some examples of this
phenomenon, and why it makes complex systems’ behavior less
predictable.</p>
<p><strong>In mathematics, a linear function is one whose outputs change
in proportion to changes in the inputs.</strong> The functions <span
class="math inline"><em>f</em>(<em>x</em>) = 3<em>x</em></span> and
<span
class="math inline"><em>f</em>(<em>x</em>) = 100(<em>x</em>−10)</span>
linear. Meanwhile, the functions <span
class="math inline"><em>f</em>(<em>x</em>) = <em>x</em><sup>2</sup></span>
and <span
class="math inline"><em>f</em>(<em>x</em>) = <em>e</em><sup><em>x</em></sup></span>
are nonlinear.</p>
<p><strong>Complex systems are nonlinear functions of their
inputs.</strong> Complex systems process inputs in a nonlinear way. For
example, when ant colonies are confronted with two food sources of
differing quality, they will often determine which source is of higher
quality and then send a disproportionately large fraction of its
foragers over to exploit it rather than form two trails in proportion to
the quality of the food source. Neural networks are also nonlinear
functions of their inputs. This is why adversarial attacks can work
well: adding a small amount of noise to an image of a cat need not
merely reduce the classifier’s confidence in its prediction, but might
instead cause the network to confidently misclassify the image
entirely.</p>
<p><strong>Nonlinearity makes neural networks hard to
decompose.</strong> A deep neural network with 10 layers cannot be
replaced by five neural networks, each with only two layers. This is due
to the nonlinear activation functions (such as GELUs) between their
nodes. If the layers in a neural network simply performed a sequence of
linear operations, the whole network could be reduced to a single linear
operation. However, nonlinear operations cannot be reduced in the same
way, so nonlinear activation functions mean that deep neural networks
cannot be collapsed to networks with only a few layers. This property
makes neural networks more capable, but also more difficult to analyze
and understand.</p>
<p><strong>A major source of nonlinearity is the presence of
feedback.</strong> Feedback occurs when the interdependencies between
different parts of a system form loops (e.g., A depends on B, which in
turn depends on A). These feedback loops can reinforce certain processes
in the system (positive feedback), and quash others (negative feedback),
leading to a nonlinear relationship between the system’s current state
and how it changes. The following are examples of feedback loops in
complex systems.</p>
<p><strong>The rich get richer.</strong> Wealthy people have more money
to invest, which brings them a greater return on investment. In a single
investment cycle, the return on investment is greater in proportion to
their greater wealth: a linear relationship. However, this greater
return can then be reinvested. Doing so forms a positive feedback loop
through which a slight initial advantage in wealth can be transformed
into a much larger one, leading to a nonlinear relationship between a
person’s wealth and their ability to make more money.</p>
<p><strong>Learning in the brain involves a positive feedback
loop.</strong> Connections between neurons are strengthened according to
Hebb’s law (“neurons that fire together, wire together”). Stronger
connections increase the probability of subsequent episodes of “firing
together”, further strengthening those connections. As a result of this
feedback process, our memories do not strengthen or weaken linearly with
time. The most efficient way to learn something is by revisiting it
after increasing intervals of intervening time, a method called “spaced
repetition.”</p>
<p><strong>Task distribution in beehives can be regulated by feedback
loops.</strong> When a forager bee finds a source of water, it performs
a “waggle dance” in front of the hive to signal to the other bees the
direction and distance of the source. However, a returning forager needs
to find a receiver bee onto which to unload the water. If too many
foragers have brought back water, it will take longer to find a
receiver, and the forager is less likely to signal to the others where
they should fly to find the source. This negative feedback process
stabilizes the number of bees going out for water, leading to a
nonlinear relationship between the number of bees currently flying out
for water and the number of additional bees recruited to the task.</p>

<p><strong>AI systems involve feedback loops.</strong> In a system where
agents can affect the environment, but the environment can also affect
agents, the result is a continual, circular process of change—a feedback
loop. Another example of feedback loops involving AIs is the
reinforcement-learning technique of self-play, where agents play against
themselves: the better an agent’s performance, the more it has to
improve to compete with itself, leading its performance to increase even
more.</p>
<p><strong>Feedback processes can make complex systems’ behavior
difficult to predict.</strong> Positive feedback loops can amplify small
changes in a system’s initial conditions into considerable changes in
its resulting behavior. This means that nonlinear systems often have
regimes in which they display extreme sensitivity to initial conditions,
a phenomenon called chaos (colloquially referred to as the <em>butterfly
effect</em>). A famous example of this is the logistic map, an equation
that models how the population of a species changes over time:<p>
<span
class="math display"><em>x</em><sub><em>n</em> + 1</sub> = <em>r</em><em>x</em><sub><em>n</em></sub>(1−<em>x</em><sub><em>n</em></sub>).</span>
This equation is formulated to capture the feedback loops that affect
how the population of a species changes: when the population is low,
food sources proliferate, enabling the population to grow; when it is
high, overcrowding and food scarcity drive the population down again.
<span class="math inline"><em>x</em><sub><em>n</em></sub></span> is the
current population of a species as a fraction of the maximum possible
population that its environment can support. <span
class="math inline"><em>x</em><sub><em>n</em> + 1</sub></span>
represents the fractional population at some time later. The term <span
class="math inline"><em>r</em></span> is the rate at which the
population increases if it is not bounded by limited resources.<p>
When the parameter <span class="math inline"><em>r</em></span> takes a
value above a certain threshold (<span
class="math inline"> ∼ 3.57</span>), we enter the chaotic regime of this
model, in which a tiny difference in the initial population makes for a
large difference in the long-run trajectory. Since we can never know a
system’s initial conditions with perfect accuracy, chaotic systems are
generally considered difficult to predict.<p>
</p>
<figure id="approaches">
<img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/logistic_map_v2.png" class="tb-img-full" style="width: 80%"/>
<p class="tb-caption">Figure 5.3: When systems are predictable, small changes in initial conditions can taper out. When
they are chaotic, small changes in initial conditions lead to wildly different outcomes.</p>
<!--<figcaption>Logistic map</figcaption>-->
</figure>
<p><strong>AIs as a self-reinforcing feedback loop.</strong> Since AIs
can process information and reach decisions more quickly than humans,
putting them in charge of certain decisions and operations could
accelerate developments to a pace that humans cannot keep up with. Even
more AIs may then be required to make related decisions and run adjacent
operations. Additionally, if society encounters any problems with AI-run
operations, it may be that AIs alone can work at the speed and level of
complexity required to address these problems. In this way, automating
processes could set up a positive feedback loop, requiring us to
continually deploy ever-more AIs. In this scenario, the long-term use of
AIs could be hard to control or reverse.</p>
<p><strong>Summary.</strong> There are multiple ways in which complex
systems exhibit nonlinearity. A small change in the system’s input will
not necessarily result in a proportional change in its behavior; it
might completely change the system’s behavior, or have no effect at all.
Positive feedback loops can amplify changes, while negative feedback
loops can quash them, leading a system to evolve nonlinearly depending
on its current state, and making its long-run trajectory difficult to
predict.</p>
<h3 id="self-organization">Self-Organization</h3>
<p>The next salient feature of complex systems we will discuss is
<em>self-organization</em>. This refers to how the components direct
themselves in a way that produces collective emergent properties without
any explicit instructions.</p>
<p><strong>Complex systems sometimes organize themselves
spontaneously.</strong> The forms and internal structure changes of
complex systems are neither imposed by a top-down design nor centrally
coordinated by “master components.” The high-level order and
organization of a complex system is itself an emergent property that
cannot be analyzed in terms of individual components. We will now look
at some examples of self-organization.</p>
<p><strong>Workers self-organize in ant colonies.</strong> In ant
colonies, worker ants perform a variety of tasks, such as nest
maintenance, brood care, foraging for food, and patrolling around the
nest for signs of danger. Task allocation is partly determined by demand
and opportunity in the environment. For example, the colony will shift
to a more forager-heavy distribution if it discovers a large food
source. The way in which individual ants are recruited to different
tasks according to environmental demand and opportunity is
self-organizing: a product of local stochastic interactions between the
individuals, not set by a central controller (there’s no ant
commander).</p>
<p><strong>The efficient market hypothesis states that economies
self-organize to set prices.</strong> Increasing the price of a product
leads to an increase in its supply (as profit margins for vendors are
higher) and a decrease in its demand (as fewer consumers can afford it).
Decreasing the price of a product has the reverse effect. In theory, the
market price of a product will stabilize around the value at which the
supply matches the demand. The system of vendors and consumers
automatically “finds” the equilibrium market price without any
centralized control or external help.</p>
<p><strong>A neural network largely self-organizes during
training.</strong> One could argue that there is an element of top-down
control in the training of a neural network, in the way the
backpropagation adjusts parameters to reduce the loss. However, there is
not a predetermined plan specifying which parts of it are supposed to
perform the different functions needed to carry out the task. Instead,
the training process starts with a disordered system and its ultimate
shape is determined by many interactions between components, resulting
in a highly decentralized organization throughout the network. To a
large extent, therefore, the training process resembles
self-organization.</p>
<p><strong>Summary.</strong> In a complex system, each component
responds to conditions and directs its own actions such that the
components collectively exhibit emergent behaviors without any external
or central control. Neural networks arrange themselves in this way
during training.</p>
<h3 id="self-organized-criticality">Self-Organized Criticality</h3>
<p>Through self-organization, complex systems can reliably reach
configurations that might seem improbable or fine-tuned. We will now
look at the phenomenon of <em>self-organized criticality</em>, which is
an important example of this.</p>
<p><strong>Criticality is when a system is balanced at a tipping point
between two different states.</strong> In nuclear engineering, the
“critical mass” is the mass of a fissile material needed for a
self-sustaining nuclear chain reaction. Below the critical mass, the
chain reaction quickly dies out; above the critical mass, it continues
at an ever-increasing rate and blows up. The critical mass is a boundary
between these two regimes—the point at which the system “tips over” from
being subcritical (stable and orderly) to supercritical (unstable and
disorderly). It is therefore referred to as the tipping point, or
critical point, of the fissile system. Under normal operations, nuclear
reactors are maintained at a critical state where the ongoing reaction
ensures continual energy generation without growing into a dangerous,
uncontrolled reaction.</p>
<p><strong>Systems at their critical point are optimally sensitive to
fluctuating conditions.</strong> In the nuclear case, an internal
fluctuation would be the spontaneous fission of a nucleus. Below the
critical point, the consequences of this event invariably remain
confined to the neighborhood of the nucleus; above the critical point,
the knock-on effects run out of control. Precisely at criticality, a
local fission event can precipitate a chain reaction of any size,
ranging from a short burst to a cascading reaction involving the entire
system. This demonstrates how, at a critical point, a small event can
have the broadest possible range of effects on the system.<p>
The concept of criticality applies far beyond nuclear engineering: one
classic example is the sandpile model. A sandpile has a critical slope,
which is the tipping point between a tall, unstable pile and a shallow,
stable pile. Shallower than this slope, the pile is relatively
insensitive to perturbations: dropping additional grains onto the pile
has little effect beyond making it taller. Once we reach the critical
slope, however, the pile is poised to avalanche, and dropping extra
grains can lead to avalanches of any size, including system-wide ones
that effectively cause the whole pile to collapse. Again, we see that,
at criticality, single events can have a wide range of effects on the
system.</p>
<p><strong>The freezing point of water is a critical temperature between
its solid and liquid phases.</strong> In ice, the solid phase of water,
there is long-range order, and fluctuations away from this (pockets of
melting ice) are small and locally contained. In the liquid phase, there
is long-range disorder, and fluctuations away from this (formation of
ice crystals) are likewise small and locally contained. But at the
freezing point of water—the critical point between the solid and liquid
phases—the local formation of an ice crystal can rapidly spread across
the whole system. As a result, a critically cooled bottle of beer can
suddenly freeze all at once when it is perturbed, for example by being
knocked against a table.</p>
<p><strong>Neural networks display critical points.</strong> Several
studies have found that certain capabilities of neural networks suddenly
‘switch on’ at a critical point as they are scaled up. For example,
grokking is a network’s ability to work accurately for general, random
datasets, not just the datasets used in training. One study trained
neural networks to recognize patterns in tables of letters and fill in
the blanks, and found that grokking switched on quite suddenly <span
class="citation" data-cites="power2022grokking">[2]</span>. The study
reported that this ability remained near zero up to <span
class="math inline">10<sup>5</sup></span> optimization steps, but then
steeply increased to near <span class="math inline">100%</span> accuracy
by <span class="math inline">10<sup>6</sup></span> steps. This could be
viewed as a critical point.</p>
<p><strong>Self-organized criticality means systems can evolve in a
“punctuated equilibrium”.</strong> According to the theory of
<em>punctuated equilibrium</em>, evolutionary history consists of long
periods of relative stasis in which species experience very little
change, punctuated by occasional bursts of rapid change across entire
ecosystems. These sudden bursts can be understood through the lens of
self-organized criticality. Ecosystems typically in equilibrium can
slowly tend towards critical points, where they are optimally sensitive
to perturbations from outside (such as geological events) or
fluctuations from within (such as an organism developing a new behavior
or strategy through a chance mutation). When the ecosystem is near a
critical point, such a perturbation can potentially set off a
system-wide cascade of changes, in which many species will need to adapt
to survive. Similarly, AI development sometimes advances in bursts
(e.g., GANs, self-supervised learning in vision, and so on) with long
periods of slow development.</p>
<p><strong>Summary.</strong> Complex systems often maintain themselves
near critical points, or “tipping points”. At these points, a system is
optimally sensitive to internal fluctuations and external inputs. This
means it can undergo dramatic changes in response to relatively minor
events. A pattern of dramatic changes that sporadically interrupt
periods of little change can be described as a punctuated
equilibrium.</p>
<h3 id="distributed-functionality">Distributed Functionality</h3>
<p>As discussed earlier in this chapter, it is usually impractical to
attempt to decompose a complex system into its parts, assign a different
function to each one, and then assume that the system as a whole is the
sum of these functions. Part of the reason for this is <em>distributed
functionality</em>, another hallmark of complexity which we will now
explore.</p>
<p><strong>Complex systems can often be described as performing tasks or
functions.</strong> Insect colonies build nests, forage for food, and
protect their queens; economies calculate market prices and interest
rates; and the human brain regulates all the bodily processes essential
for our survival, such as heartbeat and breathing. In this context, we
can understand adaptive behavior as the ability of a complex system to
maintain its functionality when placed in a new environment or faced
with new demands.</p>
<p><strong>In complex systems, different functions are not neatly
divided up between subsystems.</strong> Consider a machine designed to
make coffee. In human artifacts like this, there is a clear delegation
of functions to different parts of the system—one part grinds the beans,
another froths the milk, and so forth. This is how non-complex systems
usually work to perform their tasks. In complex systems, by contrast, no
subsystem can perform any of the system’s functions on its own, whereas
all the subsystems working together can collectively perform many
different tasks. This property is called “distributed
functionality.”<p>
Note that distributed functionality does not imply that there is
absolutely no functional specialization of the system’s components.
Indeed, the components of a complex system usually come in a diversity
of different types, which contribute in different ways to the system’s
overall behavior and function. For example, the worker ants in an ant
colony can belong to different groups: foragers, patrollers, brood care
ants, and so on. Each of these groups, however, performs various
functions for the colony, and distributed functionality implies that,
within each group of specialists, there is no rigid assignment of
functions to components.</p>
<p><strong>Partial encoding means that no single component can complete
a task alone.</strong> The group of forager ants must perform a variety
of subtasks in service of the foraging process: locating a food source,
making a collective decision to exploit it, swarming over it to break it
up, and carrying small pieces of it back to the nest. A single forager
ant working alone cannot perform this whole process—or even any one
subtask; many ants are needed for each part, with each individual
contributing only partially to each task. We therefore say that foraging
is partially encoded within any single forager ant.</p>
<p><strong>Redundant encoding means there are more components than
needed for any task.</strong> A flourishing ant colony will have many
more ants than are necessary to carry out its primary functions. This is
why the colony long outlives its members; if a few patroller ants get
eaten, or a few foragers get lost, the colony as a whole barely notices.
We therefore say that each of the functions is redundantly encoded
across the component ants.<p>
An example of distributed functionality is the phenomenon known as the
“wisdom of crowds”, which was notably demonstrated in a report from a
village fair in 1906. At this fair, attendees were invited to take part
in a contest by guessing the weight of an ox. 787 people submitted
estimates, and it was reported that the mean came to 1,197 pounds. This
was strikingly close to the actual weight, which was 1,198 pounds.<p>
In situations like this, it is often the case that the average estimate
of many people is closer to the true value than any individual’s guess.
We could say that the task of making a good estimate is only partially
encoded in any given individual, who cannot alone get close to the
actual value. It is also redundantly encoded because any individual’s
estimate can usually be ignored without noticeably affecting the
average.<p>
On a larger scale, the wisdom of crowds might be thought to underlie the
effectiveness of democracy. Ideally, a well-functioning democracy should
make better decisions than any of its individual members could on their
own. This is not because a democratic society decomposes its problems
into many distinct sub-problems, which can then be delegated to
different citizens. Instead, wise democratic decisions take advantage of
the wisdom of crowds phenomenon, wherein pooling or averaging many
people’s views leads to a better result than trusting any individual.
The “sense-making” function of democracies is therefore distributed
across society, partially and redundantly encoded in each citizen.</p>
<p><strong>Neural networks show distributed functionality.</strong> In
neural networks, distributed functionality manifests most clearly as
distributed representation. In sufficiently large neural networks, the
individual nodes do not correspond to particular concepts, and the
weights do not correspond to relationships between concepts. In essence,
the nodes and connections do not “stand for” anything specific. Part of
the reason for this is partial encoding: in many cases, any given
feature of the input data will activate many neurons in the network,
making it impossible to locate a single neuron that represents this
feature. In addition, so-called polysemantic neurons are activated by
many different features of the input data, making it hard to establish a
correspondence between these neurons and any individual concepts.</p>
<p><strong>Distributed functionality makes it hard to understand what
complex systems are doing.</strong> Distributed functionality means that
we cannot understand a complex system by attributing each task wholly
and exclusively to a particular component, as the mechanistic approach
would seek to. Distributed representation in neural networks is a
particularly troubling instantiation of this insofar as it poses
problems for using human concepts in analyzing a complex system’s
“cognition”. The presence of distributed representation might be thought
to substantiate the concern that neural networks are uninterpretable
“black boxes”.</p>
<p><strong>Summary.</strong> Distributed functionality often means that
no function in a complex system can be fully or exclusively attributed
to a particular component. Since tasks are more loosely shared among
components, this is one of the main reasons that it is so difficult to
develop a definitive model of how a complex system works.</p>
<h3 id="scalable-structure-and-power-laws">Scalable Structure and Power
Laws</h3>
<p>As discussed above, the properties of a complex system often scale
nonlinearly with its size. Instead, they often follow power laws, where
a property is proportional to the system size raised to some power that
may be more or less than 1. We will now discuss these <em>power
laws</em>, which are another hallmark of complex systems.</p>
<figure id="approaches">
<embed src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/kleiber_v2.png" class="tb-img-full" style="width: 85%"/>
<p class="tb-caption">Figure 5.4: Data on mammals and birds demonstrate Kleiber’s Law, with a power law relationship
appearing as a straight line on a log-log graph.</p>
<!--<figcaption>Kleiber’s Law</figcaption>-->
</figure>
<p><strong>Complex systems often obey power-law scalings of their
properties with system size.</strong> Perhaps the most famous example of
a power-law scaling is Kleiber’s law in biology: across all mammals and
birds, and possibly beyond, the metabolic rate of a typical member of a
species scales with the three-quarters power of its body mass.</p>
<p><span class="math display">$$R \propto M^\frac{3}{4}$$</span></p>
<p>If we know that an elephant is five times heavier than a horse, we
can guess that the elephant’s metabolic rate will be approximately 3.3
times the horse’s: <span class="math inline">$$5^\frac34 \approx
3.3$$</span>. There are several other documented cases of this power-law
scaling behavior in complex systems. The average heart-rate for a
typical member of a mammalian species scales with the minus one-quarter
power of its body mass: <span class="math display">$$R \propto
M^{-\frac{1}{4}}.$$</span></p>
<p>At the same time, the average lifespan scales with the one-quarter
power of its body mass:</p>
<p><span class="math display">$$T \propto M^{\frac{1}{4}}.$$</span></p>
<p>This leads to the wonderful result that the average number of
heartbeats per lifetime is constant across all species of mammals
(around 1.5 billion).<p>
Among cities within the same country, the material infrastructure (such
as the lengths of pipes, powerlines, and roads, and the number of gas
stations) scales with population as a power-law with an exponent of
0.85. Also among cities within the same country, socioeconomic
quantities (such as incidents of crime and cases of flu) scale with the
population size raised to the 1.15 power.</p>
<p><strong>Experiments on LLMs show that their loss obeys power laws
too.</strong> In the paper in which DeepMind introduced the Chinchilla
model (<span class="citation"
data-cites="hoffmann2022training">[3]</span>), the researchers fit the
following parametric function to the data they collected from
experiments on language models of different sizes, where <span
class="math inline"><em>N</em></span> is the size of the model and <span
class="math inline"><em>D</em></span> is the size of the training
dataset: <span class="math display">$$L(N,D) = E + \frac{A}{N^{\alpha}}
+ \frac{B}{D^{\beta}}.$$</span></p>
<p>The irreducible loss (<span class="math inline"><em>E</em></span>) is
the lowest loss that could possibly be achieved. Subtracting this off,
we see that the performance of the model as measured by the loss (<span
class="math inline"><em>L</em></span>) exhibits a power-law dependency
on each of model parameter count (<span
class="math inline"><em>N</em></span>) and dataset size (<span
class="math inline"><em>D</em></span>).<p>
For more details on scaling laws in deep learning systems, see section 2.4.</p>
<p><strong>Summary.</strong> Certain important properties of complex
systems often scale nonlinearly with the size of the system. This means
that two separate systems will not behave in the same way as one single
system of equivalent size.</p>
<h3 id="adaptive-behavior">Adaptive Behavior</h3>
<p>The final hallmark of complexity we will discuss is <em>adaptive
behavior</em>, which involves a system changing its behavior depending
on the demands of the environment.</p>
<p><strong>Complex systems often adapt flexibly to new tasks and
environmental changes.</strong> Honeybees usually need to maintain their
hives within an optimum temperature range of 32-36°C. When temperatures
rise too high, bees engage in various adaptive behaviors to counteract
this. They fan their wings at the hive entrance, increasing air
circulation to cool down the hive. Additionally, more bees are sent out
to gather water, which helps regulate the hive’s temperature back to
normal <span class="citation" data-cites="zhao2021response">[4]</span>.
This ability to adjust their behavior to maintain homeostasis during
environmental changes exemplifies one type of adaptive behavior.<p>
The human brain, on the other hand, showcases a different form of
adaptability. It possesses the remarkable capacity to navigate novel
circumstances and solve unfamiliar problems. When faced with new
challenges, the brain’s ability to think about different things allows
us to adapt and thrive in diverse environments. For example, London’s
taxi drivers (“cabbies”) have been found to have larger-than-average
memory centers. This adaptation enables them to navigate the complex
maze of London’s streets effectively. Furthermore, the brain can also
adapt in response to injury. After a stroke or head injury, it can
rewire itself, repurposing undamaged areas to compensate for the damaged
ones. This adaptive behavior showcases the brain’s remarkable plasticity
and its ability to adapt and function even after experiencing
trauma.</p>
<p><strong>Some deep learning systems exhibit adaptive
behavior.</strong> So-called “online models” learn from new data
sequentially as they encounter it, rather than remaining fixed after an
initial training phase. This enables these models to dynamically adapt
to datasets that change over time, as well as continuing to perform well
in the real world when the inputs they encounter differ from their
training data, an ability known as “test-time adaptation” or simply
“adaptation”. While other deep learning systems such as Large Language
Models remain fixed after their training phase, there are strong
incentives to make these systems adaptive to overcome current
limitations such as costs of re-training and lack of up-to-date
information after the training date.<p>
Another example of adaptive behavior in deep learning systems is
<em>few-shot prompting</em>. This technique enables general deep
learning models (such as large language models) to be used to perform
certain tasks without any task-specific fine-tuning. It involves giving
the model a few examples (“shots”) of correct performance on the task,
which stimulate the model to adapt its outputs to these examples and
thereby carry out the desired task.</p>
<p><strong>Summary.</strong> Complex systems can often undergo rapid
changes in their structures and processes in response to internal and
external fluctuations. This adaptive behavior enables the continuation
of the system in a changing environment.</p>
<h3 id="review-of-the-hallmarks-of-complexity">Review of the Hallmarks
of Complexity</h3>
<p>There are seven hallmarks of complexity that we can look out for when
identifying complex systems. These hallmarks are:</p>
<ol>
<li><p>Emergence: the appearance of novel properties that arise from
interactions between the system’s components, but which do not exist in
any single component. These properties cannot be understood or predicted
from reductive analysis of components.</p></li>
<li><p>Feedback and nonlinearity: the presence of feedback loops that
can either amplify or quash changes in a complex system, and the
multiple ways in which a change in the input to a complex system can
produce a disproportionate change in the output.</p></li>
<li><p>Self-organization: the ability of a complex system to
spontaneously self-organize through the self-directed behaviors of the
components, without any external or centralized control.</p></li>
<li><p>Self-organized criticality: the tendency of complex systems to
maintain themselves near critical points, at which they can undergo
dramatic changes in response to even relatively minor
perturbations.</p></li>
<li><p>Distributed functionality: the way in which tasks are shared
loosely among a complex system’s components. Tasks are both partially
encoded—each individual contributes only partially to a task—and
redundantly encoded—there are more individuals that can contribute to a
task than are strictly necessary to complete it.</p></li>
<li><p>Scalable structure: the way in which properties of complex
systems scale nonlinearly with size, so that a property of a single
large system may be larger or smaller than the combined properties of
two separate systems of half the size.</p></li>
<li><p>Adaptive behavior: a complex system’s ability to change its
structure and processes in response to perturbations, enabling it to
continue functioning in a changing environment.</p></li>
</ol>
<h2 id="social-systems-as-complex-systems">5.2.6 Social Systems as Complex
Systems</h2>
<p>So far, we have described how deep learning systems possess many of
the classic features of complex systems. We have shown that they satisfy
the two aspects of our working definition of complex systems and that
they display all seven hallmarks discussed above.<p>
We will now consider the organizations that develop AIs and the
societies within which they are deployed, and describe how these systems
also exhibit the characteristics of complex systems. We will argue that,
on this basis, the problem of AI safety should be treated under the
complex systems paradigm.</p>
<h3
id="worked-example-corporations-and-research-institutes-as-complex-systems">Worked
Example: Corporations and Research Institutes as Complex Systems</h3>
<p><strong>The organizations developing AI technology are complex
systems.</strong> Corporations and research institutes have multiple
emergent properties that are not found in any of the individuals working
within them. Brand identity, for example, does not exist in any employee
of a company, but rather embodies and conveys the collective activities
of all the employees and conveys the goals of the company as a whole.
Similarly, the concepts of organizational culture and research culture
refer to the general ways in which individuals tend to interact with one
another within an organization or a research field.</p>
<p><strong>Organizations developing AI are
<em>self-organizing</em>.</strong> Although companies have CEOs, these
CEOs are often selected by groups of people, such as board members, and
do not generally dictate every single activity within the company and
how it should be done. People self-select in applying to work at a
company that interests them, and managers usually make decisions
together on who to hire. Employees often come up with their own ideas
for projects and strategies, and then decisions are made collectively on
which ones to pursue. Likewise in academia, researchers investigate
their own questions, keep up to date with the findings of their peers,
and use those insights to inform their research directions, while
experts form committees to decide which projects should be funded. There
is very often no single central entity determining which researchers
should work on which questions.</p>
<p><strong>Both corporate and academic organizations can exhibit
<em>critical points</em>.</strong> Often, a lot of an organization’s
effort is focused on a particularly consequential area or problem until
a big breakthrough is made, representing a tipping point into a new
paradigm. For this reason, research and development often progresses in
the pattern of a <em>punctuated equilibrium</em>, with long periods of
incremental advancements interrupted by windows of rapid advancements,
following important breakthroughs.</p>
<p><strong>Companies and research institutes show multiple forms of
<em>adaptive behavior</em>.</strong> Examples of adaptation include
organizations incorporating new information and technology to update
their strategies and ways of working, and adjusting their research
directions based on new findings. Additionally, they may adapt to the
changing needs of customers and the changing priorities of research
funders, as well as to new government regulations.</p>
<p><strong>Companies and research institutes display <em>distributed
functionality</em>.</strong> While there may be subsystems that focus on
specialized tasks within a company or branches within a research field,
in general, no employee or researcher single-handedly performs a whole
function or advances an area alone. Even if there is just one person
working in a particular niche, they still need to be informed by related
tasks and research performed by others, and usually rely on the work of
support staff. This illustrates <em>partial encoding</em>. There are
also usually more people available to perform tasks than are absolutely
needed, meaning that processes continue over time despite employees and
researchers joining and leaving. This demonstrates <em>redundant
encoding</em>.</p>
<p><strong>There are multiple examples of feedback and nonlinearity in
companies and institutes.</strong> A small disparity in investment into
different projects or research directions may be compounded over time,
with those that receive more initial funding also achieving bigger
results, and therefore receiving even more funding. A small difference
in support for a particular candidate in senior management can be
decisive in whether or not they are selected, and thus have a large
influence over future directions. More broadly, different organizations
may imitate one another’s successes, leading to a concentration of work
in a particular area, while a small initial advantage of one
organization may be amplified over time, allowing it to dominate the
area.</p>
<p><strong>Summary.</strong> The environments in which research and
development occur display the hallmarks of complexity and are therefore
best understood as complex systems. Research organizations and
corporations possess emergent properties including safety culture, which
is paramount for AI safety. Additionally, progress may have critical
points and unfold in a nonlinear way that is difficult to predict. It is
crucial that AI safety strategies are informed by these
possibilities.</p>
<h3 id="worked-example-complex-systems-applied-to-advocacy">Worked
Example: Complex Systems Applied to Advocacy</h3>
<p><strong>The social systems within which AI is deployed are complex
systems.</strong> We find emergence and the hallmarks of complexity in
all social systems, from political structures to economic networks to
society as a whole. To illustrate this more specifically, we will now
focus on the example of policymaking structures and advocacy. This
example is particularly relevant to AI safety, because reducing risks
from AI will need to involve the implementation of policies around its
use. Advocacy will therefore be necessary to promote safety policies and
convince policymakers to adopt them.</p>
<p><strong>Social systems display <em>emergence</em> and
<em>self-organization</em>.</strong> Patterns of governance and
collective decision making, such as democracy, can be considered
emergent properties of social and political systems. Although some
individuals reach positions of power that might seem to centralize
control, social systems are nonetheless partly <em>self-organizing</em>,
in the sense that many individuals interact with one another and make
decisions about whom to support, collectively determining which
candidate is elected. Similarly, people who care about particular causes
<em>self-organize</em> to form advocacy groups and set up grassroots
campaigns. Policymakers interact with each other and various
stakeholders, including advocates, to reach policy decisions.</p>
<p><strong>Advocacy movements have <em>critical points</em> and often
unfold as <em>punctuated equilibria</em>.</strong> Movements advancing
different causes often display <em>critical points</em>, where a
critical level of awareness and support must be reached before
policymakers will pay attention. Social systems may self-organize toward
this critical level of support and maintain it over time. However, the
actual “tipping” from one state into another, wherein policies are
implemented, may be dependent on other external factors, such as whether
there are other urgent issues dominating decision-makers’ attention. For
this reason, advocacy efforts and their results tend to progress as
<em>punctuated equilibria</em>; there may be little apparent change for
a long time, despite sustained work, and then a lot of sudden progress
when momentum builds and the political climate is right for it.</p>
<p><strong>Both advocacy groups and policymaking structures also exhibit
<em>adaptive behavior</em>.</strong> Policymakers must continually adapt
to the fluctuating political landscape and changing concerns of the
public. Similarly, advocacy groups must constantly adjust their
activities to capture the attention of the public and policymakers and
convince them that a particular cause is relevant and important. They
might, for instance, use new technology to innovate an original mode of
campaigning, or link the cause to the prevailing zeitgeist—another
emergent property of social systems.</p>
<p><strong><em>Distributed functionality</em> is evident on multiple
levels in social systems.</strong> The various tasks involved in
advocacy are <em>partially</em> and <em>redundantly</em> encoded across
individuals within groups, allowing campaigns to continue even as people
leave and join them. More broadly still, there are usually several
groups campaigning for any given cause, meaning that the general
function of advocacy is distributed across different organizations.
Decision making is also partially and redundantly encoded among many
policymakers, who interact with one another and various stakeholders to
consider different perspectives and decide on policies.</p>
<p><strong>There are many nonlinear aspects of processes like
advocacy.</strong> There are numerous factors that affect whether or not
an issue is included on a policy agenda. Public interest in a cause, the
influence of opponents of a cause, and the number of other issues
competing for attention are among the many factors that affect the
likelihood that it is considered non-linearly; for instance, opponents
with low influence may permit an issue being discussed, opponents with
medium influence may try and block it from discussed, but opponents with
high influence may permit it being discussed so that they can argue
against it. Additionally, there is a degree of randomness involved in
determining which issues are considered. This means that the policy
progress resulting from a particular campaign does not necessarily
reflect the level of effort put into it, nor how well organized it
was.<p>
Together with <em>distributed functionality</em> and <em>critical
points</em>, this <em>nonlinearity</em> can make it difficult to
evaluate how well a campaign was executed or attribute eventual success.
It might be that earlier efforts were essential in laying the groundwork
or simply maintaining some interest in a cause, even if they did not
yield immediate results. A later campaign might then succeed in
prompting policy-level action, regardless of whether it is particularly
well organized, simply because the political climate becomes
favorable.<p>
Other examples of <em>nonlinearity</em> within advocacy and policymaking
arise from various feedback loops. Since people are influenced by the
opinions of those around them, a small change in the initial level of
support for a policy might be compounded over time, creating momentum
and ultimately tipping the balance as to whether or not it is adopted.
On the other hand, original activities that are designed to be
attention-grabbing may run up against negative feedback loops that
diminish their power over time. Other groups may imitate them, for
instance, so that their novelty wears off through repetition. Opponents
of a cause may also learn to counteract any new approaches that
advocates for it try out. This dynamic was understood by the military
strategist Moltke the Elder, who is reported to have said that “no plan
survives first contact with the enemy”.</p>
<p><strong>Political systems and advocacy groups have <em>scalable
structure</em>.</strong> Political systems usually have a hierarchical
structure with multiple levels of organization, such as councils
responsible for specific regions within a country and politicians
forming a national government to address countrywide issues. Advocacy
groups can also exhibit this kind of structure. There may, for example,
be a campaign manager spearheading efforts, and then many regional
leaders who organize activities at a more local level. This scalable
structure is another indication of complexity.</p>
<p><strong>Summary.</strong> The presence of these hallmarks of
complexity in social and political systems suggests they are best
described within the complex systems paradigm. Additionally, these
observations can offer some insights into how we might approach advocacy
for AI safety, suggesting it is not as simple as developing a good
policy idea and making a convincing argument for it.<p>
Instead, it is likely that successful advocacy over the long term will
be characterized by adaptability to different political circumstances
and changing public attitudes, as well as in response to opponents’
activities. Advocates will need to invest in building and maintaining
relationships with the relevant people and organizations, rather than
just presenting the case for a policy. There may need to be a lot of
work that is not immediately rewarded, but momentum should be maintained
so that advocates are ready to capitalize on moments when the political
climate becomes more favorable. It should also be understood that it
might not be possible to attribute success in any obvious way.</p>
<h3 id="it-is-difficult-to-foresee-how-the-use-of-ai-will-unfold">It Is
Difficult to Foresee How the Use of AI Will Unfold</h3>
<p><strong>Complex social systems mean the eventual impact of AI is hard
to predict.</strong> As discussed earlier, the behavior of complex
systems can be difficult to predict for many reasons, such as the
appearance of unanticipated emergent properties and feedback loops
amplifying small changes in initial conditions. This is compounded if a
system is new to us; we may be able to predict certain high-level
behaviors of complex systems we are familiar with and have a lot of
historical data on, such as weather patterns and beehives, but AI
systems are relatively new. Additionally, the deployment of AI within
society represents a case of nested complexity, where complex systems
are embedded within one another. This vastly increases the range of
potential interactions and the number of ways in which the systems can
co-evolve. As a result, it is difficult to predict all the ways in which
AI might be used and what its eventual impact will be.<p>
While this technology could have many positive effects, there is also
potential for interactions to have negative consequences. This is
especially true if AI is deployed in ways that enable it to affect
actions in the world; for example, if it is put in charge of automated
decision-making processes.</p>

<br>
<br>
<h3>References</h3>
<div id="refs" class="references csl-bib-body" data-entry-spacing="0"
role="list">
<div id="ref-cilliers2014complexity" class="csl-entry" role="listitem">
<div class="csl-left-margin">[1] P.
Cilliers and D. Spurrett, <span>“Complexity and post-modernism:
Understanding complex systems,”</span> <em>South African Journal of
Philosophy</em>, vol. 18, pp. 258–274, Sep. 2014, doi: <a
href="https://doi.org/10.1080/02580136.1999.10878187">10.1080/02580136.1999.10878187</a>.</div>
</div>
<div id="ref-power2022grokking" class="csl-entry" role="listitem">
<div class="csl-left-margin">[2] A.
Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra,
<span>“Grokking: Generalization beyond overfitting on small algorithmic
datasets.”</span> 2022. Available: <a
href="https://arxiv.org/abs/2201.02177">https://arxiv.org/abs/2201.02177</a></div>
</div>
<div id="ref-hoffmann2022training" class="csl-entry" role="listitem">
<div class="csl-left-margin">[3] J.
Hoffmann <em>et al.</em>, <span>“Training compute-optimal large language
models.”</span> 2022. Available: <a
href="https://arxiv.org/abs/2203.15556">https://arxiv.org/abs/2203.15556</a></div>
</div>
<div id="ref-zhao2021response" class="csl-entry" role="listitem">
<div class="csl-left-margin">[4] H.
Zhao <em>et al.</em>, <span>“Response mechanisms to heat stress in
bees,”</span> <em>Apidologie</em>, vol. 52, Jan. 2021, doi: <a
href="https://doi.org/10.1007/s13592-020-00830-w">10.1007/s13592-020-00830-w</a>.</div>
</div>
</div>