Let’s pick up where the previous paper left off. In 2012, a convolutional neural network (CNN) crushes the ImageNet competition. Why does convolution work so well on images? Because an image is a spatial object: a rhinoceros is still a rhinoceros whether it stands on the left or the right of the photo. The CNN exploits this property by sliding small filters across the image, like a magnifying glass sweeping over a photograph.

Language poses a different problem. A sentence is a sequence: a temporal, ordered object of variable length.

• “The dog bites the man” and “The man bites the dog” contain the same words, but order changes everything.
• The meaning of a word can depend on words that are far away: in “The key that I left on the kitchen table last night has disappeared,” the verb agrees with “the key,” a dozen words earlier. This is called a long-range dependency.
• A sentence can be 3 words or 300: the network must accept inputs of variable size.

Remember the formula from the previous post: for connectionists, the goal is to “put the world into a vector.” Two questions immediately arise, and the whole story that follows is a series of answers to them:

1. How do you turn a word into a vector? (the representation problem)
2. How do you process a variable-length sequence of vectors? (the architecture problem)

The word as a number (and why it is not enough)

The naive approach gives each dictionary word a number: “cat” = 4,812, “dog” = 1,953. Technically, one uses a so-called one-hot vector: a huge vector of zeros with a single 1 at the word’s position. The problem? In this representation, “cat” is no closer to “dog” than to “umbrella.” All resemblance between words is lost. It is a symbolic representation in the sense of the previous post: each word is a discrete symbol, identical to or different from another, with no internal structure.

Word2vec (2013): meaning as neighborhood

The solution was already hinted at in Cardon’s paper: word2vec (Mikolov et al., 2013). The idea rests on an old linguists’ intuition, summed up by John R. Firth in 1957: “You shall know a word by the company it keeps.” “Cat” and “dog” appear in similar contexts (“feed the ___”, “the ___ is sleeping”); they should therefore receive nearby vectors.

Word2vec trains a small neural network on a simple task: guess a word from its neighbors (or the reverse). The network is not the point; what you keep are the vectors learned along the way, called embeddings. Each word becomes a point in a space of a few hundred dimensions, and this space has astonishing properties:

• words that are close in meaning are close in distance;
• some directions of the space mean something: the famous computation king − man + woman ≈ queen actually works.

Now reread that sentence from Cardon’s paper: semantic proximity is not deduced from a symbolic categorization, but induced from statistical neighborhoods. This is exactly the symbolic-to-connectionist reversal, applied to the meaning of words. No linguist wrote a rule; meaning emerged from data. The first of our two problems — representing words — is solved.

But word2vec has a serious limitation: each word gets one single, fixed vector. Yet “bank” does not mean the same thing in “I called my bank” and “I sat on the river bank.” What we would need are contextual representations that change with the sentence. Keep this limitation in mind: the Transformer is, among other things, the machine that will blow it away.

The RNN: a loop over time (1986–1990)

There remained the second problem: processing a variable-length sequence. The historical answer is the recurrent neural network (RNN), popularized by Jeffrey Elman in 1990 and already present in the work of the PDP group of Rumelhart and Hinton (1986) — the same group as in Cardon’s paper.

The idea is elegant: the network reads the sequence one word at a time, left to right, and maintains a hidden state — a vector acting as working memory. At each word, the network combines what it reads with what it remembers, and updates its memory:

memory(t) = f( memory(t−1), word(t) )

Picture someone reading a sentence under their breath while keeping a mental summary they revise at every word. That is an RNN. The architecture naturally accepts sentences of any length: you simply loop for more or fewer steps.

The vanishing gradient problem

In practice, simple RNNs have a crippling flaw. Remember backpropagation: to learn, the error is propagated backwards through the network. In an RNN, “backwards” means back through time, across as many steps as there are words. At each step, the error signal is multiplied by coefficients; over a long sentence it gets multiplied dozens of times by numbers that are often smaller than 1… and it melts like snow in the sun. This is the famous vanishing gradient problem, identified notably by Sepp Hochreiter as early as 1991.

The concrete consequence: the network cannot learn long-range dependencies. By the time it reaches the verb “has disappeared,” it has “forgotten” the key at the beginning of the sentence.

The LSTM (1997): a memory with gates

The most celebrated solution arrives in 1997: the LSTM (Long Short-Term Memory), by Sepp Hochreiter and Jürgen Schmidhuber. The idea: equip the memory cell with learned gates — small mechanisms that decide, at each step, what to write into memory, what to forget, and what to read. A kind of whiteboard with a doorkeeper choosing what gets noted and what gets erased. Thanks to an internal “conveyor belt” where information circulates almost untouched, the gradient survives much longer.

LSTMs (and their simplified cousin, the GRU, 2014) would dominate natural language processing for twenty years. They are a perfect example of what Cardon’s paper calls the work on hyper-parameters and architecture: you do not dictate grammar rules to the network — you sculpt its structure so it can learn what it needs.

Around 2014–2015, boosted by GPUs and large corpora, LSTMs power the speech recognition in your phone and the first neural versions of Google Translate. Connectionism, victorious over images in 2012, is winning over language. But it is precisely by pushing LSTMs to their limits that the missing link will be discovered.

Encoder–decoder (2014)

Machine translation is THE queen of tasks, because it demands everything: understanding one sequence and producing another, of a different length. In 2014, two teams (Sutskever, Vinyals and Le at Google; Cho and Bengio in Montreal) propose the seq2seq (sequence-to-sequence) architecture, made of two RNNs:

• an encoder reads the source sentence (“Le chat dort”) and compresses it into a single vector — a numerical summary of the whole sentence, sometimes called a “thought vector”;
• a decoder starts from this vector and generates the target sentence word by word (“The,” then “cat,” then “sleeps”), each produced word being fed back in to produce the next.

Note this word-by-word generation mechanism, each word conditioned on the previous ones: it is called auto-regressive, and it is exactly how ChatGPT or Claude write their answers today. This point from 2014 has never changed.

The bottleneck

But seq2seq has an obvious Achilles heel: the entire source sentence, whether 5 or 60 words long, must fit into a single fixed-size vector. It is like asking a translator to read a whole paragraph, close the book, then translate from memory without ever reopening it. On long sentences, quality collapses. Researchers call this the information bottleneck.

The solution will give its name to the paper you are about to read.

In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio (him again — find him in the connectionist core of Cardon’s figure 2) publish an idea that changes everything: what if, instead of closing the book, the decoder could keep it open?

Concretely: the encoder no longer produces a single vector, but keeps one vector per word of the source sentence. Then, for every word it generates, the decoder computes a relevance score between what it is currently doing and each of the source words. These scores, turned into percentages (via a function called softmax), are used to build a weighted average of the source vectors: the decoder “focuses” on the words that are useful at that instant. To produce “sleeps,” it looks mostly at “dort.” This mechanism is called attention.

Three things to remember, because they directly prepare your reading of Vaswani et al.:

1. Attention is learned, not programmed. Nobody wrote a French–English alignment rule: the network discovers by itself that “dort” explains “sleeps.” Once again, the inductive move described by Cardon — empty the calculator, let the world provide the structure.
2. Attention is soft: it is not a binary choice but a weighting, therefore it is differentiable, therefore backprop applies to it.
3. Attention creates shortcuts: every generated word is directly connected to every source word, without going through the fragile chain of recurrent memory. Long-range dependencies become connections… of distance 1.

Between 2015 and 2017, the “LSTM + attention” recipe becomes the world state of the art in translation (Google deploys it at the end of 2016). But an irritation grows among engineers, and it is a hardware one — remember the role of GPUs in Cardon’s story. An RNN reads a sentence word after word: computing word 50 must wait for word 49. Yet GPUs are massively parallel machines, built to perform millions of operations at the same time. Recurrence wastes the hardware and forbids training on truly gigantic corpora in reasonable time.

Hence a question asked ever more insistently in the labs: now that we have attention, what is recurrence still for?

The answer from eight Google researchers (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin) fits in their slightly provocative title: attention is all you need. Their architecture, the Transformer, purely and simply removes recurrence (and convolution). Here, as a preview, are the ideas you will meet in the paper — consider this section your reading map.

Self-attention

Until now, attention connected the decoder to the encoder (target to source). The Transformer’s stroke of genius is to apply it inside a single sentence: every word looks at every other word of its own sentence — including itself — to build its representation. In “The animal didn’t cross the street because it was tired,” the word “it” learns to attend to “the animal.” In one operation, each word enriches its meaning with its whole context.

Notice what this solves: a word’s vector is no longer fixed as in word2vec — it is contextual, recomputed for every sentence. “Bank” will not have the same representation at the counter and by the river.

Query, Key, Value: the library metaphor

The paper formalizes attention with three vectors per word, obtained through three learned transformations: a Query, a Key, and a Value. The classic image: in a library, you arrive with a question (your query), you compare it to the labels on the spines of the books (the keys), and you leave with the content of the relevant books (the values), in proportion to their relevance. Mathematically: dot products between Q and K → scores → softmax → weighted average of the V. That is equation (1) of the paper, the only truly indispensable one:

Attention(Q, K, V) = softmax(QKᵀ / √d) · V

The √d in the denominator is a mere numerical safeguard (keeping the scores from growing too large). Everything else in the paper is engineering around this formula.

Multi-head, position, and the full architecture

• Multi-head attention: rather than one attention, you run 8 (or 16, or 96…) in parallel, each with its own Q, K, V. Each “head” specializes — one tracks syntax, another coreference, and so on — without anyone asking it to. Emergent behavior, again.
• Positional encoding: removing recurrence has a cost — the network no longer knows in what order the words are! (Attention is a set computation, insensitive to order.) The fix: add to each embedding a small vector encoding its position, built from sines and cosines of varying frequencies. Do not drown in the formulas: just remember that we re-inject the order we lost.
• Stacking: one Transformer “block” = self-attention + a small classical network (feed-forward), with two training stabilizers (residual connections and normalization). These blocks are stacked: 6 in the 2017 paper, 96 and more in today’s LLMs. The original paper keeps the encoder–decoder structure inherited from seq2seq, since it targets translation.
• Parallelism: all words are processed at the same time. No more sequential waiting: the Transformer fits GPUs perfectly. This is the decisive argument — less training time, more data ingested. Reread Cardon: the connectionist victory has always been as much about hardware as about ideas.

The 2017 paper is about translation. How do we get to ChatGPT? Through an idea of disarming simplicity: next-word prediction.

Take a text, hide the next word, ask the model to guess it, correct it with backprop, repeat — trillions of times, over the whole web. No human labels are needed: the text is its own correction. This is called self-supervised learning. Think back to Cardon’s “horizon”: here, the horizon of the computation (the next word) is supplied by the world itself, for free, at unlimited scale. It is the ultimate cybernetic loop.

The key milestones:

• 2018 — GPT-1 (OpenAI): keep only the decoder of the Transformer, trained to predict the next word, then fine-tuned on specific tasks. BERT (Google) makes the opposite choice — keep only the encoder, trained to guess masked words — and crushes the comprehension benchmarks. The two descendants of the 2017 paper divide up the world.
• 2019 — GPT-2: same recipe, ×10 on size (1.5 billion parameters). Surprise: the model can summarize, translate, answer questions without having been trained to — simply because predicting the next word across the whole Internet forces it to learn a bit of everything.
• 2020 — GPT-3 and the scaling laws (Kaplan et al.): performance grows in a regular, predictable way with three ingredients — parameters, data, compute. The message: bigger is better, and nobody sees the ceiling yet. GPT-3 (175 billion parameters) reveals in-context learning: give it two examples inside the question (the prompt), and it picks up the pattern without any retraining.
• 2022 — ChatGPT and RLHF: a raw LLM completes text; it does not “answer.” It is aligned in two stages: fine-tuning on dialogues written by humans, then reinforcement learning from human feedback (RLHF) — annotators rank answers, the model learns to aim for the best-ranked ones. Note the historical irony: the horizon of the computation, expelled from symbolic rules, comes back through the door of human preferences.
• 2023–today: the era of open models (Meta’s Llama, Mistral, Alibaba’s Qwen), of multimodal models (text + image + sound), and of the Transformer’s extension to… image and video generation. Modern diffusion models (Stable Diffusion 3, Flux, Qwen-Image, Sora, Wan) have also replaced their old internal networks with Transformers (the so-called DiT, Diffusion Transformer, architecture). The 2017 architecture has become the universal architecture of connectionism.

One last thing, to close the loop with Cardon. The final table of his paper describes deep learning as: world = vectors of massive data, calculator = deep network, horizon = error optimization on an objective. The LLM is its most extreme culmination: the world is all the text ever written, the calculator is a Transformer with hundreds of billions of coefficients, and the horizon fits in three words — predict the next word. That capacities for reasoning, translation and dialogue emerge from such a poor objective is perhaps the most beautiful posthumous victory of the connectionist camp — and the philosophical question remains wide open, exactly where Cardon left it: what does a machine “understand” when its entire thought is, in Hinton’s phrase, “a big vector of neural activity”?

The paper is 11 dense pages. Reading advice:

1. Read in this order: the abstract → the introduction (§1) → figure 1 (the architecture — keep it in view at all times) → §3.2 (attention, the heart of the paper) → the conclusion. The rest (§5–6, training details and translation results) can be skimmed.
2. Do not get stuck on the math. Only one equation matters (Attention(Q,K,V), equation 1), and you already know its intuition: query → labels → weighted contents.
3. Spot the words you now own: recurrent, sequential computation, long-range dependencies, parallelizable — you now know why they are there and what the paper is fighting against.
4. A question to hold onto as you close the paper: why does removing recurrence make the addition of positional encoding mandatory?

Attention — A learned mechanism that computes, for a given element, relevance weights over a set of other elements, then takes their weighted average. Enables direct connections between distant words.

Auto-regressive — Generation mode in which each new word is produced conditioned on all previous ones, one by one. This is how GPT, Claude and Qwen “write.”

Backpropagation — The algorithm (1986) that propagates the error from output to input to adjust the network’s weights.

BERT (2018) — Encoder-only model, trained to guess masked words; excellent at understanding text (classification, search).

Context window — The maximum number of tokens the model can consider at once. A major practical limit of LLMs.

Decoder — The half of the Transformer that generates the output sequence. GPT and most current LLMs are decoder-only.

Embedding — Representation of an object (word, image, molecule…) as a dense vector in a space where geometric proximity reflects similarity.

Encoder — The half of the Transformer that reads and represents the input sequence.

GRU / LSTM — Gated recurrent cells (1997 for the LSTM) that protect information over long stretches.

Hyper-parameters — Architecture and training choices set by humans (number of layers, heads, learning rate…), as opposed to learned parameters.

In-context learning — A model’s ability (from GPT-3 onwards) to pick up a task from a few examples placed directly in the prompt, without retraining.

LLM (Large Language Model) — A Transformer (usually decoder-only) with billions of parameters, trained by next-word prediction on immense corpora.

Long-range dependency — A grammatical or semantic link between words that are far apart in a sentence. Achilles heel of RNNs, strong point of attention.

Multi-head attention — Running several independent attentions (“heads”) in parallel, each free to specialize in one type of relation.

Parameters (weights) — The coefficients adjusted during learning. GPT-3: 175 billion. These are what you download when you fetch a model from Hugging Face.

Positional encoding — Vectors added to the embeddings to re-inject word order, which attention alone ignores.

Prompt — The input text given to the LLM; since GPT-3, you “program” the model in natural language through the prompt.

Query / Key / Value (Q, K, V) — The three learned projections of each word used in the attention computation: the question asked, the label compared, the content retrieved.

RLHF — Reinforcement Learning from Human Feedback: aligning an LLM with human preferences via reinforcement (the basis of ChatGPT, 2022).

RNN (recurrent neural network) — A network that processes a sequence step by step while maintaining a hidden state (memory). Dominant in NLP from 1990 to 2017.

Scaling laws — Empirical relations (2020) showing that LLM performance improves predictably with model size, data and compute.

Self-supervised learning — Learning without human labels: the data provides its own target (e.g., the next word of a text).

Seq2seq — The encoder–decoder architecture (2014) turning one sequence into another; the framework of neural translation and the direct ancestor of the Transformer.

Softmax — The function that converts a list of scores into a probability distribution (percentages summing to 100%). Used inside attention and to pick the next word — it is on this function that the temperature parameter operates.

Token — The elementary unit of text a model manipulates (often a word fragment, ~¾ of a word on average).

Transformer (2017) — Architecture based solely on attention (no recurrence, no convolution), massively parallelizable; the foundation of all LLMs and, by now, of diffusion models (DiT).

Word2vec (2013) — A method producing static word embeddings from their contexts of occurrence; the ancestor of the Transformer’s contextual representations.

This is the first post in a series of lectures intended to bring a reader with no formal background in machine learning up to the state of the field as it stands today. The trajectory runs from the artificial neuron of 1943 to the systems currently deployed in research and industry: deep convolutional networks, word and image embeddings, transformer architectures, large language models, vision-language models, mixture-of-experts routing, retrieval-augmented generation, and autonomous agents. Each lecture pairs a primary reading with a preparatory guide of this kind, plus a glossary and a set of questions to keep in mind while reading.

The series begins somewhere unexpected: a forty-page article written by sociologists. This is deliberate. Before diving into LLM and agents, it is worth understanding where all of it came from, and above all understanding that none of these techniques was obvious or inevitable. Modern artificial intelligence is the outcome of a seventy-year scientific battle, with winners, losers, public humiliations, wilderness years, and spectacular comebacks. That is the story told by Dominique Cardon, Jean-Philippe Cointet and Antoine Mazières.

The article opens on a scene that reads like a western: October 2012, a scientific conference, a largely unknown student walks on stage and announces a result that pulverises ten years of work by an entire research community. The room is stunned. That scene, the earthquake of 2012, is the article’s destination rather than its starting point. Everything else explains how the field arrived there.

The thesis in one sentence: the history of AI is a war between two visions of the intelligent machine, a machine that is given rules and a machine that learns from examples, and after fifty years of symbolic domination it is the connectionists, long mocked and marginalised, who won.

Suppose you want to build a machine that recognises cats in photographs.

The symbolic approach says: sit down and write the rules. A cat has two pointed ears, whiskers, four legs; IF pointed ears AND whiskers THEN cat. This is intelligence as reasoning, the manipulation of symbols such as “ear” and “cat” by means of logic. It is intuitive, explainable and elegant, and it is how computers have always been programmed: the human writes the program, the machine executes it.

The connectionist approach says: rules are hopeless. A cat seen from behind, at night, half hidden behind a curtain, matches no rule at all. Instead, show the machine a hundred thousand photographs labelled “cat” or “not cat”, and let a network of small interconnected computing units, artificial neurons very loosely inspired by the brain, adjust the strength of its own connections until its answers are good. Nobody writes a rule; the rule emerges from the examples. This is intelligence as learning.

The strength of the article is that it shows this technical choice to be simultaneously a philosophical choice (is thinking reasoning, or perceiving?), an economic choice (who receives the funding?) and a social choice (which research communities dominate?).

The article’s central image, figure 1, is worth studying closely:

• Classical machine, hypothetico-deductive: inputs plus program produce outputs. The human supplies the program.
• Inductive machine: inputs plus outputs produce the program. The human supplies examples, photographs paired with correct answers, and the program is what comes out of the machine.

This reversal is the single most important idea in the entire series. If you retain one thing from Lecture 1, retain that one.

The authors analyse each era of AI using three notions, announced early and reused all the way to the final synthesis table (table 1, page 28, an excellent summary of the whole article):

• The world: what enters the machine. Data? Rules? Expert knowledge? An environment?
• The calculator: what does the processing. A logic engine? A neural network? A black box?
• The horizon: the goal of the computation. Solving a problem? Minimising an error? Imitating examples?

Their key formula is slightly cryptic on first reading but becomes clear by the end. The symbolists wanted to put everything into the calculator, both the world and the goal, whereas the connectionists empty the calculator so that the world gives itself its own horizon. In other words, massive data supplies both the material and the correction: the labelled examples are what tell the network when it is wrong.

The article follows four major periods. The characters introduced here reappear throughout the series.

Cybernetics and the first connectionism, 1943 to 1969. The age of the pioneers. Warren McCulloch and Walter Pitts invent the artificial neuron in 1943. Norbert Wiener founds cybernetics, the science of machines that self-correct through feedback. Frank Rosenblatt builds the Perceptron in 1957, the first machine that learns to recognise patterns; the press goes wild and conscious machines are promised.

Symbolic AI, 1956 to 1970, then expert systems in the 1980s. In 1956 John McCarthy and Marvin Minsky coin the term “artificial intelligence”, explicitly against cybernetics. With Herbert Simon and Allen Newell they capture the bulk of military funding and impose the symbolic vision. In 1969 Minsky publishes a book that proves neural networks have no future; this is the excommunication. Funding dries up, Rosenblatt dies in 1971, and connectionism enters a long winter. In the 1980s symbolic AI enjoys a second wind with expert systems, thousands of IF-THEN rules extracted from human doctors, geologists and engineers, before a second collapse.

The return of the neurons, 1986 to 2010. A small group of holdouts, Geoffrey Hinton, Yann LeCun and Yoshua Bengio, later nicknamed the neural conspiracy, keeps the flame alive. In 1986 backpropagation finally makes it possible to train multi-layer networks. In 1989 LeCun gets a network to read postal codes, the first industrial application. But the years 1995 to 2007 are a colossal winter of rejected papers, mockery and isolation. The article contains excellent first-hand testimony from French researchers about that period.

The triumph of deep learning, 2010 onward. Three ingredients converge: massive data, meaning the web and ImageNet with its fourteen million hand-labelled images; GPUs, the graphics processors built for video games and perfectly suited to the massively parallel computations of neural networks; and the algorithms patiently matured during the winter. Then 2012, and the earthquake. Since then, domain after domain, image, speech and text, deep networks have swept everything away.

It is an academic sociology article: dense, but well written and full of anecdotes. Some practical advice.

Take your time with the opening, the 2012 narrative on pages 2 and 3. The entire article is contained in that scene.

Lean on the figures. Figure 1, the two machines, and table 1, the four ages, are your two anchors. Figure 3, the timeline, shows the shifting dominations visually. Figures 4 and 5 give a first look at an artificial neuron and at backpropagation.

Do not get stuck on the details of the Web of Science queries in notes 5 and 6, or on the philosophical references to Fodor, Smolensky and the computational theory of mind. Grasp the general idea and move on. The section on convexity is the most technical; the glossary below gives the minimum needed to get through it.

Savour the interview quotations, set in indented blocks in spoken register. That is where the history comes alive.

Keep three questions in mind while reading:

1. For each era, what is in the world, the calculator and the horizon?
2. Why did the connectionists win when they did, rather than in 1960? The hint is that it was not the ideas that changed.
3. The title speaks of striking back. Who was humiliated, when, and by whom?

Budget two to two and a half hours of attentive reading. It is the longest reading of the series and probably the one that will stay with you.

Terms are listed roughly in their order of appearance in the article.

Computer vision. Research field aiming to make machines see: recognising objects, faces and scenes in images.

ImageNet. A database of fourteen million images across roughly twenty-one thousand categories, hand-labelled by thousands of micro-workers. It serves as an annual competition, and it is the benchmark on which the 2012 earthquake took place.

Benchmark. A standardised test set allowing objective comparison between the performance of different methods.

GPU, graphics processing unit. A processor originally designed for video games, capable of performing millions of simple operations in parallel, which is exactly what a neural network requires.

Deep learning. Neural networks with many layers, hence deep. The term was coined by Hinton in 2006, partly to escape the poor reputation of the word connectionism.

Machine learning. The family of methods in which a machine learns from examples instead of being explicitly programmed. Deep learning is one branch of it.

Parameters, also weights or coefficients. The numbers adjusted during learning, encoding the strength of the connections between neurons. A hundred million parameters means a hundred million small knobs tuned automatically.

Symbolic AI, also GOFAI, Good Old-Fashioned AI. The rules-and-logic approach. Thinking equals manipulating symbols.

Connectionism. The neural network approach. Thinking equals parallel, distributed computation by simple units, with intelligence emerging from the connections.

Cybernetics. The science founded by Norbert Wiener in 1948, studying systems, whether machines or organisms, that regulate themselves through feedback. The direct ancestor of connectionism.

Feedback. Reinjecting the measured output error as a new input so that the system corrects itself. The thermostat is the canonical example. It is the founding principle of cybernetics and, in a sense, of all machine learning.

Black box. A system whose inputs and outputs are observable but whose internal workings are not understood. A recurring criticism of neural networks, and a property their defenders have claimed proudly since the cybernetic era.

Formal neuron. An ultra-simplified mathematical model of a neuron, due to McCulloch and Pitts in 1943. It sums its inputs weighted by weights and activates if the sum exceeds a threshold. Figure 4 shows that this amounts to three operations, no more.

Perceptron. The first learning machine built on a neural network, developed by Rosenblatt between 1957 and 1961, funded by the US Navy and designed for image recognition. It is the emblem of early connectionism and the target of Minsky’s attack.

Layers and hidden layers. Neurons are organised in tiers: an input layer holding the data, intermediate layers called hidden where the work happens, and an output layer holding the answer. Minsky’s 1969 book attacked a single-layer perceptron.

XOR, exclusive OR. The elementary logical function “A or B, but not both”, which a single-layer perceptron cannot learn. This was Minsky and Papert’s decisive argument for burying connectionism. Multiple layers solve the problem.

AI winter. A period of collapse in funding and credibility following excessive promises. There have been two general ones, in the early 1970s and the late 1980s, plus the specifically connectionist winters of 1969 to 1986 and 1995 to 2007.

Expert system. A 1980s program encoding human expert knowledge as thousands of IF-THEN rules, MYCIN for medical diagnosis being the standard example, driven by an inference engine that decides which rule to apply when. Expert systems mark both the peak and the collapse of symbolic AI.

Knowledge base. The stock of rules and facts held by an expert system, to be contrasted with a dataset: a knowledge base contains intelligible rules, a dataset contains raw examples.

Backpropagation, or backprop. The algorithm popularised in 1986 by Rumelhart, Hinton and Williams that makes learning possible. The error is measured at the output, then propagated backwards layer by layer to adjust each weight in the right direction. See figure 5. This is the central mechanism of all deep learning.

Loss function. The number measuring how wrong the network is. All of learning consists in minimising it.

Gradient descent. The minimisation method: compute the slope of the loss function and take a small step downhill, then repeat millions of times. The usual image is walking down a mountain in fog by following the slope underfoot. Stochastic means the slope is estimated on a small sample of the data at a time, which is faster.

Convolution and convolutional networks, CNNs. A technique invented by LeCun in 1989 for images. Rather than connecting every pixel to every neuron, small filters are slid across the image to detect local patterns such as edges and corners independently of their position. This is the architecture behind AlexNet in 2012.

Feature engineering. The now largely extinct art of hand-programming the relevant characteristics of the data, detecting edges, corners and contrasts, before passing them to an algorithm. Deep learning made it obsolete because the network discovers its own features. An entire scientific community lost its object of research this way, which accounts for the bitterness audible in some of the testimony in the article.

End-to-end. Processing raw data, the pixels, all the way to the final answer, “cat”, within a single network, with no intermediate step programmed by a human.

SVM, support vector machines, and kernel methods. A rival learning method dating from 1992, mathematically elegant and dominant between roughly 1995 and 2010. It was the great internal adversary within the learning camp, and the duel between SVMs and neural networks structures an entire section of the article.

Convexity. A mathematical criticism levelled at neural networks. A convex function is shaped like a bowl, with a single hollow, so gradient descent is guaranteed to reach the bottom, the global minimum. A neural network’s loss function is instead a landscape of mountains with countless valleys, local minima, with no guarantee of finding the best one. The SVM mathematicians treated this as a fatal flaw; LeCun’s answer amounted to saying that the theoretical guarantee matters less than the fact that it works better in practice, and experience proved him right.

Overfitting. When a network learns its training examples by heart instead of extracting general regularities, performing excellently on known data and poorly on new data. Dropout, the random switching-off of neurons during training, is one countermeasure mentioned in the article.

Hyper-parameters. All the architectural choices fixed by the human before learning begins: the number of layers, the number of neurons, the learning rate. These stand in contrast to parameters, which are learned automatically. The article shows that human labour does not disappear, it shifts from writing rules to setting these values.

Dataset. A collection of examples used to train and test a model, often as input-output pairs such as a photograph and its label.

Labelled data. Examples accompanied by the correct answer, supplied by humans. This is the fuel of supervised learning.

Crowdsourcing and Mechanical Turk. Micro-work platforms where thousands of people, paid by the task, label data by drawing a box around the dog or typing the spoken word. This is the human face, invisible and poorly paid, of so-called raw data, and ImageNet is its product.

Embedding, or vector. The transformation of an object, a word, an image, a social network, into a list of numbers so that a network can compute with it. The article cites word2vec and LeCun’s formula for putting the world into a vector, world2vec. This is the direct bridge to Lecture 2.

Induction and deduction. Deduction starts from general rules and applies them to particular cases, which is the symbolic approach. Induction starts from particular cases, the examples, and extracts a general rule, which is the connectionist approach. The article’s full subtitle, “the invention of inductive machines”, says the essential.

Watch these after finishing the article to go further.

• The moment we stopped understanding AI [AlexNet], Welch Labs, about eighteen minutes, in English. The 2012 earthquake seen from inside the network, with excellent visualisations of AlexNet’s layers, and the perfect complement to the article’s opening scene.
• Heroes of Deep Learning: Andrew Ng interviews Geoffrey Hinton, followed by the LeCun interview in the same series, in English. Both are cited in the article’s own footnotes, notes 3 and 23. Hearing Hinton and LeCun recount the connectionist winter in their own voices is worth the time.
• Le deep learning, ScienceEtonnante (David Louapre), in French. Neural networks, backpropagation and convolution explained in twenty minutes.

Cardon, D., Cointet, J.-P. and Mazières, A. (2018). Neurons spike back. The invention of inductive machines and the artificial intelligence controversy. Réseaux, 211(5), 173–220.