LEARN, BUILD, APPLY

This is the first post in a series of lectures intended to bring a reader with no formal background in machine learning up to the state of the field as it stands today. The trajectory runs from the artificial neuron of 1943 to the systems currently deployed in research and industry: deep convolutional networks, word and image embeddings, transformer architectures, large language models, vision-language models, mixture-of-experts routing, retrieval-augmented generation, and autonomous agents. Each lecture pairs a primary reading with a preparatory guide of this kind, plus a glossary and a set of questions to keep in mind while reading.

The series begins somewhere unexpected: a forty-page article written by sociologists. This is deliberate. Before diving into LLM and agents, it is worth understanding where all of it came from, and above all understanding that none of these techniques was obvious or inevitable. Modern artificial intelligence is the outcome of a seventy-year scientific battle, with winners, losers, public humiliations, wilderness years, and spectacular comebacks. That is the story told by Dominique Cardon, Jean-Philippe Cointet and Antoine Mazières.

The article opens on a scene that reads like a western: October 2012, a scientific conference, a largely unknown student walks on stage and announces a result that pulverises ten years of work by an entire research community. The room is stunned. That scene, the earthquake of 2012, is the article’s destination rather than its starting point. Everything else explains how the field arrived there.

The thesis in one sentence: the history of AI is a war between two visions of the intelligent machine, a machine that is given rules and a machine that learns from examples, and after fifty years of symbolic domination it is the connectionists, long mocked and marginalised, who won.

Suppose you want to build a machine that recognises cats in photographs.

The symbolic approach says: sit down and write the rules. A cat has two pointed ears, whiskers, four legs; IF pointed ears AND whiskers THEN cat. This is intelligence as reasoning, the manipulation of symbols such as “ear” and “cat” by means of logic. It is intuitive, explainable and elegant, and it is how computers have always been programmed: the human writes the program, the machine executes it.

The connectionist approach says: rules are hopeless. A cat seen from behind, at night, half hidden behind a curtain, matches no rule at all. Instead, show the machine a hundred thousand photographs labelled “cat” or “not cat”, and let a network of small interconnected computing units, artificial neurons very loosely inspired by the brain, adjust the strength of its own connections until its answers are good. Nobody writes a rule; the rule emerges from the examples. This is intelligence as learning.

The strength of the article is that it shows this technical choice to be simultaneously a philosophical choice (is thinking reasoning, or perceiving?), an economic choice (who receives the funding?) and a social choice (which research communities dominate?).

The article’s central image, figure 1, is worth studying closely:

• Classical machine, hypothetico-deductive: inputs plus program produce outputs. The human supplies the program.
• Inductive machine: inputs plus outputs produce the program. The human supplies examples, photographs paired with correct answers, and the program is what comes out of the machine.

This reversal is the single most important idea in the entire series. If you retain one thing from Lecture 1, retain that one.

The authors analyse each era of AI using three notions, announced early and reused all the way to the final synthesis table (table 1, page 28, an excellent summary of the whole article):

• The world: what enters the machine. Data? Rules? Expert knowledge? An environment?
• The calculator: what does the processing. A logic engine? A neural network? A black box?
• The horizon: the goal of the computation. Solving a problem? Minimising an error? Imitating examples?

Their key formula is slightly cryptic on first reading but becomes clear by the end. The symbolists wanted to put everything into the calculator, both the world and the goal, whereas the connectionists empty the calculator so that the world gives itself its own horizon. In other words, massive data supplies both the material and the correction: the labelled examples are what tell the network when it is wrong.

The article follows four major periods. The characters introduced here reappear throughout the series.

Cybernetics and the first connectionism, 1943 to 1969. The age of the pioneers. Warren McCulloch and Walter Pitts invent the artificial neuron in 1943. Norbert Wiener founds cybernetics, the science of machines that self-correct through feedback. Frank Rosenblatt builds the Perceptron in 1957, the first machine that learns to recognise patterns; the press goes wild and conscious machines are promised.

Symbolic AI, 1956 to 1970, then expert systems in the 1980s. In 1956 John McCarthy and Marvin Minsky coin the term “artificial intelligence”, explicitly against cybernetics. With Herbert Simon and Allen Newell they capture the bulk of military funding and impose the symbolic vision. In 1969 Minsky publishes a book that proves neural networks have no future; this is the excommunication. Funding dries up, Rosenblatt dies in 1971, and connectionism enters a long winter. In the 1980s symbolic AI enjoys a second wind with expert systems, thousands of IF-THEN rules extracted from human doctors, geologists and engineers, before a second collapse.

The return of the neurons, 1986 to 2010. A small group of holdouts, Geoffrey Hinton, Yann LeCun and Yoshua Bengio, later nicknamed the neural conspiracy, keeps the flame alive. In 1986 backpropagation finally makes it possible to train multi-layer networks. In 1989 LeCun gets a network to read postal codes, the first industrial application. But the years 1995 to 2007 are a colossal winter of rejected papers, mockery and isolation. The article contains excellent first-hand testimony from French researchers about that period.

The triumph of deep learning, 2010 onward. Three ingredients converge: massive data, meaning the web and ImageNet with its fourteen million hand-labelled images; GPUs, the graphics processors built for video games and perfectly suited to the massively parallel computations of neural networks; and the algorithms patiently matured during the winter. Then 2012, and the earthquake. Since then, domain after domain, image, speech and text, deep networks have swept everything away.

It is an academic sociology article: dense, but well written and full of anecdotes. Some practical advice.

Take your time with the opening, the 2012 narrative on pages 2 and 3. The entire article is contained in that scene.

Lean on the figures. Figure 1, the two machines, and table 1, the four ages, are your two anchors. Figure 3, the timeline, shows the shifting dominations visually. Figures 4 and 5 give a first look at an artificial neuron and at backpropagation.

Do not get stuck on the details of the Web of Science queries in notes 5 and 6, or on the philosophical references to Fodor, Smolensky and the computational theory of mind. Grasp the general idea and move on. The section on convexity is the most technical; the glossary below gives the minimum needed to get through it.

Savour the interview quotations, set in indented blocks in spoken register. That is where the history comes alive.

Keep three questions in mind while reading:

1. For each era, what is in the world, the calculator and the horizon?
2. Why did the connectionists win when they did, rather than in 1960? The hint is that it was not the ideas that changed.
3. The title speaks of striking back. Who was humiliated, when, and by whom?

Budget two to two and a half hours of attentive reading. It is the longest reading of the series and probably the one that will stay with you.

Terms are listed roughly in their order of appearance in the article.

Computer vision. Research field aiming to make machines see: recognising objects, faces and scenes in images.

ImageNet. A database of fourteen million images across roughly twenty-one thousand categories, hand-labelled by thousands of micro-workers. It serves as an annual competition, and it is the benchmark on which the 2012 earthquake took place.

Benchmark. A standardised test set allowing objective comparison between the performance of different methods.

GPU, graphics processing unit. A processor originally designed for video games, capable of performing millions of simple operations in parallel, which is exactly what a neural network requires.

Deep learning. Neural networks with many layers, hence deep. The term was coined by Hinton in 2006, partly to escape the poor reputation of the word connectionism.

Machine learning. The family of methods in which a machine learns from examples instead of being explicitly programmed. Deep learning is one branch of it.

Parameters, also weights or coefficients. The numbers adjusted during learning, encoding the strength of the connections between neurons. A hundred million parameters means a hundred million small knobs tuned automatically.

Symbolic AI, also GOFAI, Good Old-Fashioned AI. The rules-and-logic approach. Thinking equals manipulating symbols.

Connectionism. The neural network approach. Thinking equals parallel, distributed computation by simple units, with intelligence emerging from the connections.

Cybernetics. The science founded by Norbert Wiener in 1948, studying systems, whether machines or organisms, that regulate themselves through feedback. The direct ancestor of connectionism.

Feedback. Reinjecting the measured output error as a new input so that the system corrects itself. The thermostat is the canonical example. It is the founding principle of cybernetics and, in a sense, of all machine learning.

Black box. A system whose inputs and outputs are observable but whose internal workings are not understood. A recurring criticism of neural networks, and a property their defenders have claimed proudly since the cybernetic era.

Formal neuron. An ultra-simplified mathematical model of a neuron, due to McCulloch and Pitts in 1943. It sums its inputs weighted by weights and activates if the sum exceeds a threshold. Figure 4 shows that this amounts to three operations, no more.

Perceptron. The first learning machine built on a neural network, developed by Rosenblatt between 1957 and 1961, funded by the US Navy and designed for image recognition. It is the emblem of early connectionism and the target of Minsky’s attack.

Layers and hidden layers. Neurons are organised in tiers: an input layer holding the data, intermediate layers called hidden where the work happens, and an output layer holding the answer. Minsky’s 1969 book attacked a single-layer perceptron.

XOR, exclusive OR. The elementary logical function “A or B, but not both”, which a single-layer perceptron cannot learn. This was Minsky and Papert’s decisive argument for burying connectionism. Multiple layers solve the problem.

AI winter. A period of collapse in funding and credibility following excessive promises. There have been two general ones, in the early 1970s and the late 1980s, plus the specifically connectionist winters of 1969 to 1986 and 1995 to 2007.

Expert system. A 1980s program encoding human expert knowledge as thousands of IF-THEN rules, MYCIN for medical diagnosis being the standard example, driven by an inference engine that decides which rule to apply when. Expert systems mark both the peak and the collapse of symbolic AI.

Knowledge base. The stock of rules and facts held by an expert system, to be contrasted with a dataset: a knowledge base contains intelligible rules, a dataset contains raw examples.

Backpropagation, or backprop. The algorithm popularised in 1986 by Rumelhart, Hinton and Williams that makes learning possible. The error is measured at the output, then propagated backwards layer by layer to adjust each weight in the right direction. See figure 5. This is the central mechanism of all deep learning.

Loss function. The number measuring how wrong the network is. All of learning consists in minimising it.

Gradient descent. The minimisation method: compute the slope of the loss function and take a small step downhill, then repeat millions of times. The usual image is walking down a mountain in fog by following the slope underfoot. Stochastic means the slope is estimated on a small sample of the data at a time, which is faster.

Convolution and convolutional networks, CNNs. A technique invented by LeCun in 1989 for images. Rather than connecting every pixel to every neuron, small filters are slid across the image to detect local patterns such as edges and corners independently of their position. This is the architecture behind AlexNet in 2012.

Feature engineering. The now largely extinct art of hand-programming the relevant characteristics of the data, detecting edges, corners and contrasts, before passing them to an algorithm. Deep learning made it obsolete because the network discovers its own features. An entire scientific community lost its object of research this way, which accounts for the bitterness audible in some of the testimony in the article.

End-to-end. Processing raw data, the pixels, all the way to the final answer, “cat”, within a single network, with no intermediate step programmed by a human.

SVM, support vector machines, and kernel methods. A rival learning method dating from 1992, mathematically elegant and dominant between roughly 1995 and 2010. It was the great internal adversary within the learning camp, and the duel between SVMs and neural networks structures an entire section of the article.

Convexity. A mathematical criticism levelled at neural networks. A convex function is shaped like a bowl, with a single hollow, so gradient descent is guaranteed to reach the bottom, the global minimum. A neural network’s loss function is instead a landscape of mountains with countless valleys, local minima, with no guarantee of finding the best one. The SVM mathematicians treated this as a fatal flaw; LeCun’s answer amounted to saying that the theoretical guarantee matters less than the fact that it works better in practice, and experience proved him right.

Overfitting. When a network learns its training examples by heart instead of extracting general regularities, performing excellently on known data and poorly on new data. Dropout, the random switching-off of neurons during training, is one countermeasure mentioned in the article.

Hyper-parameters. All the architectural choices fixed by the human before learning begins: the number of layers, the number of neurons, the learning rate. These stand in contrast to parameters, which are learned automatically. The article shows that human labour does not disappear, it shifts from writing rules to setting these values.

Dataset. A collection of examples used to train and test a model, often as input-output pairs such as a photograph and its label.

Labelled data. Examples accompanied by the correct answer, supplied by humans. This is the fuel of supervised learning.

Crowdsourcing and Mechanical Turk. Micro-work platforms where thousands of people, paid by the task, label data by drawing a box around the dog or typing the spoken word. This is the human face, invisible and poorly paid, of so-called raw data, and ImageNet is its product.

Embedding, or vector. The transformation of an object, a word, an image, a social network, into a list of numbers so that a network can compute with it. The article cites word2vec and LeCun’s formula for putting the world into a vector, world2vec. This is the direct bridge to Lecture 2.

Induction and deduction. Deduction starts from general rules and applies them to particular cases, which is the symbolic approach. Induction starts from particular cases, the examples, and extracts a general rule, which is the connectionist approach. The article’s full subtitle, “the invention of inductive machines”, says the essential.

Watch these after finishing the article to go further.

• The moment we stopped understanding AI [AlexNet], Welch Labs, about eighteen minutes, in English. The 2012 earthquake seen from inside the network, with excellent visualisations of AlexNet’s layers, and the perfect complement to the article’s opening scene.
• Heroes of Deep Learning: Andrew Ng interviews Geoffrey Hinton, followed by the LeCun interview in the same series, in English. Both are cited in the article’s own footnotes, notes 3 and 23. Hearing Hinton and LeCun recount the connectionist winter in their own voices is worth the time.
• Le deep learning, ScienceEtonnante (David Louapre), in French. Neural networks, backpropagation and convolution explained in twenty minutes.

Cardon, D., Cointet, J.-P. and Mazières, A. (2018). Neurons spike back. The invention of inductive machines and the artificial intelligence controversy. Réseaux, 211(5), 173–220.