These notes teach some parts of data analysis and machine learning by letting the chapters follow the steps of a data science project, whatever its subject. In the earlier chapters we start with small problems such as matching two translations of the same text, the Wordle game,¹¹Wikipedia: Wordle. and making a tool that can plan pleasant walking routes. It is not our aim to cover all of data science and machine learning; these fields are too large to cover in a single course. We develop the machinery these problems need, and no more. We hope that the reader remembers the games, and from the games the techniques we used to solve them.²²Toy problems make good mnemonics for a general technique. No two projects meet quite the same difficulties, and certainly not all of the ones described here, so we hope too that the reader can abstract from our examples.

Along the way we invest in making formal representations of the problems we deal with. Well-designed representations help us to formalize what we want to achieve and to write good pseudocode to compute solutions.

For ease, we will reference nearly only to Wikipedia pages. These pages contain further references if you want to pursue a topic in more detail.

Data science (DS) is the practice of using data to answer empirical questions.³³Wikipedia: Data science. Such questions can be:

1. Descriptive. What is in the data? For example: How many employees reported illness periods that lasted more than two weeks?
2. Predictive. What will happen for a new case? For example: Given a patient’s disease and age, how long will this patient remain ill?
3. Causal. What would change under an intervention? For example: What would be the effect of a salary cut on the number of sick-leave days?

A data science project should start by asking the right question; this includes investigating the problem and settling what a good answer would be. After formulating the question, the next task is to analyze how to answer it: what data must be assembled, what must be measured, how records should be stored. Sometimes the available database is enough; sometimes we need additional measurements,⁴⁴Such as a questionnaire or an experiment. or a different way to store future records.

Raw data rarely contain exactly the quantities needed for the question. A first step is to construct useful variables from the raw fields. For instance, from dates one may construct the duration of an illness period; from repeated records one may construct the number of earlier sick-leave periods; from a text code one may construct an indicator for pregnancy-related absence. In practice, data science therefore starts with analyzing the combination of the question and the data. This involves checking where the data come from, what the variables mean, whether records are comparable, and whether there are missing values or obvious errors.

Simple tables and graphs are often the most useful first tools to represent and analyze data: they reveal scale, variation, outliers, and structure, and they expose errors before any formal prediction method is fitted. Real data is messy. For example, a database may report a person as ill even though the person has already returned to work, or it may code pregnancy-related absence as illness even though the underlying interpretation is different.⁵⁵For instance, women between 20 and 40 may be reported ill for 16 weeks, while this is pregnancy leave, not illness. The difficult task of checking, cleaning, filtering, and interpreting the data therefore precedes model fitting.⁶⁶Often much time in data analysis is spent on these tasks and profound domain knowledge is needed. Model fitting and coding matter, but they should always come after the harder question of whether the data represent the phenomenon we want to study and are suitable for analyzing it. We skip this step in these notes as it is too problem-specific; general advice is hard to give. Hence, we assume henceforth that data is clean.

Often, the primary data is text, and text stays hard to analyze even after cleaning. To show this problem in detail, we consider in Chapter 2 the challenge of matching two translations of the same story. We develop a dynamic programming (DP) framework to compute the best matching between chunks of sentences of each of the texts. DP also underlies reinforcement learning, a major branch of machine learning. Besides this, several relevant general machine learning ideas are addressed in passing.

Interestingly, machine translation itself has been one of the driving problems in machine learning. The Transformer paper Attention Is All You Need⁷⁷Wikipedia: Attention Is All You Need., which led to the architecture behind GPT-style models, was focused on machine translation. Thus, these notes start with a program to match human-made translations; in Section 8.3.4 they end with a far more powerful machine learning tool for working with translated text.

Once the question and the data have been made clear, we can ask what kind of model is useful.

Machine Learning (ML) develops and analyzes algorithms that learn patterns from data and use these patterns for tasks such as prediction and decision making.⁸⁸Wikipedia: Machine learning. In supervised learning, the training data contain input-output pairs.⁹⁹Other branches exist: in unsupervised learning the data contain no labels, and the goal is to find structure such as clusters or low-dimensional summaries; in reinforcement learning an agent learns from rewards obtained by interacting with an environment. These notes focus on supervised learning. In Chapter 3 we study the common supervised learning tasks: regression, where the output is numerical, nominal classification, where the output is categorical without a natural order, and ordinal prediction, where the output is categorical with a natural order.¹⁰¹⁰Wikipedia: Supervised learning. Wikipedia: Nominal category Wikipedia: Ordinal data. For each category we say what counts as a good prediction by choosing a loss function. Fitting is then the problem of finding a predictor that makes the loss small on the data. The chapter also takes up the classical themes of ML. First, overfitting and regularization. Second, the division of the data into a training set to fit on, a validation set to choose between candidate rules that have been trained with different hyperparameter settings, and a test set that is used only at the very end to evaluate whether the found predictor works also on new data. The last two are hold-out sets, which means that they are disjoint from the training data. Third, imbalanced data, where one class is so much rarer than the other that a rule can look accurate while never predicting it. If one in a thousand screened patients carries a certain disease, the rule that declares everybody healthy is 99.9% accurate, and probably useless.

Supervised learning requires labeled data. But where do the labels come from? Sometimes we are lucky and the observations of interest carry labels already. More often they do not, and the labels must be acquired from humans, sensors, or historical records: a radiologist labels X-rays, employees read and classify the feedback on customer evaluation forms, a farmer walks around in a field to check whether the plants flagged as weeds in drone photographs really are weeds. The list is long, and none of this is free. On the contrary, labeling is often the most expensive part of a project.

Building a machine capable of making predictions is the next step. In Chapter 4 we build decision trees and their generalization random forests, which remove the shortcomings of single trees. This class of predictors is versatile: trees cope with large amounts of data and with mixed types of features, they are robust, and they remain easy to understand.

As in the earlier chapters, a concrete application drives our interest in random forests. We wanted to build a walking-route planner that, given a start point \(A\) and an end point \(B\) on the map of the Netherlands, produces a nice route between \(A\) and \(B\).¹¹¹¹This is not the same thing as the shortest path. As it happens, the most expensive steps of a typical data science project have already been done for us. OpenStreetMap (OSM)¹²¹²Wikipedia: OpenStreetMap. stores edges with features for the entire Netherlands. As labels we can use published tracks, which are also in OSM. With this information we can train a random forest to rank edges from good to bad.

However, a ranking is not yet enough. Routing algorithms need edge costs to find the cheapest route. A difference in rank does not automatically correspond to a real cost difference in how pleasant an edge is to walk. The subject of Chapter 6 is to turn the output of a random forest, i.e., a ranking score, into a cost. We will discuss in particular isotonic regression, which is a common technique in statistics.

The last step of a data science project is to put the predictor to work, and to show what it produces, so that the user can judge whether the answer is any good. In the walking app the edge costs go to OSRM,¹³¹³Open Source Routing Machine. which computes the cheapest route, and a Leaflet map draws that route in the browser. The result can be enjoyed as the webapp De Wandelvogel.¹⁴¹⁴De Wandelvogel. Chapter B contains the technical details.¹⁵¹⁵That we delegate this step to an appendix does not make it unimportant. It is the counterpart of the first step: having settled with the user what the right question is and what an answer should look like, this is where the answer is finally delivered. However, it is fairly technical; in these notes we concentrate on the ML techniques.

End of 2022 there was a breakthrough. Suddenly machines, i.e., large language models (LLMs), became available that help us do all of the above steps. We can discuss with them to clarify the question and what a good answer would look like. They write all the code necessary to make simple tables for statistical analysis, and simple graphs to obtain insight into outliers and data errors. They do not mind when data is not clean, they can deal with ambiguity in language, and they seem to be able to make good translations.¹⁶¹⁶At least much better than Google translate before 2024. What they are capable of is impressive.¹⁷¹⁷But see my remarks below.

In the last two chapters of these notes we explain how neural networks work, because LLMs are neural networks with an enormous number of tunable parameters. Then we discuss the particular architectures of LLMs. We will not touch on the question of why LLMs, which essentially are just machines that are trained to predict the next word¹⁸¹⁸More precisely: a token. of a sequence, are able to do all these tasks.

Prediction is not causation. An econometrician may ask whether \(x\) predicts \(y\), but also whether changing \(x\) would cause \(y\) to change. The distinction between ML and causal models is not a distinction between formulas. Ordinary least squares (OLS) can be used as a prediction method, as an econometric model for estimating an interpretable parameter, or as one component of a causal analysis. The difference lies in the question and in the assumptions. A predictive analysis asks whether a rule predicts well for new observations. An econometric estimation problem asks what parameter is being estimated and how uncertain that estimate is. Causal models are concerned with interventions, not only with prediction.¹⁹¹⁹Wikipedia: Causal inference. A causal model asks what would happen to \(Y\) if a treatment \(D\) were set to \(1\) rather than \(0\).

The basic difficulty is that we cannot observe both outcomes for the same unit. If a worker receives a training course, we can observe the worker’s wage after the course, but not the wage that same worker would have had without the course. A clean solution is random assignment: if some workers receive the course by lottery and others do not, then the two groups are comparable on average before treatment. The difference in their average outcomes can then be interpreted as a causal effect.

Prediction and causality can diverge. A variable can be useful for predicting \(Y\) without having the direct causal effect suggested by the prediction: often a third variable, a confounder, drives both. For example, houses with more bedrooms tend to sell for higher prices, but this does not mean that changing the living room of a given house into an extra bedroom would raise its price. So, the number of bedrooms in a house correlates with the price, but the confounder is the size of the house, as larger houses tend to have more bedrooms. Conversely, a causal effect can be important even if it adds little predictive power. Prediction accuracy alone therefore does not establish causality.

Two themes underlie these notes. The first is optimization. Nearly every problem in these notes is formulated as an objective, that is, a number saying how good a candidate solution is, together with the set of candidate solutions.

The second theme is recursion. A hard problem is reduced to a smaller problem of exactly the same form, and the reduction is then applied to what remains. Dynamic programming aligns two texts by aligning two shorter texts. A guess in Wordle leaves a smaller set of candidate words. A tree splits its data in two and grows a subtree on each part. The algorithm for isotonic regression merges two neighboring blocks and starts afresh on the coarser partition. Backpropagation to train neural networks computes the error at one layer from the error at the layer that follows it.

The intended reader of these notes is a student with some background in linear algebra, probability and statistics.

While developing this document I used two LLMs²⁰²⁰Claude and ChatGPT.. I took the role of architect, and the LLMs as discussion partners and builders. I started by telling them that I wanted to use random forests to make a routing app and took it from there. With these tools I learned many interesting topics, such as probability calibration.

In the writing process I noticed some interesting things. These LLMs are very capable but need a lot of coaching at the same time. If left unguided, they can go south. I needed to work very hard on keeping a good overall structure, and stay alert, so as not to be misled by the confident way in which the LLMs write and argue. However, once I managed to get the structure clear, they helped fill in many details, making figures, searching the Internet for literature, answering small questions, repairing typos, finding accurate vocabulary, and checking the mathematics.

I also noticed that at times the language of LLMs can be annoying. LLMs seem to have learned from a lot of marketing text, which makes their language slick and blabby, sometimes to the extent that I had no clue what it would mean. I had to reorganize and rewrite every sentence of this set of notes.

With respect to coding after more than 30 years of programming, I now discuss coding concepts with LLMs in the form of pseudocode because I still like the underlying elegant algorithmic ideas.²¹²¹I am not so keen on reading production code as this includes many checks and offers many options. Such details are important but often hinder the understanding of the basic ideas. Once both of us were satisfied, I followed a test-driven development paradigm. That means that before writing the actual code, the coder develops lots of tests. Once the test suite is somewhat complete, the coder builds code until the first test passes, then the second, and so on, until all tests pass.²²²²The code counts then as correct, or at least as correct as the test suite is complete. Thus, I discussed in words with one LLM what kind of tests were necessary; then I let the LLM write the tests and another LLM comment on them, and back and forth until both LLMs were satisfied. Finally I let an LLM port the pseudocode to python and work until all tests passed.