Where we left off

All you need from Part 2 is this. We fitted an ARMA(1,1) model to 400 minutes of healthy thermostat data:

It predicts each minute’s temperature from the previous minutes, and the typical size of its error while the room is healthy is °C. In statsmodels:

from statsmodels.tsa.arima.model import ARIMA

model = ARIMA(y_healthy, order=(1, 0, 1), trend="c").fit()

The full script is in part3_catch_fault.py.

The scenario

The data is simulated again, once per minute. For the first 400 minutes the heating is healthy. After that the heater starts to fail: a heating shortfall that grows a little every minute. The model has never seen this part of the data.

The idea is simple. While the room is healthy, the model’s guesses are close to the real temperature. Once the heater fails, the real temperature falls below the guesses, and the gap between them (the residual) keeps growing. We turn that gap into an alarm.

Step 1: measure the gap

We apply the same fixed model to all the data, healthy and faulty, without refitting it. For each minute the model makes a guess, and the residual is how far the real temperature is from that guess:

full = model.apply(y)                  # same parameters, new data
resid = full.resid
sigma = resid[5:400].std()             # typical size of the error while healthy
z = resid / sigma                      # residual measured in "sigmas"

The number is the typical size of the model’s error while everything is healthy, here about 0.10 °C. Dividing by it gives , which says how many typical errors the current error is. A value of is an ordinary wobble, and is very unusual. This puts every residual on the same scale, whatever the sensor measures.

Step 2: a simple alarm, the ±3σ threshold

The simplest alarm is a threshold: raise it when the residual goes beyond , meaning . If the noise is normally distributed, only about 1 point in 370 crosses this line by chance, so a crossing is a good sign that something has changed.

Its weakness is that it reacts to single points. A single spike can give a false alarm, and a slow drift that stays just inside the line can go unnoticed for a long time.

Step 3: CUSUM, the alarm that remembers

The CUSUM (cumulative sum) fixes this by adding up small but persistent errors, like a bucket that slowly fills with water. Since a failing heater makes the temperature fall, we watch for downward shifts:

In words: take the previous total and add the new drop . Subtract a small allowance and never let the total go below zero. The constant means that wobbles within half a sigma are ignored, so while the room is healthy the bucket drains as fast as it fills and stays near zero. When the residual keeps falling below zero, the bucket fills steadily. When rises above a threshold , we raise the alarm.

What the figure shows

• During the healthy period the residual looks like noise. The ±3σ rule raised one false alarm at minute 22, a single spike. A few false alarms are expected over hundreds of samples.
• After the fault the residual slowly drifts below zero. The first ±3σ alarm came at minute 477, 77 minutes after the fault began.
• The CUSUM alarmed at minute 440, 40 minutes after the fault began. By then the temperature had fallen from about 20.9 °C to 20.0 °C, a drop you would hardly notice in the room. The model noticed it because it knows how the healthy signal normally behaves.

Is this a lucky run?

To check, I repeated the experiment with 30 different random seeds. A single ±3σ threshold raised at least one false alarm in 21 of the 30 runs. The CUSUM raised none, and detected the fault after a median of about 40 minutes. With instead of 8, about half of the runs had false alarms, so thresholds must be tuned on healthy data. You can repeat this check with the robustness() function in the script.

How control systems use this

In control and supervision, the residual is the main signal for fault detection. It goes through three steps:

1. Generate: compare the model’s prediction with the real measurement.
2. Evaluate: is the residual the size we would expect from a healthy machine? (, CUSUM)
3. Decide: raise an alarm, or not.

One catch: a feedback controller can hide a fault by working harder to compensate. A failing heater may keep the room at 21 °C by running at full power. That is why supervision monitors residuals and control effort, not only the output. In our example the thermostat was not compensating, which is why the temperature itself fell.

Conclusion

The workflow for catching a fault:

1. Build and check a model on healthy data (Part 2).
2. Apply the fixed model to new data and compute the residual.
3. Measure the residual in sigmas, using the spread from the healthy period.
4. Watch it with a threshold and a CUSUM, and tune both on healthy data.

A fault is anything the model cannot explain, and the CUSUM notices small drifts that a single threshold misses.

Next time

So far we have looked at one signal at a time. Real machines have many inputs and many outputs, such as heater power, outside temperature and several rooms. In Part 4 we move to MIMO (multiple-input, multiple-output) models.

The data

I use the thermostat example again, with simulated data so that we know what is really going on. The room temperature is recorded once per minute. For the first 400 minutes the heating is healthy, and this is the only part we use in this post. (The heater starts to fail after minute 400, but that is the story of Part 3.)

The healthy data comes from an ARMA(2,1) process around 21 °C. That means a “right answer” exists, but we will pretend we don’t know it, as in real life. The full script is in part2_choose_check_model.py, and you can run it yourself.

The rule for everything below: the model is fitted on healthy data only.

Step 1: choose d

What “stationary” means

Picture a dog on a leash walking around its owner. It wanders, but it never goes far, and the size of its wandering stays about the same. That is a stationary signal. The thermostat is like this: the room temperature wiggles around 21 °C, and both the average and the size of the wiggles stay the same hour after hour.

Now picture the dog walking away down the road. It has no “home” position, so the average keeps moving. That is a non-stationary signal. A room that warms up all morning as the sun comes in is like this.

AR and MA models learn a fixed recipe (the weights and ) from past data. That recipe only works if the signal behaves the same way all the time. If the average keeps drifting, a recipe learned on the first hour does not fit the next hour.

from statsmodels.tsa.stattools import adfuller

adf_p = adfuller(y_healthy)[1]     # [1] picks the p-value; p < 0.05 -> stationary

Step 2: get a first guess for p and q

Two plots give a starting point.

The ACF (autocorrelation function) asks: how much does the signal resemble its own past? Take the temperature series, shift a copy of it by one minute and measure how similar the two are. That is the correlation at lag 1. Do the same with a five-minute shift for lag 5, and so on. Values run from −1 to 1, and 1 means identical behavior. For our thermostat, lag 1 is about 0.93: the temperature now is very similar to a minute ago. The bars shrink as the lag grows, because the room gradually forgets. That slow fading is the memory of the system.

The PACF (partial autocorrelation function) fixes a flaw of the ACF. The temperature 3 minutes ago seems related to the temperature now, but partly only because both are related to the minutes in between. It is like gossip: A tells B, and B tells C. A and C look connected, but once you know B, A adds nothing. The PACF removes these indirect links. At each lag it asks: once I already know the more recent values, does this older value tell me anything new?

The shaded band marks the range where a bar could be just random chance, so bars inside it count as zero. Use these rules to read the plots:

What you see	It suggests
PACF cuts off after lag , ACF decays slowly	AR()
ACF cuts off after lag , PACF decays slowly	MA()
Both decay	a mix, so let AIC and BIC decide

In Figure 2 the ACF decays slowly, and the PACF has one clear spike at lag 1, then falls inside the band. Once you know last minute’s temperature, older minutes add nothing new, so this points to . It is only a guess, and real data is rarely this clean, so in the next step we let the numbers compare candidates.

Step 3: compare models with AIC and BIC

A bigger model always fits the training data better, but it may just be memorizing noise. AIC and BIC reward a good fit and penalize complexity:

Here is the number of parameters (the AR weights, the MA weights, the mean and the noise variance), the number of samples, and the best likelihood the model reaches. Lower is better. BIC punishes extra parameters more, so it tends to pick simpler models.

We fit every combination of and from 0 to 3:

from statsmodels.tsa.arima.model import ARIMA

for p in range(4):
    for q in range(4):
        res = ARIMA(y_healthy, order=(p, 0, q), trend="c").fit()
        print(p, q, res.aic, res.bic)

Here are some of the results, together with the Ljung-Box check from the next step:

Model		AIC	BIC	Ljung-Box p
MA(2)	4	−451.0	−435.0	< 0.001
AR(1)	3	−696.8	−684.9	0.363
AR(2)	4	−699.5	−683.5	0.701
ARMA(1,1)	4	−700.1	−684.1	0.754
ARMA(2,1)	5	−699.0	−679.1	0.781
ARMA(3,3)	8	−697.3	−665.4	0.422

Your numbers may differ in the last digits.

Three things stand out:

• MA(2) is far behind. It has no memory of past values, which this signal clearly needs.
• The next four are almost tied. A difference of less than about 2 points is usually not meaningful.
• The big ARMA(3,3) does not win. More parameters did not buy a better model, and the penalty shows.

AIC picks ARMA(1,1) and BIC picks the simpler AR(1). When they disagree, AIC favors prediction accuracy and BIC favors simplicity. Our goal is good predictions, because the residual is what we monitor, so I go with ARMA(1,1). AR(1) would also be a reasonable choice.

Step 4: is the model good enough?

AIC and BIC pick the best among the candidates, but they cannot tell you whether the winner is actually good. For that, look at the residuals. If the model captured every real pattern, what is left should be white noise: random, with no structure left to learn.

The Ljung-Box test checks this with one number. Its starting assumption is that the residuals are white. It adds up the squared autocorrelations of the residuals up to lag :

and compares with a distribution with degrees of freedom.

from statsmodels.stats.diagnostic import acorr_ljungbox

acorr_ljungbox(res.resid[5:], lags=[10], model_df=p + q)

• p-value above 0.05: no leftover pattern. The model is good enough.
• p-value below 0.05: structure remains, so the model is too simple (underfitted). Try a higher or , recheck , or add a seasonal component if the pattern repeats daily.

Figure 3 shows the difference. The workflow is: AIC and BIC choose the winner, Ljung-Box confirms it.

Our final model, fitted on the healthy data, is:

The number 21.01 is the mean temperature. The weight 0.911 says the room has a long memory: this minute’s temperature is mostly last minute’s, which is the thermal inertia we talked about in Part 1. The weight 0.130 on the previous error is small. In the statsmodels output these appear as const, ar.L1 and ma.L1, which correspond to the mean, and in the notation of Part 1. The typical size of the residual is °C. We will use this model in Part 3.

Conclusion

The workflow for choosing and checking a model of a single signal:

1. Check stationarity (ADF) and pick .
2. Look at ACF and PACF for a first guess of and .
3. Fit candidates on healthy data and compare AIC and BIC.
4. Confirm with Ljung-Box that the residuals are white.

Every step uses healthy data to learn what normal looks like. Once the model passes these checks, we can trust its predictions, and anything it cannot explain is worth a closer look.

Next time

In Part 3, we apply this model to new data from the thermostat, where the heater slowly starts to fail, and use the residual to raise an alarm.

Why this series

At university, I meet system identification and estimation as theory: equations, proofs and toolbox commands. It is easy to pass an exam and still not feel how any of it works on a real machine. This series is my attempt to close that gap. I document what I learn as I go, and for each concept I follow one real-life example, such as a thermostat, from the idea to the code to a working fault detector. I write for people who are new to the topic, because not long ago that was me.

A bit about me

I am an engineering student specializing in data science and data engineering, with a focus on System Identification & Data Analysis and Adaptive Learning, Estimation & Supervision.

How I got here

It all started with trains. I wondered how complicated it must be to perform maintenance on trains. While doing some research, I read an article explaining how sensors are placed on trains to collect data. This data is then analyzed and used for forecasting, to predict when maintenance will be needed. By doing this, operators were cutting costs significantly.

That was my introduction to predictive maintenance and anomaly detection, and I found it interesting enough to pursue a postgraduate degree in the field.

What this series covers

System identification means building a mathematical model of a system from its input and output data. There are three approaches:

• White box: you know the physics and write the equations directly.
• Grey box: you know part of the physics, and data fills in the rest.
• Black box: you do not model the internals at all. You let the data describe the behavior.

This series focuses on black box, data-driven models. We will start with the simplest kind: SISO (single-input, single-output) models, which look at one signal at a time, such as the temperature reading from your house thermostat.

AR, MA and ARIMA

In this part, we will look at three models. They all predict the next value of a signal from its own history.

AR (AutoRegressive). The next value depends on the previous values of the signal itself:

where:

• : the value of the signal now, for example the thermostat temperature at the current time step .
• : the value one time step ago, meaning the previous measurement. In the same way, is two steps ago, and so on.
• : weights that the model learns from data. Each one says how much the matching past value matters for predicting the next one.
• : how many past values the model looks back at.
• : the error (or noise) term. It is the part of the current value that the past values cannot explain: random, unpredictable, and zero on average. In engineering it is called white noise.

Think of your thermostat reading. The room temperature changes slowly, so where it is now tells you where it will be a few minutes from now.

MA (Moving Average). The next value depends on past prediction errors (the “shocks” the model did not see coming):

where:

• : the error at the current time step, the new random shock that has just hit the system.
• : the error one step ago, the shock from the previous time step. is the one before that, and so on.
• : weights the model learns. Each says how much of an earlier shock is still affecting the signal.
• : how many past errors the model remembers.

(I use for these weights so they are not confused with the weights of the AR model.)

Imagine someone opens a window. The temperature dips, then recovers over the next few minutes. That short-lived disturbance is what MA captures.

ARIMA. AR and MA combined, with an added step called integration (I). Many real signals trend upward or downward over time, like the room warming up all morning as the sun comes in. ARIMA handles this by modeling the change between samples (differencing) instead of the raw values:

The model is written , where:

• : how many past values to use (AR order)
• : how many times to difference the data
• : how many past errors to use (MA order)

Why this matters for anomaly detection

Here is the idea that ties it all together. Train a model on healthy data. Then, as new data comes in, compare the model’s prediction with what the sensor actually reports, . The difference is the residual:

While the machine is healthy, the residual stays small. When something starts to go wrong, the residual grows, and that is your early warning.

For the thermostat: if the heating is working normally, the model predicts the temperature well. If the heater starts failing, the real temperature keeps falling below what the model expects, and the residual grows.

Conclusion

To sum up: AR models the memory of a signal, how its past values shape the next one. MA models disturbances, the short-lived shocks that a healthy system experiences, like an open window. ARIMA combines both and also removes trends through differencing.

An important point is that the disturbances MA captures are normal, not faults. A fault only shows up when the residual becomes much larger than these normal shocks. All of this relies on past data, and that data must come from a healthy system, so the model learns what normal looks like.

Next time

In Part 2, we will look at how to choose the model’s parameters and check that it is good enough. In Part 3, we will use it to catch a fault in Python.