Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

3.3 Learning from Examples: Where Do Sensor Models Come From?

Source

In Section 3.2 the sensor models were simply given: “a plate’s ratio is about 0.10 ± 0.04”, and so on. Where do such numbers come from? In practice, an engineer puts real vessels through the machine, writes down what each one was (the label), and records what every sensor reported. The robot then learns its models from these labelled examples.

Learning brings new questions. How many examples are enough? What if some vessel types are rare? And how do we know whether the robot’s remaining mistakes come from too little data or from sensors that simply cannot tell certain vessels apart?

Source

A labelled dataset

The engineer washes 300 vessels from the family’s kitchen, one at a time, and records the label and the three sensor readings for each. The first few rows of the log look like this:

  # |          label | ratio | weight [g] | greasy
  1 |        tumbler |  1.34 |         98 | no
  2 | cooking vessel |  0.37 |        823 | yes
  3 |           bowl |  0.43 |        164 | yes
  4 |          plate |  0.10 |        140 | no
  5 |          plate |  0.16 |        114 | no
  6 | cooking vessel |  0.47 |       1045 | yes
  7 |          plate |  0.10 |        178 | no
  8 |           bowl |  0.39 |        162 | no
…
vessels of each type: {'plate': np.int64(120), 'tumbler': np.int64(74), 'bowl': np.int64(61), 'cooking': np.int64(33), 'other': np.int64(12)}

Learning by counting and averaging

The prior. The simplest estimate of how often each type appears is its fraction in the data. If NxN_x vessels of type xx appear among NN in total:

P^(x)=NxN\hat{P}(x) = \frac{N_x}{N}

The grease sensor. If Nx,1N_{x,1} of the NxN_x vessels of type xx came out greasy, the natural estimate is the fraction that did:

P^(greasy∣x)=Nx,1Nx\hat{P}(\text{greasy} \mid x) = \frac{N_{x,1}}{N_x}

The camera and the scale. For a Gaussian model, the estimates are the sample mean and sample standard deviation of the readings for each type, exactly as for the tadka thermometer in Section 2.1.

These are called maximum-likelihood estimates: of all possible parameter values, they are the ones that make the observed data most probable. For counts and Gaussians, they turn out to be the everyday “fraction” and “average” we would have guessed anyway.

    type |  prior true/est  | ratio mean true/est | weight mean true/est | P(greasy) true/est
   plate |   0.40 / 0.40   |     0.10 / 0.10     |       180 /  175     |   0.30 / 0.34
 tumbler |   0.25 / 0.25   |     1.30 / 1.26     |        90 /   90     |   0.05 / 0.08
    bowl |   0.20 / 0.20   |     0.45 / 0.44     |       150 /  150     |   0.35 / 0.30
 cooking |   0.10 / 0.11   |     0.40 / 0.38     |       900 /  977     |   0.90 / 0.86
   other |   0.05 / 0.04   |     0.60 / 0.50     |       250 /  263     |   0.40 / 0.21

With 300 examples most estimates land close to the truth. The rare “other” type is the least accurate, because only about a dozen of the 300 vessels were “other”. Estimates are only as good as the number of examples behind them.

The zero-probability problem

Suppose the engineer is in a hurry and collects only 40 vessels. Among them are 10 tumblers, and by chance none of them came out greasy (greasy tumblers are uncommon, only 5%). The plain counting rule (2) then says

P^(greasy∣tumbler)=010=0\hat{P}(\text{greasy} \mid \text{tumbler}) = \frac{0}{10} = 0

The robot now believes that a greasy tumbler is impossible. That sounds harmless until a tumbler of buttermilk arrives and the grease sensor fires. Watch what happens:

            plain counting: 0/10: P(tumbler) = 0.000, most likely =   other, programme = gentle
 Laplace smoothing: (0+1)/(10+2): P(tumbler) = 0.993, most likely = tumbler, programme = quick rinse

With plain counting, a single zero wipes out the tumbler completely. The camera and scale both shout “tumbler”, but multiplying by zero overrules them, and the robot picks a different type and the wrong programme. One unlucky zero in the training data can silence every other sensor.

The fix is Laplace smoothing: before counting, pretend we saw each outcome a small number of extra times, α\alpha (usually 1):

P^(greasy∣x)=Nx,1+αNx+2α\hat{P}(\text{greasy} \mid x) = \frac{N_{x,1} + \alpha}{N_x + 2\alpha}

With α=1\alpha = 1, the tumbler estimate becomes (0+1)/(10+2)≈0.08(0 + 1)/(10 + 2) \approx 0.08: small, but not impossible. The camera and scale can now be heard, and the robot correctly runs a quick rinse. The same idea protects the prior and the Gaussian spreads, which is why fit above adds α\alpha to the type counts and never allows a spread of almost zero.

How much data is enough? The learning curve

To see how performance grows with data, we train on datasets of increasing size and test each model on a large, separate set of vessels it has never seen. Testing on fresh data matters: a model tested on its own training data looks better than it really is. A plot of test cost against training size is called a learning curve.

   10 training vessels → average test cost 2.691
   20 training vessels → average test cost 2.496
   40 training vessels → average test cost 2.136
   80 training vessels → average test cost 2.033
  160 training vessels → average test cost 1.975
  320 training vessels → average test cost 1.969
  640 training vessels → average test cost 1.964
 1280 training vessels → average test cost 1.962
true sensor models → 1.959   |   oracle → 1.865
<Figure size 770x396 with 1 Axes>

Figure 1:Learning curve for the vessel-washing robot. Test cost falls quickly as training data grows, then flattens at the level set by the sensors themselves (orange dashed line). The remaining gap to the oracle (green dashed line) cannot be closed with more data.

The curve tells a clear story:

  • Early on, data is precious. Going from 10 to 80 training vessels cuts the cost sharply. With very few examples, rare types have unreliable models, and the robot makes costly mistakes.

  • Then returns diminish. After a few hundred vessels the curve flattens against the orange line, the best these three sensors could ever do even with perfect models.

  • The final gap is not a data problem. The distance between the orange and green lines is caused by vessels the sensors genuinely cannot tell apart. A million more examples would not remove it.

Where the remaining mistakes are: the confusion matrix

A confusion matrix shows, for each true type (row), how often the robot predicted each type (column). The diagonal holds the correct answers, and everything off the diagonal is a mistake.

correct fraction for each true type: {'plate': np.float64(1.0), 'tumbler': np.float64(1.0), 'bowl': np.float64(0.98), 'cooking': np.float64(0.98), 'other': np.float64(0.53)}
where 'other' vessels end up: {'plate': np.float64(0.09), 'tumbler': np.float64(0.05), 'bowl': np.float64(0.24), 'cooking': np.float64(0.09), 'other': np.float64(0.53)}
<Figure size 594x484 with 1 Axes>

Figure 2:Confusion matrix of a model trained on 500 vessels and tested on 20 000 new ones. Rows are true types; columns are predictions; each row sums to 1. Nearly all mistakes involve the “other” type, whose shapes and weights overlap with everything else.

Plates, tumblers, bowls, and cooking vessels are recognised almost perfectly. The weak spot is “other”: the non-stick tawa, sieve, and glass jar have shapes and weights that overlap with ordinary vessels. Only about half are recognised; about a quarter are mistaken for bowls, and the rest are spread over the other types.

The mistakes are not equally serious. An “other” item mistaken for a bowl gets standard, which is merely too rough. One mistaken for a cooking vessel (about 9% of them here) gets heavy degrease, the one programme that ruins a non-stick coating. “Other” is exactly the type where a wrong programme can damage a vessel, so it deserves attention even though it is rare.

More data, or a better sensor?

The learning curve and the confusion matrix together answer the engineer’s most practical question.

What you seeWhat it meansWhat to do
Test cost still falling steeply as data growsthe models are data-starvedcollect more labelled examples
Test cost flat, near the limit of the true sensor modelsthe sensors cannot separate some typesadd or improve a sensor
Mistakes concentrated in one typethat type needs its own cluedesign a sensor aimed at that type

For our robot, the curve is flat and the mistakes are concentrated in “other”. More data will not help. A sensor that recognises non-stick coatings and glass, for instance a small reflectance sensor that measures how shiny the surface is, would target exactly the remaining gap.

The chapter summary collects the ideas and exercises from all three sections.