What is Sequential Attention?

Sequential Attention is a method from Google Research for deciding which columns of a dataset a model should be trained on. A dataset is a table: each row is one example, and each column is one thing recorded about it. In one of Google's tests the rows were images of handwritten digits and the columns were the 784 pixels, each holding how bright that pixel was. Training on all 784 works, but many pixels say the same thing, and a model trained on fewer columns is smaller and faster. Sequential Attention finds the useful ones with a trial training run. The model learns a score for each of the 784 columns, the top-scoring column is kept, training continues, the scores shift, and the next column is kept. After 50 picks the trial model is thrown away and a clean model is trained on those 50 columns only. It still got the digit right 95.6% of the time.

Sequential Attention is a method from Google Research for deciding which columns of a dataset a model should be trained on. A dataset is a table. Each row is one example, each column is one thing recorded about it, and in this field a column is called a feature. Choosing which columns to train on is feature selection, and it matters because a model trained on fewer columns is smaller, faster, needs less data collected, and shows which columns it relies on. In one of Google's tests the rows were images of handwritten digits and the columns were the 784 pixels, each holding a brightness, with the aim of keeping 50. The difficulty is that a column's worth depends on which others are in. Two pixels that always brighten together are each useful alone and worthless together. Scoring each column once and keeping the top 50 ignores that and keeps pairs. Trying every column with a fresh training run catches it but costs 784 runs for the first pick, 783 for the second, down to 735 for the fiftieth, 37,975 in all. Sequential Attention does it in one trial run. The model is trained with one extra score per column, learned along with everything else and squashed so the 784 scores add up to 1. The column with the top score is kept. Training continues with that pick fixed, and the scores settle again, lower for any column that now only repeats a kept one. After 50 picks the trial model is discarded, and a clean model with no scoring part is trained on the 50 columns alone. That model still got the digit right 95.6% of the time, matching or beating the other selection methods tested. The scores are a kind of attention, the same idea as the attention in a transformer, which weighs words against each other, borrowed here to weigh columns. Nothing in the method is about pixels: any table of columns can be trimmed this way. No method can promise the best 50 in general, and this one is only proven to behave well for the simplest kind of model. It is not distillation or quantization, which shrink a model after training. This decides what the model is trained on in the first place.

Sequential Attention is a method for deciding which columns of a dataset a model should be trained on, found by a trial run that keeps the most useful column, re-scores the rest, and keeps the next. Packing a suitcase under an airline weight limit works the same way. The careful packer puts in the single most useful item, then asks what adds the most given what is already in. After one warm jacket goes in, a second jacket adds almost nothing, so it stays on the bed. The lazy packer ranks each item by usefulness once and takes the top 20, and three jackets make the cut, because each one scored high on its own. The items are the columns, the question of what adds most given what is packed is the re-scoring after each pick, and the three jackets are the duplicate columns a one-time ranking keeps. The match ends here: a packer judges by eye, while Sequential Attention trains a model to judge, and once a column is in it stays in.