​Sequential Attention: Making AI models leaner and faster without sacrificing accuracy

Sequential Attention is a greedy subset-selection method designed to make large machine-learning models smaller and faster without materially reducing accuracy. It selects features, layers, blocks, or weights one at a time using attention scores that are recalculated after each choice, allowing the model to account for nonlinear interactions and redundancy. By integrating selection into a single training process, it aims to retain the quality of traditional greedy methods while avoiding their prohibitive computational cost.

The Subset-Selection Challenge

• Feature selection removes irrelevant or redundant inputs, but finding the optimal subset is NP-hard.
• Deep neural networks make selection harder because:
  • A feature that seems unimportant alone may be essential in combination with others.
  • Features that appear valuable individually may become redundant when selected together.
• The same problem applies beyond input features:
  • Selecting embedding dimensions or chunks.
  • Pruning entries or blocks from weight matrices.
  • Choosing layers or other model components.

How Sequential Attention Works

• The method builds a subset step by step rather than weighting all candidates at once.
• At each stage:
  • Previously selected candidates provide context.
  • Attention scores estimate the importance of every remaining candidate.
  • The highest-scoring candidate is added permanently.
  • The model recalculates scores to reflect the candidate’s marginal contribution.
• This adaptive process can identify high-order nonlinear interactions that simpler filter methods may miss.
• It uses softmax-based attention scores for ranking, but applies them sequentially instead of in a single pass.
• Although greedy selection can be expensive when each candidate requires model retraining or evaluation, Sequential Attention performs selection within one training process, greatly reducing overhead.

Main Benefits

• Efficiency and accuracy: Candidates can be evaluated in parallel once attention scores are available, while sequential updates preserve adaptive selection.
• Interpretability: Attention scores provide a view into which inputs or components the model considered important.
• Scalability: The approach is intended for large candidate sets and modern deep-learning architectures.
• Reduced redundancy: Recalculating scores after each selection helps prevent the model from repeatedly choosing overlapping or unnecessary components.

Feature Selection

• Traditional greedy feature selection repeatedly retrains or reevaluates a model for every possible feature at every step.
• Sequential Attention replaces these expensive marginal-gain calculations with the model’s internal attention weights.
• The algorithm:
  • Scores all unselected features.
  • Adds the feature with the highest score.
  • Reruns the model and updates the scores for the remaining features.
• The method reportedly achieved state-of-the-art or competitive results across proteomics, image, and activity-recognition benchmarks.
• Its one-pass implementation makes greedy-style selection substantially faster.
• For linear regression, Sequential Attention is mathematically equivalent to Orthogonal Matching Pursuit (OMP), an established method with theoretical reliability and performance guarantees.

Block Sparsification

• Neural-network pruning removes unnecessary weights to reduce model size and improve deployment efficiency.
• Block sparsification removes groups of parameters rather than individual weights, making the resulting sparsity more compatible with hardware acceleration.
• Earlier approaches generally fell into two categories:
  • Differentiable pruning, which learns continuous importance proxies.
  • Combinatorial optimization, which searches directly for sparse structures.
• The referenced work, “SequentialAttention++ for Block Sparsification,” aims to combine these differentiable and combinatorial approaches into a unified pruning framework.

Sequential Attention is best understood as an adaptive, attention-based alternative to costly repeated subset searches. It is particularly promising when model components interact nonlinearly and when hardware-friendly sparsity or feature reduction is needed at scale.