I built ALHR: A tree based sparse attention system that achieves sub-quadratic inference while retaining accuracy. [P]
<!-- SC_OFF --><div class="md"><p>ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read.</p> <p>It does use a dense teacher while phase 1 of training however.</p> <p>MQAR TEST AT 1024 TOKENS - Average keys read per query by dense - 512 Keys</p> <p>Average keys read per query by ALHR - 30 keys</p> <p>Top - 1 accuracy of dense - 94.9%</p> <p>Top - 1 accuracy of ALHR - 92.1%</p> <p>KV Compression of dense - 1x(100%
![[2610.08927] Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station](https://external-preview.redd.it/q3evP6JeDpAC2MdSQHWYxnCYTqbJkElIQsLFqVSdkss.png?width=640&crop=smart&auto=webp&s=de730fbf7ecace6df0036b21470c16a2d4feacfb)

























