GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
arXiv:2507.18043v2 Announce Type: replace Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights. However, most existing approaches rely on fixed, global intervention vectors, overlook the causal influence of individual input tokens, and fail to leverage informative gradients from the model's logits, particularly in multimo