Associated with the Linz Institute of Technology and the Institute of Computational Perception @ JKU Linz, Austria

ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale

Abstract

Multi-task learning (MTL) has shown considerable practical benefits, particularly when using language models (LMs). While this is commonly achieved by learning tasks under a joint optimization procedure, some methods, such as AdapterFusion, divide the problem into two stages: (i) task learning, where knowledge specific to a task is encapsulated within sets of parameters (e.g., adapters), and (ii) transfer, where this already learned knowledge is leveraged for a target task. This separation of concerns provides numerous benefits (e.g., promoting reusability). However, current two stage MTL introduces a substantial number of additional parameters. We address this issue by leveraging the usefulness of linearly scaling the output representations of source adapters for transfer learning. We introduce ScaLearn, a simple and highly parameter-efficient two-stage MTL method that capitalizes on the knowledge of the source tasks by learning a minimal set of scaling parameters that enable effective transfer to a target task. Our experiments on three benchmarks (GLUE, SuperGLUE, and HumSet) and two encoder LMs show that ScaLearn consistently outperforms strong baselines with a small number of transfer parameters (~0.35% of those of AdapterFusion). Remarkably, we observe that ScaLearn maintains its strong abilities even when further reducing parameters, achieving competitive results with only 8 transfer parameters per target task. Our proposed approach thus demonstrates the power of simple scaling as a promise for more efficient task transfer. Our code is available at https://github.com/CPJKU/ScaLearn.


Citation

Markus Frohman, Carolin Holtermann, Shahed Masoudian, Anne Lauscher, Navid Rekab-saz
ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale
Findings of the Association for Computational Linguistics ACL 2024, 11743--11776, 2024.

BibTeX

@inproceedings{Frohman2024Scalearn_ACL_2024,
    title = {ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale},
    author = {Frohman, Markus and Holtermann, Carolin and Masoudian, Shahed and Lauscher, Anne and Rekab-saz, Navid},
    booktitle = {Findings of the Association for Computational Linguistics ACL 2024},
    location = {Bangkok},
    pages = {11743--11776},
    month = {August 11-16},
    year = {2024}
}