Qingqing Cao

2024

1. arXiv

   APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

   Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao

   Jan 2024

   Abstract arXiv BibTeX

   Fine-tuning and inference with large Language Models (LM) are generally known to be expensive. Parameter-efficient fine-tuning over pretrained LMs reduces training memory by updating a small number of LM parameters but does not improve inference efficiency. Structured pruning improves LM inference efficiency by removing consistent parameter blocks, yet often increases training memory and time. To improve both training and inference efficiency, we introduce APT that adaptively prunes and tunes parameters for the LMs. At the early stage of fine-tuning, APT dynamically adds salient tuning parameters for fast and accurate convergence while discarding unimportant parameters for efficiency. Compared to baselines, our experiments show that APT maintains up to 98% task performance when pruning RoBERTa and T5 models with 40% parameters left while keeping 86.4% LLaMA models’ performance with 70% parameters remained. Furthermore, APT speeds up LMs fine-tuning by up to 8x and reduces large LMs memory training footprint by up to 70%.

   @misc{zhaoAPTAdaptivePruning2024,
     title = {{{APT}}: {{Adaptive Pruning}} and {{Tuning Pretrained Language Models}} for {{Efficient Training}} and {{Inference}}},
     shorttitle = {{{APT}}},
     author = {Zhao, Bowen and Hajishirzi, Hannaneh and Cao, Qingqing},
     year = {2024},
     month = jan,
     number = {arXiv:2401.12200},
     eprint = {2401.12200},
     primaryclass = {cs},
     publisher = {{arXiv}},
     urldate = {2024-01-23},
   }

2. ICLR 2024

   BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

   Qingqing Cao, Sewon Min, Yizhong Wang, and Hannaneh Hajishirzi

   In The Twelfth International Conference on Learning Representations, Jan 2024

   Abstract arXiv BibTeX Spotlight (5%)

   Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance. Our code will be publicly available.

   @inproceedings{caoBTRBinaryToken2024,
     title = {{{BTR}}: {{Binary Token Representations}} for {{Efficient Retrieval Augmented Language Models}}},
     shorttitle = {{{BTR}}},
     booktitle = {The {{Twelfth International Conference}} on {{Learning Representations}}},
     author = {Cao, Qingqing and Min, Sewon and Wang, Yizhong and Hajishirzi, Hannaneh},
     year = {2024},
     urldate = {2024-01-23},
     langid = {english},
     highlight = {Spotlight (5%)}
   }

2023

1. ACL 2023

   PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

   Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023

   Abstract arXiv BibTeX Code Poster Slides

   Large-scale vision language (VL) models use Transformers to perform cross-modal interactions between the input text and image. These cross-modal interactions are computationally expensive and memory-intensive due to the quadratic complexity of processing the input image and text. We present PuMer: a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text, improving model inference speed and reducing memory footprint. PuMer learns to keep salient image tokens related to the input text and merges similar textual and visual tokens by adding lightweight token reducer modules at several cross-modal layers in the VL model. Training PuMer is mostly the same as finetuning the original VL model but faster. Our evaluation for two vision language models on four downstream VL tasks shows PuMer increases inference throughput by up to 2x and reduces memory footprint by over 50% while incurring less than a 1% accuracy drop.

   @inproceedings{cao-etal-2023-pumer,
     title = {{P}u{M}er: Pruning and Merging Tokens for Efficient Vision Language Models},
     author = {Cao, Qingqing and Paranjape, Bhargavi and Hajishirzi, Hannaneh},
     editor = {Rogers, Anna and Boyd-Graber, Jordan and Okazaki, Naoaki},
     booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.acl-long.721},
     doi = {10.18653/v1/2023.acl-long.721},
     pages = {12890--12903},
   }