Shivanshu Gupta

selected publications

1. ArXiv

   GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks

   Shivanshu Gupta, Clemens Rosenbaum, and Ethan R. Elenberg

   2023

   Abs arXiv Bib PDF

   Large language models (LLMs) have the ability to perform in-context learning (ICL) of new tasks by conditioning on prompts comprising a few task examples. This work studies the problem of selecting the best examples given a candidate pool to improve ICL performance on given a test input. Existing approaches either require training with feedback from a much larger LLM or are computationally expensive. We propose a novel metric, GistScore, based on Example Gisting, a novel approach for training example retrievers for ICL using an attention bottleneck via Gisting, a recent technique for compressing task instructions. To tradeoff performance with ease of use, we experiment with both fine-tuning gist models on each dataset and multi-task training a single model on a large collection of datasets. On 21 diverse datasets spanning 9 tasks, we show that our fine-tuned models get state-of-the-art ICL performance with 20% absolute average gain over off-the-shelf retrievers and 7% over the best prior methods. Our multi-task model generalizes well out-of-the-box to new task categories, datasets, and prompt templates with retrieval speeds that are consistently thousands of times faster than the best prior training-free method.

   @misc{gupta2023gistscore,
     title = {GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks},
     author = {Gupta, Shivanshu and Rosenbaum, Clemens and Elenberg, Ethan R.},
     year = {2023},
     eprint = {2311.09606},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. ArXiv

   Leveraging Code to Improve In-context Learning for Semantic Parsing

   Ben Bogin, Shivanshu Gupta, Peter Clark, and 1 more author

   2023

   Abs arXiv Bib PDF

   In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. However, learning to parse to rare domain-specific languages (DSLs) from just a few demonstrations is challenging, limiting the performance of even the most capable LLMs. In this work, we improve the effectiveness of ICL for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs, and (2) augmenting prompts with a structured domain description that includes, e.g., the available classes and functions. We show that both these changes significantly improve accuracy across three popular datasets. Combined, they lead to dramatic improvements (e.g. 7.9% to 66.5% on SMCalFlow compositional split), nearly closing the performance gap between easier i.i.d. and harder compositional splits when used with a strong model, and reducing the need for a large number of demonstrations. We find that the resemblance of the target parse language to general-purpose code is a more important factor than the language’s popularity in pre-training corpora. Our findings provide an improved methodology for building semantic parsers in the modern context of ICL with LLMs.

   @misc{bogin2023leveraging,
     title = {Leveraging Code to Improve In-context Learning for Semantic Parsing},
     author = {Bogin, Ben and Gupta, Shivanshu and Clark, Peter and Sabharwal, Ashish},
     year = {2023},
     eprint = {2311.09519},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

3. EMNLP Findings

   Coverage-based Example Selection for In-Context Learning

   Shivanshu Gupta, Matt Gardner, and Sameer Singh

   Dec 2023

   Abs arXiv Bib PDF

   In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance. The standard approach of independently ranking and selecting the most similar examples selects redundant examples while omitting important information. In this work, we show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects, e.g. reasoning patterns, of the test input. We further extend BSR and many standard metrics to easily optimizable set-level metrics, giving still better coverage of those salient aspects. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, set selection using Set-BSR outperforms independent ranking by up to 17 points on average and, despite being training-free, surpasses methods that leverage task or LLM-specific training.

   @inproceedings{gupta-etal-2023-coverage,
     title = {Coverage-based Example Selection for In-Context Learning},
     author = {Gupta, Shivanshu and Gardner, Matt and Singh, Sameer},
     editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.findings-emnlp.930},
     pages = {13924--13950},
   }

4. EMNLP

   Successive Prompting for Decomposing Complex Questions

   Dheeru Dua, Shivanshu Gupta, Sameer Singh, and 1 more author

   Dec 2022

   Abs arXiv Bib PDF Code

   Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available. Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass. We introduce “Successive Prompting” where, we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well. The intermediate supervision is typically manually written, which can be expensive to collect. We introduce a way to generate synthetic dataset which can be used to bootstrap model’s ability to decompose and answer intermediate questions. Our best model (with successive prompting) achieves an improvement in F1 of ~5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset.

   @inproceedings{dua-etal-2022-successive,
     title = {Successive Prompting for Decomposing Complex Questions},
     author = {Dua, Dheeru and Gupta, Shivanshu and Singh, Sameer and Gardner, Matt},
     booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
     month = dec,
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.emnlp-main.81},
     pages = {1251--1265},
   }