Jesse Dodge

Publications

For a more complete list of my publications, please check my Google Scholar.

2024

Paloma: A Benchmark for Evaluating Language Model Fit
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, Jesse Dodge
Neural Information Processing Systems Datasets and Benchmarks (NeurIPS), 2024
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
Clara Na, Ian Magnusson, Ananya Harsh Jha, Tom Sherborne, Emma Strubell, Jesse Dodge, Pradeep Dasigi
Empirical Methods on Natural Language Processing (EMNLP), 2024
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Pang Wei Koh, Jesse Dodge, Pradeep Dasigi
Findings of Empirical Methods on Natural Language Processing (EMNLP Findings), 2024
OLMES: A Standard for Language Model Evaluations
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, Hannaneh Hajishirzi
arXiv, 2024
OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi
Association for Computational Linguistics (ACL), 2024
Won Best Theme Paper award
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, Kyle Lo
Association for Computational Linguistics (ACL), 2024
Won Best Resource award
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, Jesse Dodge
Association for Computational Linguistics (ACL), 2024
Language Models Hallucinate, but May Excel at Fact Verification
Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, Hao Peng
North American Chapter of the Association for Computational Linguistics (NAACL), 2024
What's In My Big Data?
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A Smith, Jesse Dodge
International Conference on Machine Learning (ICLR), 2024
Spotlight Presentation (Top 5%)

2023

Catwalk: A Unified Language Model Evaluation Framework for Many Datasets
Dirk Groeneveld, Anas Awadalla, Iz Beltagy, Akshita Bhagia, Ian Magnusson, Hao Peng, Oyvind Tafjord, Pete Walsh, Kyle Richardson, Jesse Dodge
arXiv, 2023
The Rise of Open Science: Tracking the Evolution and Perceived Value of Data and Methods Link-Sharing Practices
Hancheng Cao, Jesse Dodge, Kyle Lo, Daniel A. McFarland, Lucy Lu Wang
arXiv, 2023
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
Hao Peng, Qingqing Cao, Jesse Dodge, Matthew E. Peters, Jared Fernandez, Tom Sherborne, Kyle Lo, Sam Skjonsberg, Emma Strubell, Darrell Plessas, Iz Beltagy, Evan Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi
arXiv, 2023
Surveying (Dis) Parities and Concerns of Compute Hungry NLP Research
Ji-Ung Lee, Haritz Puerto, Betty van Aken, Yuki Arase, Jessica Zosa Forde, Leon Derczynski, Andreas Rücklé, Iryna Gurevych, Roy Schwartz, Emma Strubell, Jesse Dodge
arXiv, 2023
Evaluating the Social Impact of Generative AI Systems in Systems and Society
Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III, Jesse Dodge, Ellie Evans, Sara Hooker, Yacine Jernite, Alexandra Sasha Luccioni, Alberto Lusoli, Margaret Mitchell, Jessica Newman, Marie-Therese Png, Andrew Strait, Apostol Vassilev
arXiv, 2023
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text [data]
Wanrong Zhu*, Jack Hessel*, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi
Neural Information Processing Systems Datasets and Benchmarks (NeurIPS), 2023
Press: [MarkTechPost] [GeekWire]
Efficient Methods for Natural Language Processing: A Survey
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
Transactions of the ACL (TACL), 2023
Detecting Personal Information in Training Corpora: an Analysis
Nishant Subramani*, Alexandra Sasha Luccioni*, Jesse Dodge, Margaret Mitchell
Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), 2023
Reproducibility in NLP: What Have We Learned from the Checklist?
Ian Magnusson, Noah A. Smith, Jesse Dodge
Findings of the Association for Computational Linguistics (ACL), 2023
Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications
Li Lucy, Jesse Dodge, David Bamman, Katie Keith
Findings of the Association for Computational Linguistics (ACL), 2023
Stubborn Lexical Bias in Data and Models
Sofia Serrano, Jesse Dodge, Noah A. Smith
Findings of the Association for Computational Linguistics (ACL), 2023
AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, Jesse Dodge
Findings of European Chapter of the Association for Computational Linguistics (EACL), 2023

2022

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Teven Le Scao, ..., Jesse Dodge, et al. (300+ authors)
arXiv, 2022
Modeling the Machine Learning Multiverse
Samuel J. Bell, Onno P. Kampman, Jesse Dodge, Neil D. Lawrence
Neural Information Processing Systems (NeurIPS), 2022
Staged Training for Transformer Language Models [video] [code]
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, Iz Beltagy
International Conference on Machine Learning (ICML), 2022
Measuring the Carbon Intensity of AI in Cloud Instances [blog] [video]
Jesse Dodge, Taylor Prewitt, Remi Tachet des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A. Smith, Nicole DeCario, Will Buchanan
Conference on Fairness, Accountability, and Transparency (FAccT), 2022
Press: [Nature] [MIT Technology Review] [IEEE Spectrum] [MarkTechPost] [SDxCentral] [TechTarget] [Dataconomy] [INDIAai] [TechCrunch] [ScienceTimes]
Data Governance in the Age of Large-Scale Data-Driven Language Technology [video]
Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Gérard Dupont, Jesse Dodge, Kyle Lo, Zeerak Talat, Dragomir Radev, Somaieh Nikpoor, Aaron Gokaslan, Peter Henderson, Rishi Bommasani, Margaret Mitchell
Conference on Fairness, Accountability, and Transparency (FAccT), 2022
Efficient NLP ACL Policy
Yuki Arase, Phil Blunsom, Mona Diab, Jesse Dodge, Iryna Gurevych, Percy Liang, Colin Raffel, Andreas Rücklé, Roy Schwartz, Noah A. Smith, Emma Strubell, Yue Zhang
Official ACL Policy Documents (website), 2022
Efficient Hierarchical Domain Adaptation for Pretrained Language Models [blog] [code]
Alexandra Chronopoulou, Matthew E. Peters, Jesse Dodge
North American Chapter of the Association for Computational Linguistics (NAACL), 2022
Press: [MarkTechPost]

2021

Can Machines Learn Morality? The Delphi Experiment
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, Yejin Choi arXiv, 2021
Press: [NYTimes] [Wired] [The Guardian] [GeekWire] [Vox.com]
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner Empirical Methods on Natural Language Processing (EMNLP), 2021
Press: [Wired] [Wired] [Unite.AI]
Competency Problems: On Finding and Removing Artifacts in Language Data
Matt Gardner*, William Merrill*, Jesse Dodge, Matthew E. Peters, Alexis Ross, Sameer Singh, Noah A. Smith
Empirical Methods on Natural Language Processing (EMNLP), 2021
* denotes equal contribution
Expected Validation Performance and Estimation of a Random Variable’s Maximum
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith
Findings of Empirical Methods on Natural Language Processing (EMNLP Findings), 2021

2020

Towards Efficient and Reproducible Natural Language Processing
Jesse Dodge
PhD Thesis, Carnegie Mellon University, 2020
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping [data]
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah A. Smith
arXiv, 2020
The Right Tool for the Job: Matching Model and Instance Complexities
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A. Smith
Association for Computational Linguistics (ACL), 2020
Green AI
Roy Schwartz*, Jesse Dodge*, Noah A. Smith, Oren Etzioni
Communications of the ACM (CACM), 2020
* denotes equal contribution
Press: [NYTimes] [Forbes] [Fortune] [Slate] [VentureBeat] [GeekWire] [MIT Tech Review x2] [Synced] [Stanford HAI] [Haartez (Hebrew)] [YNet (Hebrew)]
Chosen as Cover Article for CACM's December issue

2019

Show Your Work: Improved Reporting of Experimental Results [code] [video]
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith
Empirical Methods on Natural Language Processing (EMNLP), 2019
Press: [Wired]
RNN Architecture Learning with Sparse Regularization
Jesse Dodge, Roy Schwartz, Hao Peng, Noah A. Smith
Empirical Methods on Natural Language Processing (EMNLP), 2019

2017

Open Loop Hyperparameter Optimization and Determinantal Point Processes
Jesse Dodge, Kevin Jamieson, Noah A. Smith
AutoML Workshop at International Conference on Machine Learning (AutoML at ICML), 2017

2016

Key-Value Memory Networks for Directly Reading Documents [data]
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, Jason Weston
Empirical Methods on Natural Language Processing (EMNLP), 2016
Press: [Slate]
Evaluating Prerequisite Qualities for Learning End-to-end Dialog Systems [poster] [data]
Jesse Dodge*, Andreea Gane*, Xiang Zhang*, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, Jason Weston
International Conference on Learning Representations (ICLR), 2016
* denotes equal contribution

2015

Retrofitting Word Vectors to Semantic Lexicons
Manaal Faruqui, Jesse Dodge, Sujay Jauhar, Chris Dyer, Eduard Hovy, Noah A. Smith
North American Chapter of the Association for Computational Linguistics (NAACL), 2015
Won best student paper award
Large scale retrieval and generation of image descriptions
Vicente Ordonez, Xufeng Han, Polina Kuznetsova, Girish Kulkarni, Margaret Mitchell, Kota Yamaguchi, Karl Sratos, Amit Goyal, Jesse Dodge, Alysssa Mensch, Hal Daumé III Alexander C. Berg, Yejin Choi, Tamara L. Berg
International Journal of Computer Vision, 2015

2014

CMU: Arc-Factored, Discriminative Semantic Dependency Parsing
Sam Thomson, Brendan O'Connor, Jeffrey Flanigan, David Bamman, Jesse Dodge, Swabha Swayamdipta, Nathan Schneider, Chris Dyer, Noah A. Smith
International Workshop on Semantic Evaluations (SemEval), 2014
Context-dependent Semantic Parsing for Time Expressions [demo] [code] [tool]
Kenton Lee, Yoav Artzi, Jesse Dodge, Luke Zettlemoyer
Association for Computational Linguistic (ACL), 2014

2012

Midge: Generating Image Descriptions From Computer Vision Detections
Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Sratos, Xufeng Han, Alysssa Mensch, Alexander C. Berg, Tamara L. Berg, Hal Daumé III
European Chapter of the Association for computational Linguistics (EACL), 2012
Won 10-year Test of Time award
Understanding and Predicting Importance in Images
Alexander C. Berg, Tamara L Berg Hal Daumé III, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, Kota Yamaguchi
Computer Vision and Pattern Recognition (CVPR), 2012
Detecting Visual Text
Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa Mensch, Margaret Mitchell, Karl Stratos, Kota Yamaguchi, Yejin Choi, Hal Daumé III, Alexander C. Berg, Tamara L. Berg
North American Chapter of the Association for Computational Linguistics (NAACL), 2012