Jieyu Zhang
Ph.D. Student @ UW CSE
(* denotes equal contribution, = denotes student I mentor)
Preprint
- ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models
Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma, Juan Carlos Niebles, silvio savarese, Caiming Xiong, Zeyuan Chen, Ranjay Krishna, Ran Xu. - Generate Any Scene: Evaluating and Improving Text-to-Vision Generation with Scene Graph Programming
Ziqi Gao*=, Weikai Huang*=, Jieyu Zhang, Aniruddha Kembhavi, Ranjay Krishna. - TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action
Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, Silvio Savarese. - EcoAct: Economic Agent Determines When to Register What Action
Shaokun Zhang, Jieyu Zhang, Dujian Ding, Mirian Hipolito Garcia, Ankur Mallick, Daniel Madrigal, Menglin Xia, Victor Rühle, Qingyun Wu, Chi Wang. - Language Model Preference Evaluation with Multiple Weak Evaluators
Zhengyu Hu=, Jieyu Zhang, Zhihan Xiong, Alexander Ratner, Hui Xiong, Ranjay Krishna. - Adaptive In-conversation Team Building for Language Model Agents
Linxin Song*, Jiale Liu*, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, Chi Wang.
2024
- Task Me Anything
Jieyu Zhang, Weikai Huang*, Zixian Ma*, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, Ranjay Krishna.
NeurIPS 2024 | Video-Language Models @ NeurIPS 2024 Oral Presentation
Blog at Snorkel AI | Talk at Snorkel AI - DataComp-LM: In search of the next generation of training sets for language models
59 authors.
NeurIPS 2024 - xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, Ran Xu.
ECCV EVAL-FoMo Workshop 2024 - m&m’s: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks
Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, Ranjay Krishna.
ECCV 2024 - Offline Training of Language Model Agents with Functions as Learnable Weights
Shaokun Zhang*, Jieyu Zhang*, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, Qingyun Wu.
ICML 2024 - SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
Xiaoxuan Wang*, Ziniu Hu*, Pan Lu*, Yanqiao Zhu*, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, Wei Wang.
ICML 2024
Nature News Feature - Iterated Learning Improves Compositionality in Large Vision-Language Models
Chenhao Zheng=, Jieyu Zhang, Aniruddha Kembhavi, Ranjay Krishna.
CVPR 2024. - EcoAssistant: Using LLM Assistant More Affordably and Accurately
Jieyu Zhang, Ranjay Krishna, Ahmed Awadallah, Chi Wang.
LLM Agents @ ICLR 2024 - AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Awadallah, Ryen White, Doug Burger, Chi Wang.
COLM 2024 | LLM Agents @ ICLR 2024 Best Paper
Github 30K+ Star & 4K+ Fork
The Economist article | The Forbes article
2023
- SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
Cheng-Yu Hsieh*, Jieyu Zhang*, Zixian Ma, Aniruddha Kembhavi, Ranjay Krishna.
NeurIPS 2023 - DataComp: In Search of the Next Generation of Multimodal Datasets
34 authors.
NeurIPS 2023 Oral Presentation - On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training
Jieyu Zhang*, Bohan Wang*, Zhengyu Hu, Pang Wei Koh, Alexander Ratner.
NeurIPS 2023 - Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
Yue Yu*, Yuchen Zhuang*, Jieyu Zhang*, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, Chao Zhang.
NeurIPS 2023 - Characterizing the Impacts of Semi-supervised Learning for Weak Supervision
Jeffrey Li, Jieyu Zhang, Ludwig Schmidt, Alexander Ratner.
NeurIPS 2023 - Subclass-balancing Contrastive Learning for Long-tailed Recognition
Chengkai Hou=, Jieyu Zhang, Haonan Wang, Tianyi Zhou.
ICCV 2023. - When to Learn What: Model-Adaptive Data Augmentation Curriculum
Chengkai Hou=, Jieyu Zhang, Tianyi Zhou.
ICCV 2023. - Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision
Jieyu Zhang*, Linxin Song=*, Alexander Ratner.
AISTATS 2023
2022
- Understanding Programmatic Weak Supervision via Source-aware Influence Function
Jieyu Zhang*, Haonan Wang*, Cheng-Yu Hsieh, Alexander Ratner.
NeurIPS 2022 - Creating Training Sets via Weak Indirect Supervision
Jieyu Zhang, Bohan Wang, Xiangchen Song, Yujing Wang, Yaming Yang, Jing Bai, Alexander Ratner.
ICLR 2022 - Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Cheng-Yu Hsieh, Jieyu Zhang, Alexander Ratner.
VLDB 2022
2021
- WRENCH: A Comprehensive Benchmark for Weak Supervision
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, Alexander Ratner.
NeurIPS 2021 Oral Presentation