ML Revisit: Unified Multimodal Models

Transfusion

论文：(Zhou et al., 2024)

大概就是用一个 Transfusion 去同时做生成和理解两件事情：

Attention mask 大概是这样，就是文字看到前面的，然后图片互相都能看到。

Transfusion causal and bidirectional attention mask
	A	cute	cat	<BOI>					<EOI>	What
A	Allowed	Masked	Masked	Masked	Masked	Masked	Masked	Masked	Masked	Masked
cute	Allowed	Allowed	Masked	Masked	Masked	Masked	Masked	Masked	Masked	Masked
cat	Allowed	Allowed	Allowed	Masked	Masked	Masked	Masked	Masked	Masked	Masked
<BOI>	Allowed	Allowed	Allowed	Allowed	Masked	Masked	Masked	Masked	Masked	Masked
	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Masked	Masked
	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Masked	Masked
	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Masked	Masked
	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Masked	Masked
<EOI>	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Masked
What	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed	Allowed

References

Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., & Levy, O. (2024). Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. arxiv.org

Transfusion

References

Cite this post