publications

2024

1. 

   DiMSUM: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation

   Hao Phung*, Quan Dao*, Trung Dao, Hoang Phan, Dimitris N. Metaxas, and Anh Tran

   In NeurIPS, 2024

     Abs   arXiv   Bib   Code   Page

   We introduce a novel state-space architecture for diffusion models, effectively harnessing spatial and frequency information to enhance the inductive bias towards local features in input images for image generation tasks. While state-space networks, including Mamba, a revolutionary advancement in recurrent neural networks, typically scan input sequences from left to right, they face difficulties in designing effective scanning strategies, especially in the processing of image data. Our method demonstrates that integrating wavelet transformation into Mamba enhances the local structure awareness of visual inputs and better captures long-range relations of frequencies by disentangling them into wavelet subbands, representing both low- and high-frequency components. These wavelet-based outputs are then processed and seamlessly fused with the original Mamba outputs through a cross-attention fusion layer, combining both spatial and frequency information to optimize the order awareness of state-space models which is essential for the details and overall quality of image generation. Besides, we introduce a globally-shared transformer to supercharge the performance of Mamba, harnessing its exceptional power to capture global relationships. Through extensive experiments on standard benchmarks, our method demonstrates superior results compared to DiT and DIFFUSSM, achieving faster training convergence and delivering high-quality outputs. The codes and pretrained models are released at https://github.com/VinAIResearch/DiMSUM.

   @inproceedings{phung2024dimsum,
     title = {DiMSUM: Diffusion Mamba - A Scalable and Unified Spatial-Frequency Method for Image Generation},
     author = {Phung*, Hao and Dao*, Quan and Dao, Trung and Phan, Hoang and Metaxas, Dimitris N. and Tran, Anh},
     booktitle = {The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS)},
     year = {2024},
   }

2023

1. 

   Flow Matching in Latent Space

   Quan Dao*, Hao Phung*, Binh Nguyen, and Anh Tran

   arXiv preprint arXiv:2307.08698, 2023

     Abs   arXiv   Bib   Code   Page

   Flow matching is a recent framework to train generative models that exhibits impressive empirical performance while being relatively easier to train compared with diffusion-based models. Despite its advantageous properties, prior methods still face the challenges of expensive computing and a large number of function evaluations of off-the-shelf solvers in the pixel space. Furthermore, although latent-based generative methods have shown great success in recent years, this particular model type remains underexplored in this area. In this work, we propose to apply flow matching in the latent spaces of pretrained autoencoders, which offers improved computational efficiency and scalability for high-resolution image synthesis. This enables flow-matching training on constrained computational resources while maintaining their quality and flexibility. Additionally, our work stands as a pioneering contribution in the integration of various conditions into flow matching for conditional generation tasks, including label-conditioned image generation, image inpainting, and semantic-to-image generation. Through extensive experiments, our approach demonstrates its effectiveness in both quantitative and qualitative results on various datasets, such as CelebA-HQ, FFHQ, LSUN Church & Bedroom, and ImageNet. We also provide a theoretical control of the Wasserstein-2 distance between the reconstructed latent flow distribution and true data distribution, showing it is upper-bounded by the latent flow matching objective.

   @article{dao2023lfm,
     title = {Flow Matching in Latent Space},
     author = {Dao*, Quan and Phung*, Hao and Nguyen, Binh and Tran, Anh},
     journal = {arXiv preprint arXiv:2307.08698},
     year = {2023},
   }

2. 

   Anti-DreamBooth: Protecting users from personalized text-to-image synthesis

   Thanh Van Le*, Hao Phung*, Thuan Hoang Nguyen*, Quan Dao, Ngoc Tran, and Anh Tran

   In ICCV, 2023

     Abs   arXiv   Bib   Code   Page

   Text-to-image diffusion models are nothing but a revolution, allowing anyone, even without design skills, to create realistic images from simple text inputs. With powerful personalization tools like DreamBooth, they can generate images of a specific person just by learning from his/her few reference images. However, when misused, such a powerful and convenient tool can produce fake news or disturbing content targeting any individual victim, posing a severe negative social impact. In this paper, we explore a defense system called Anti-DreamBooth against such malicious use of DreamBooth. The system aims to add subtle noise perturbation to each user’s image before publishing in order to disrupt the generation quality of any DreamBooth model trained on these perturbed images. We investigate a wide range of algorithms for perturbation optimization and extensively evaluate them on two facial datasets over various text-to-image model versions. Despite the complicated formulation of DreamBooth and Diffusion-based text-to-image models, our methods effectively defend users from the malicious use of those models. Their effectiveness withstands even adverse conditions, such as model or prompt/term mismatching between training and testing. Our code will be available at https://github.com/VinAIResearch/Anti-DreamBooth.

   @inproceedings{le_etal2023antidreambooth,
     title = {Anti-DreamBooth: Protecting users from personalized text-to-image synthesis},
     author = {Le*, Thanh Van and Phung*, Hao and Nguyen*, Thuan Hoang and Dao, Quan and Tran, Ngoc and Tran, Anh},
     booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
     year = {2023},
   }

3. 

   Wavelet Diffusion Models are fast and scalable Image Generators

   Hao Phung*, Quan Dao*, and Anh Tran

   In CVPR, 2023

     Abs   arXiv   Bib   Code   Video

   Diffusion models are rising as a powerful solution for high-fidelity image generation, which exceeds GANs in quality in many circumstances. However, their slow training and inference speed is a huge bottleneck, blocking them from being used in real-time applications. A recent DiffusionGAN method significantly decreases the models’ running time by reducing the number of sampling steps from thousands to several, but their speeds still largely lag behind the GAN counterparts. This paper aims to reduce the speed gap by proposing a novel wavelet-based diffusion scheme. We extract low-and-high frequency components from both image and feature levels via wavelet decomposition and adaptively handle these components for faster processing while maintaining good generation quality. Furthermore, we propose to use a reconstruction term, which effectively boosts the model training convergence. Experimental results on CelebA-HQ, CIFAR-10, LSUN-Church, and STL-10 datasets prove our solution is a stepping-stone to offering real-time and high-fidelity diffusion models. Our code and pre-trained checkpoints will be available at https://github.com/VinAIResearch/WaveDiff.git.

   @inproceedings{phung2023wavelet,
     title = {Wavelet Diffusion Models are fast and scalable Image Generators},
     author = {Phung*, Hao and Dao*, Quan and Tran, Anh},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     year = {2023},
   }

2020

1. NICS

   VQASTO: Visual Question Answering System for Action Surveillance based on Task Ontology

   Huy Quoc Vo*, Tien-Hao Phung*, and Ngoc Quoc Ly

   In NICS, 2020

     Abs   Bib   PDF

   Question answering (QA) is a popular research topic for its applications in reality. In advance, there are Visual Question Answering (VQA) researches that aim to combine visual and textual information for question answering. Their drawback is the dependence of learning models, which impedes human intervention and interpretation. To the best of our knowledge, most of them concentrate on the general problem or some specific contexts but no one puts the QA system under action surveillance context. In this paper, we propose a QA system based on Task Ontology which is mainly responsible for mapping from a question sentence to corresponding tasks carried out to reach the appropriate answer. The advantages of task ontology are the adoption of human knowledge to solve a specific problem and the reusability. The performance of the system thus heavily depends on subtasks/models. In our scope, we focus on two main subtasks: Pose estimation/tracking and Skeleton-based action recognition. Besides, we give some enhancements to improve the time efficiency of Pose estimation/tracking and propose a new spatial-temporal feature based on drawing skeleton sequence to image for skeleton-based action recognition of videos in the wild. This method, to some extent, can overcome the challenge of bad-shape pose/skeleton produced by Pose estimation on real-world videos. The hard part of Action recognition in VQASTO is that it has to get input from Pose estimation and Pose tracking which is markedly different from having available good skeletons and merely do recognition.

   @inproceedings{vo2020vqasto,
     author = {Vo*, Huy Quoc and Phung*, Tien-Hao and Ly, Ngoc Quoc},
     booktitle = {2020 7th NAFOSTED Conference on Information and Computer Science (NICS)},
     title = {VQASTO: Visual Question Answering System for Action Surveillance based on Task Ontology},
     year = {2020},
     volume = {},
     number = {},
     doi = {10.1109/NICS51282.2020.9335891},
   }