This repository provides a PyTorch implementation of layout2im, which can generate images from layouts.

Online demo can be found here.

Image Generation from Layout
Bo Zhao, Lili Meng,Weidong Yin, Leonid Sigal
The Univerisity of British Columbia, Vector Institute
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, (Oral)

Overview of our Layout2Im networks for generating images from layout during training.

All code was developed and tested on Ubuntu 18.04 with Python 3.6 (Anaconda) and PyTorch 1.0.

$ conda create -n layout2im python=3.6
$ conda activate layout2im

$ cd ~
$ git clone https://github.com/cocodataset/cocoapi.git
$ cd cocoapi/PythonAPI/
$ python setup.py install
$ cd ..

$ git clone [email protected]:zhaobozb/layout2im.git
$ cd layout2im

$ pip install -r requirements.txt

To download COCO dataset to datasets/coco:

$ bash scripts/download_coco.sh

To download Visual Genome dataset to datasets/vg and preprocess it:

$ bash scripts/download_vg.sh
$ python scripts/preprocess_vg.py

Download the trained models to checkpoints/pretrained/.

1. trained model on COCO
2. trained model on Visual Genome

Testing on COCO dataset:

$ python layout2im/test.py --dataset coco --coco_dir datasets/coco \
                           --saved_model checkpoints/pretrained/netG_coco.pt \
                           --results_dir checkpoints/pretrained_results_coco

Testing on Visual Genome dataset:

$ python layout2im/test.py --dataset vg --coco_dir datasets/vg \
                           --saved_model checkpoints/pretrained/netG_vg.pt \
                           --results_dir checkpoints/pretrained_results_vg

$ python layout2im/train.py

The training script has a number of command-line flags that you can use to configure the model architecture, hyperparameters, and input / output settings:

• dataset: The dataset to use for training; must be either coco or vg. Default is coco.
• vg_dir: The directory of visual genome dataset. Default is datsets/vg.
• coco_dir: The directory of coco dataset. Default is dataset/coco.
• batch_size: How many pairs of (layout, image) to use in each minibatch during training. . Default is 8.
• niter: How many iterations will the model be trained. Default is 300000.
• image_size: The size of images to generate. Default is 64.
• object_size: The size of object to be resized to. Default is 32.
• embedding_dim: Integer giving the dimension for the word embedding for objects category. Default is 64.
• z_dim: Integer giving the dimension for the appearance code for objects. Default is 64.
• learning_rate: Learning rate to use in Adam optimizer for the generator and discriminators. Default is 1e-4.
• resi_num: How many residual blocks are used before image decoder. Default is 6.
• clstm_layers: How many layers are used for convolutional lstm. Default is 3.
• lambda_img_adv: The weight to assign to adversarial loss for image. Default is 1.0.
• lambda_obj_adv: The weight to assign to adversarial loss for object. Default is 1.0.
• lambda_obj_cls: The weight to assign to auxiliary loss for object classification. Default is 1.0.
• lambda_z_rec: The weight to assign to appearance code regression. Default is 10.0.
• lambda_img_rec: The weight to assign to image reconstruction loss. Default is 1.0.
• lambda_kl: The weight to assign to KL divergence. Default is 0.01.
• resume_iter: Which iteration to resume from; must be either l, s or the integer of previously saved iteration. l means training from latest saved model, s means training from scratch. Default is s.
• log_step: How many steps to print log information. Default is 10.
• tensorboard_step: How many steps to update tensorboard. Default is 100.
• save_step: How many steps to save models. Default is 1000.
• use_tensorboard: Whether to use tensorboard. Default is true.

If you find this code useful in your research then please cite

@inproceedings{zhaobo2019layout2im,
  title={Image Generation from Layout},
  author={Zhao, Bo and Meng, Lili and Yin, Weidong and Sigal, Leonid},
  booktitle={CVPR},
  year={2019}
}

Our project borrows some source files from sg2im. We thank the authors. This research was partially supported by NSERC Discovery, NSERC DAS, NSERC CFI grants and Mitacs GlobaLink Award. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research.