Automated front-end using deep learning
Overview
Creating intuitive and engaging experiences for users is a critical goal for companies of all sizes, and it’s a process driven by quick cycles of prototyping, designing, and user testing. Large corporations like Facebook have the bandwidth to dedicate entire teams to the design process, which can take several weeks and involve multiple stakeholders; small businesses don’t have these resources, and their user interfaces may suffer as a result.
A typical design workflow might look like the following:
- Product managers perform user research to generate a list of specifications
- Designers take those requirements and explore low-fidelity prototypes, eventually creating high-fidelity mockups
- Engineers implement those designs into code and finally ship the product to users
The problem I was solving falls under a broader umbrella of tasks known as program synthesis, the automated generation of working source code. Though much of program synthesis deals with code generated from natural language specifications or execution traces, in my case I could leverage the fact that I had a source image (the hand-drawn wireframe) to start with.
There’s a well-studied domain in machine learning called image captioning that seeks to learn models that tie together images and text, specifically to generate descriptions of the contents of a source image.
Using an image captioning model architecture
Now that I had my data ready, I could finally feed it into the model!
I leveraged a model architecture used in image captioning that consists of three major parts:
- A computer vision model that uses a Convolutional Neural Network (CNN) to extract image features from the source images.
- A language model consisting of a Gated Recurrent Unit (GRU) that encodes sequences of source code tokens.
- A decoder model (also a GRU), which takes in the output from the previous two steps as its input, and predicts the next token in the sequence.
Dependencies
- Python
- Keras
- tensorflow
- nltk
- opencv
- numpy
- matplotlib
- Pillow
- scipy
How to run
Converting an example drawn image into HTML code, using pretrained weights:
python convert_single_image.py –png_path ../examples/drawn_example1.png \ –output_folder ./generated_html \ –model_json_file ../bin/model_json.json \ –model_weights_file ../bin/weights.h5
