paint with ml

09/16/2020 Data.

Paint with ML is an experimental ML art web application that lets you design your own Bob Ross style landscape painting. You build a semantic segmentation mask using a set of nine different semantic brushes, each one a different landscape painting element. Clicking on the "Run" button feeds this semantic map through an image-to-image translation network, generating a painting as output.

This project was initially a freelance consulting project for Spell, an end-to-end machine learning infrastructure platform startup. Some of the finishing touches, the application design especially, were made after I was hired onto Spell full time.

The deep learning model powering the application is a PyTorch GauGAN. The model was trained on the ADE20K scene parsing benchmark dataset, then fine-tuned on a custom dataset of 250 Bob Ross paintings. Model training and deployment (to a Kubernetes-based model server endpoint) was via the Spell SDK.

The application frontend uses React, HTML Canvas, and AWS S3 Static Hosting. The code is available publicly on GitHub; you can try it out for yourself here.

— Aleksey