An incomplete storybook  where you can fill in blanks with sketches

Doodlering

Doodlering is a mobile web storybook generator that I made in mididice, which is a media art team that I made and is based in Seoul. I used the resources I learned from Yining Shi's Machine Learning for the Web class.

Idea

2020 has been a year of online communication. I focused on the way how people try to interpret others' intentions using technology.

While my intentions seem obvious to me, will it be still obvious to others when we are communicating online? If a machine learning model tries to interpret my intentions, how it will be different from the interpretation made by other humans?

Diagram of how Doodlering works(1): Our intentions are different from how other people or how technology perceives them.
Diagram of how the Doodleling website works(2): a SktechRNN model recognizes your sketch and lets it compare to other people's interpretations about your sketch.

Doodlering is a storybook generator made of incomplete sentences. It lets you complete the stories by drawing your own pictures. I wanted to show that while you might think other people have understood your intentions clearly, your actions interpreted by the ML model and other people in the online world might turn out as a completely different story.

Demo

​​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​doodlering.mididice.works

The website is currently still in progress and works best at 360 * 760.

ML Models

Text generation

Generating Korean text via GPT-2 in Runway, based on a dataset of Korean fables and fairytales
Editing the sentences manually to make it sound plausible

I generated the storybook sentences in GPT-2 via Runway, based on a dataset of Korean fables and fairytales where I could find from Korean Learner's Corpus Search Engine and the National Institute of Korean Language's open database.

I deliberately chose fairytales as dataset to set the story's background as recalling a dream. The most primary reason is also because I knew combining different sentences won't produce a solid storyline, and I needed a background that would still be plausible even if the sentences looked strange.

Sketch recognition

Doodlering uses DoodleNet's CNN image classifier and ml5.js to recognize the user's sketches.

Credits

mididice

  • Interaction design: Jeeyoon Hyun, Juyeon Park
  • Software engineering: Hyunki Min, Changmin Kim