Dreamscape | meichuncai

Dreamscape

Data-driven Auto-Generation of Dreams

DreamScape is a multi-sensory representation of dream reports for self-reflection and dream sharing. We created a system of auto-generated and immersive 3D scenes based on users' inputs (i.e. text) to support individual emotions and collective empathy in a long term.

Role | HCI research, system design and prototyping;

collaborated with Boyu Z. and Chenlu W.

Affiliation | MIT Media Lab Fluid Interfaces Group

Tools | Unity, C#, Rhinoceros, GANs

Year | 2021

Demo.

View Full Paper

A demo generated by a dream report of happiness

[Dream Report of Happiness] I had a dream where I was riding in a beautiful car on a highway over the Pacific Ocean. The beautiful scenery and the ocean I can recall. I remember how relaxed I was taking this trip and how much I was enjoying it. We got off by the ocean and stood on the cliff and watched the waves come up as we felt the clean air and the beauty of nature. We traveled around visiting different places and enjoying the various sights. The participants of this dream were my boyfriend, 23, and I. The colors remembered are blue and green.

Characters: adult*2
Activities: riding, traveling
Social Interactions: with boyfriend
Emotions: happy (relaxed, enjoy)
Objects: car, highway, ocean, cliff, scenery
Descriptive Elements: beautiful, clean, blue, green
Setting: outdoor

A demo generated by a dream report of sadness

[Dream Report of Sadness] I found myself walking on a huge green lawn and saw distant tall building. The lawn was flanked by banks of flowers. The building looked new and white. As I was approaching it, I felt a sense of loneliness, and felt myself a stranger as I neared the building. There was no one else in the dream, and I did feel a little sad because I did not know where my family was.

Characters: adult*1
Activities: walking, approaching
Social Interactions: none
Emotions: a little sad (lonely)
Objects: lawn, flowers, tall building
Descriptive Elements: huge, distance, green, white
Setting: outdoor

Process.

Video & Audio Generation

We aim to generate videos and audios from dream reports, i.e., text data. We first utilized the Dreambank [5] dataset, a collection of over 20,000 dream reports. These dream reports are documented by various dreamers (ages 7 to 74) from the first-person perspective, and there is no particular gender, occupation, or race group.

Generated videos based on dream reports

In order to recreate the dreams with videos, we adapted VQ-GAN [7] and CLIP [6]. VQ-GAN is a generative adversarial neural network that leverages the Transformer [8] architecture to generate realistic images from image constituents through CNN feature extraction. CLIP (Contrastive Language–Image Pre-training) is trained to measure how well a text description matches an image. Specifically, we utilized CLIP as the “coach” of VQ-GAN, pushing VQ-GAN to generate images that best match the dream report (text prompt) during the process. Each dream report is essentially a narrative story, and it can be converted to a video. In the end, we treat each sentence in the dream report as a frame and generate a single image for each sentence. By connecting the generation process of each image, we obtain the final video.

Anger

Happiness

Sadness

Confusion

apprehension

Generated audios based on emotions indicated in dream reports

For the audio generation, we focused on the emotions in dream reports. We adapted the sequence and conditional generation methods from [9]. We first trained a 4-class emotion classification model on all the dream reports where the four categorical labels are the four quadrants from the valence-arousal dimensions. We then applied the Compound Word Transformer model to generate a piece of pop piano music for the dream report based on its overall valence and arousal.

Dream Mapping

For each dream, we parsed the test and got each dream labeled to indicate the emotion, interactions, figures, and activities according to the Hall/Van de Castle Dream Coding System dream coding rule. We focused on the characters, social interactions, activities, emotions, and settings as the key elements to be analyzed. For example, we used different body animations to indicate the social interactions and the emotional levels. We also took descriptive words including modifiers(i.e. color, size, density, evaluation), time(i.e. day, night, long time, a few mine), and negatives (i.e. not, no one, never, none) and mapped them to the environments in Unity.

Hall/Van de Castle Dream Coding System

Rules for Coding a Dream
Elements of a Dream:
- Characters
- Social Interactions
  - Aggression
  - Friendliness
- Activities
- Success and Failure
- Misfortune and Good Fortune
- Emotions
- Settings

Result.

We created a series of generated 3D scenes in Unity based on the existing dream reports. The figures distinguished themselves from each other according to their sex, ages, and identities. Body animations are applied to represent different kinds of social interactions such as sexuality, friendliness, and aggression. The overall moods reflect the emotions as well as their intensities suggested in the dream. For example, a very sad dream is shown in monochrome. Other environmental elements(i.e. weather, brightness, colors) are also included to further enhance the atmosphere. We learned that it’s inspiring to represent dreams and invite people to share, experience, and reflect in a multisensory method. The scene mapping system design can be more creative. For this stage, we mainly focus on producing a general system that can apply to all - the grid. But when applied to wider varieties of dreams, we need to make the scene more customized.

Anger

annoyed, irritated, mad, provoked, furious, enraged, belligerent, incensed, and indignant

Apprehension

terrified, horrified, frightened, scared, worried, nervous, concerned, panicky, alarmed, uneasy, upset, remorseful, sorry, apologetic, regretful, and ashamed

Sadness

disappointed, distressed, hurt, depressed, lonely, lost, miserable, hopeless, crushed, and heartbroken

Confusion

surprised, astonished, amazed, awestruck, mystified, puzzled, perplexed, strange, bewildered, doubtful, conflicted, undecided, and uncertain

Happiness

contented, pleased, relieved, amused, cheerful, glad, relaxed, gratified, gay, wonderful, elated, joyful, and exhilarated

Discussion & Future Work.

There are three directions for possible future work:

Providing more dynamic 3D representations of dream contents
Integrating with more modalities as inputs and outputs
Conducting usability testing of the product’s boosting creativity and enhancing collective empathy

Digital Human <<

View All Projects

>> HCI/Computation