Problem Space

We visited CMNH several times to find out pain points from the visitors' perspective, observe visitors, and study how dioramas in the botany hall were communicating deeper information.

What Visitors Feel

1

2

The museum is too large and complicated.

Only share thoughts within the group.

3

4

The experience ends leaving the museum.

Text is heavy.

Besides the interview, we also studied the vision of the Carnegie Museum of Natural History which is to become the world’s most relevant history museum by enabling visitors to discover greater stories, such as climate change and connections with humans. 

What CMNH Needs

We had an opportunity to speak with Nicole Heller, an associate curator of CMNH. We could come up with some points to consider from an information providers' perspective.

Highlighting the interdependent relationship between human and non-human life

It seems irrelevant, but human activity and other species and nature are interdependent.

Fostering communications among people

Exchanging thoughts and feelings would make visitors' experience richer, and help curators learn what influences they are making.

Inviting visitors beyond the first floor

How might we help or trigger visitors to explore the museum's massive collections?

Adding little touches than reinventing

Instead of renovating the whole exhibition hall, adding up-to-date scientific discoveries would be quicker and easier.

1

2

3

4

"This refresh emphasizes the Hall of Botany’s relevance in the context of environmental change and the importance of plants in our daily lives."

We found that the museum renovated its botany hall in May 2021, and the emphasis of the renovation aligned with the museum's vision and our insights. However, it seemed that people just skim the space and pass by. So I spent hours in the Hall of Botany to learn why people just walk around and head out. After a while, we color-coded each diorama into two categories, PURPLE for the usefulness of plants, and GREEN for the interdependent relationship.

Two main themes. Environmental change or importance of plants in human life.

3

"Can you find it?" games are barely noticeable.

We decided to enhance this 'Can you find it?' experience more engaging using technology.

There is no one narrative. Each diorama has a separate theme from the other.

1

2

The Hall of Botany,

one of the most empty halls despite the renovation in May 2021.

Mixed Reality Museum Experience

Beyond

My Roles

Research, Concept Development, AR UI Design, Mobile UI Design(Main), Visual Design(Main), Motion Design, 3D modeling, AR filter(Main)

Tools

After Effect, Illustrator, Figma, Spark AR, Protopie, Unity, Finalcut Pro

Timeline

4 weeks (2021)

Team

Greg Chen, Wei-chieh Wang, Zhuoneng Wang

Overview

We were tasked to improve the user experience of the Carnegie Natural History Museum which has remained relatively static for the past decade.

Museums play an important role in society, as illustrated by the Smithsonian’s vision “...to engage and to inspire more people, where they are, with greater impact, while catalyzing critical conversation on issues affecting our nation and the world.” Today, many museums are also facing the challenge of transitioning from rigid institutions to experiential and flexible spaces. This is driven by such factors as expanding collections, increased competition for visitors, and visitor expectations for greater engagement. Museums are turning to virtual reality, apps, and interactive experiences to attract visitors.

Outcome

A Mixed Reality museum filled with gamified learning experiences.

Learn like playing a game.

Learn deeper stories behind the scenes and collect objects as rewards.

Get inspirations.

Create AR filter with your collections, and share your story with other people. #CMNH #Beyond

It's easy to send your collections to your smartphone. 

Take your learnings with you.

Discover what people encountered in the museum that you might never have imagined.

Have fun and share.

Concept Development

How might we provide engaging learning experiences and enable visitors to advocate for a sustainable future?

Project Objectives

We set project objectives that we should achieve at the end of our project out of the insights from our research. 

Concept Ideation

We got inspiration from various existing ar experiences. From AR games to social media content, we tried to combine each unique experience into a coherent and engaging journey.

1

Scavenger hunt

2

Mini-games

3

AR filter for Instagram/ Tiktok

Technology

1

HoloLens 2

In-museum Experience

In 5-10 years, we expected that people could own MR/AR glasses. 

We combined mainly two technologies; MR glasses, and mobile AR. For the compatibility between two devices, we adopted Bluetooth as the network technology between MR glasses and mobile devices.

Considering privacy issues that may occur when people share public devices, we decided to use Bluetooth as a network technology between glasses and personal devices.

2

Mobile AR

Post-museum Experience

Storyboard

As a starting point, we sketched interactions in storyboards. This process helped us imagine not only how users interact with virtual objects, but also about onboarding, signage, systems, customer services, and technologies.

Sketchs for initial ideas about mobile experience.

Sketchs for initial ideas about scavanger hunt.

Journey Map

By multiple iterations, we could organize steps and designate spots for each step of the journey based on the museum's map.

Design Opportunities

We combined mainly two technologies; MR glasses, and mobile AR. For the compatibility between two devices, we adopted Bluetooth as the network technology between MR glasses and mobile devices.

Typography in Mixed/Augmented Reality

Even though main objects in virtual world are 3D based, there should be text-based information too. Here, typography plays a key role not only in delivering information, but also in minimizing distractions derived from surrounding visual elements. It was a great chance to think about how to achieve that learning basic rules to follow for legibility.

Reflections

Devices, Technology 

We researched and tried different kinds of glasses. It helped me deepen my knowledge about the technology.

Knowing the Client

I spent a quite long time in the museum. I read all texts in the plaque in the Hall of Botany. That helped me point out what their intention, and what is successful and what is not.

Learnings

Next Step

Prototyping for typography, interaction testing

One thing I really want to know is how legible the information and how visible the visual signals in actual context. If I had capability and time, I would love to try them built in the device and test my design decisions.

Reflections

Interaction Design

Gestures

While wearing MR glasses, hand gestures became critical in interacting with user interfaces. For each step that requires interactions with interactable objects, we spent a long time to discuss intuitive gestures

To start, the instruction guide users to raise their wrist to introduce wrist UI.

1

Wrist-raise

In front of the diorama, interactable objects are highlighted when hovered.

2

Palm-hover

After hovering selection, pinching and dragging motion pulls objects in front of the user.

3

Pinch and Drag

The pinching gesture applies to every interaction with 3D objects when trying to grab them.

After hovering selection, pinching and dragging motion pulls objects in front of the user.

4

Drag and Drop

AR UI Research

Before diving into designing interfaces, I referred to Microsoft's Mixed Reality documents to learn basic rules.

1

Optimal Zone

2

Size and Feedback

I could get how interactions differ depending on the distance.

3

Typography

I was able to check the minimum font sizes for each distance interaction.

4

Common controls and behaviors

I looked into MRTK Figma files to learn about the sizes of components.

Beyond's main experience was diorama games to unlock collectable items. As this part has fair amount of information, it was inevitable to show lots of texts. Texts in this UI could be categorized into a piece of information or labels of interface. 

Color/ Shape/ Saturation test for different use and variants. 

The difficulty here was how to represent button features as 3D objects. Using only colors and shapes were not clear enough. But we didn't want to make buttons look like a screen UI element (rectangular flat button with labels.) So we put labels under the 3d diamond button using the same style as the wrist UI components to look like a lable of the button components.

We created 3D objects for interaction, 2D graphics for information.

Mobile UI Design

We sorted out key screens of mobile app. Users can see other people's thoughts and filters which are presented on the sharing wall in the museum lobby. We included users' visiting history, collection, and filter tabs.

Users can access the information they learned in the museum through history and collection tabs. Also, there are two routes to creat AR filters through collection and filter tabs. 

The AR filter creation is consisted of simple steps. 1) resize, 2) place at a few preset anchor points, 3) record&share.

Wrist UI

The control panel on the wrist is anchored on the wrist band which also serves as an admission ticket. The MR glasses recognize the anchor, which is marked on the center of the wrist with Beyond's logo, and place the UI on the anchor. 

The main panel transforms to different states from an assistant's screen to a collecting spot according to the users' gestures and situations.

When the top handle being pulled, a collection panel opens, and when the users tough and hold one of their collections, the panel turns into a panel where they can place their collections around their body.

1

2

3

Prototyping

Zhuoneng built prototyping to test basic interactions. We could resize, rotate, and move the lily, and made the information tags rotate accordingly, while text panels facing front.

We presented our demo using a labtop and a leap motion controller for our audience to try basic interactions.

Mobile AR Filter

I tried creating an AR filter in Spark AR to explore how we can combine the AR filter creation process in our users' journey. It took some time to get used to the tool, so we decided to make the step as simple as possible. 

Designing Interactions