Augmented Reality Interface for NASA Lunar Missions
Problem: How can we give astronauts autonomy to complete their tasks in space without the help of Mission Control?
​
Solution: Provide AR tools to help assist and automate the process of major tasks in Space.
​
​
​
Time Frame: 1.5 years
​
Team: 16-20 Designers & Engineers
​
My Role: Project Co-Lead, Design Lead
​
Client: NASA
​
​
Finished Product
AR Interfaces
To assist astronauts during their extravehicular activities
Including navigation on the lunar surface, vitals display, geological sampling tools, search and rescue
Context
Our team's HoloLens Interfaces getting tested by NASA Designer Skye​
NASA Spacesuit User Interface Technologies for Students (SUITS) Challenge selected our team to develop Augmented Reality (AR) software on the HoloLens.
With the HoloLens, we can create 3 types of visual elements...
Registered in the environment in a set location
Sticky interface registered in the environment following the user's view
​
Registered to the hud following the user's view
Team & Milestones
Team: 16-20 members
(responsibility changing throughout the challenge)
​
2 Team Leads: Bowen(Me) & Selena
​
Design Lead: Bowen(Me)
4 Design Sub-Leads:
Geological Sampling
EVA
Navigation
Auxiliary
​
2 Software-Leads:
Frontend
Backend
Research
We talked to: 5 AR, voice, navigation designers, 2 NASA Designers, 2 Brown Planetary Geo-science Professors, and 2 Astronauts!​
James H. Newman
Former NASA astronaut who flew on four Space Shuttle missions.
Steve Swanson
Retired NASA astronaut who flew on three missions to the ISS.
James W Head
Worked on previous Apollo Programs and trained astronauts. Geological Science Professor at Brown.
What are some challenges astronauts face on moon?
Difficult to Navigate
Dangerous terrain with tripping hazards
Environment with no distinctive features for landmark recognition
No GPS, relied on route memorization and photos
" There are no high mountains or volcano - something distinct that you could recognize - a landmark would help tremendously."
" Nothing is given to us right now. We practice memorizing our paths and rely on some pictures. There was one time where astronauts got lost and couldn't find a destination so they had to abandon the tasks there."
Limited Movements
Whole-body movements are required to accomplish simple tasks
Small hand gestures require a large amount of effort
Both hands are required for most tasks
"Glove bulky, no tactile feedback. you are basically in a balloon. Any time you move you are squeezing the balloon."
Limited time
Each minute in space wasted is a large sum of money lost. Mistakes are costly and there are a lot of tasks to accomplish.
What are their main objectives on the moon?
Navigation
Long-Range Point A to Point B
Short Range Terrain mapping & Obstacle Avoidance
"We bounce around a lot to navigate. The lighting conditions can be extreme and a lot of hazards are hiding in the shadows. We must always pay attention to the ground in front of us."
Geological Sampling
Taking Samples
Documentations
"We have specific goals when we arrive at a site, we communicate with the geologists at Mission Control and take exact samples. We take notes, measurements, and photos of the site. "
Staying Alive: Vitals & Rescue
Vitals Display
Emergency Navigation & Communication
"If something goes wrong, how quickly can you go back? You always need to know what's wrong, how much oxygen you have left, and what pace you need to travel at under an emergency."
Design Process
Objective-based User Profiles
I separated our team into design sub-teams based on the main mission objectives and conducted 3 scenario-specific user research.
Navigation
Extravehicular Activities (EVA)
Geological Sampling
Central Profile
Anne McClain
​
Anne is going on the Artemis Mission where she would travel to multiple unexplored locations on the moon for scientific discovery as well as traveling to established sites to update equipment left by previous missions. Show would have a partner traveling with her and another astronaut in the spacecraft.
Anne's Artemis mission mostly consist of...​
​
Traveling on foot for around 15-20min intervals
Taking samples and documentation from unexplored sites
Maintenance and setting up new equipments
Ideation Summery
Based on the challenges & objectives, the teams went through lighting rounds of ideations.
.jpg)
💡 Learning to eliminate
We learned to rigorously eliminate ideas and concepts. There were a lot of amazing ideas that would improve the AR experience which we had cut out because we couldn't do everything.
​
We have to ask ourselves, is this function/feature necessary to achieve our goals? If not, let's put these aside into our future features bank.
Flow Diagrams
Simplified This...
To This!
Navigation
.jpg)
Vitals, Geo-sampling, Emergency Communication
Design Iterations
7+ Rounds of Iterations
_Page_2.jpg)
Iteration Learnings
💡Lessen & attach elements in the environment
Problem: During testing, we observed that users are often confused when there is more than 2 sticky interfaces floating and following them in the environment. It became hard for them to understand where things are especially when they are doing other physical tasks at the same time.
Solution: Minimize the amount of interface registered in the enviorment that follows the user's view to one at all times. Attach elements to each other into a unity to avoid spatial confusion.
💡Set separate use case for each plane of interaction
Problem: Users' attention can only focus on 1 plane of interaction at a time. when they are looking at elements registered in the far environment, they miss elements on the floating interface right in front of them, and vice versa.
Solution: Create different use cases for each plane of interaction so users' attention doesn't have to be separated during a flow. We limited all decision-making processes where users have to be focused on the screen to the sticky interfaces floating right in front of them, all guiding tools such as compass and time to the hud, and all elements that are used while users are actively doing something else to the environment, such as the guiding pathway in navigation mode.
Iterations: instead of a sticky arrow, directions are given with a pathway in the environment.
Decisions such as setting a destination and checking vitals are limited to the sticky interfaces close to the user.
_Page_4.jpg)
_Page_3.jpg)
💡Automate processes & combine features
Problem: From astronauts, we learned that they have limited movements and small hand gestures require extra effort. To save astronauts' physical energy and attention, we have to minimize the amount of input needed in a flow.
Solution: On top of designing voice commands, we automated each flow as much as possible. letting one action directly leads to the next, and interfaces open automatically. We combined multiple features into one, and ask ourselves what is the least amount of elements needed to achieve a goal?
Automate Processes
Combining Features
Mini-map's function of illustrating direction could be eliminated by adding a direction indicator on the compass. It's close range function of showing the route could be performed by having a navigational pathway rather than arrows in the environment.
Mini-map
%20%E2%80%93%201.png)
Red direction indicator on compass
_Page_3.jpg)
Pathway in environmental showing approximate closeness to destination with color
💡Clear the field of view,
Test in the HoloLens early!
Problem: From astronauts, we learned that they have limited movements and small hand gestures require extra effort. To save astronauts' physical energy and attention, we have to minimize the amount of input needed in a flow.
Solution: On top of designing voice commands, we automated each flow as much as possible. letting one action directly leads to the next, and interfaces open automatically. We combined multiple features into one, and ask ourselves what is the least amount of elements needed to achieve a goal?
Final Designs
_Page_4.jpg)
Finished Products & NASA on-site Testing
Feedbacks from NASA
Validations
Speeds up the Current Meeting System
" If you have access to all of this, it will walk you through the process. You would know this is something you have to talk about, and you can set it up right here. "
Opens a conversation up
" I like that you are forcing people to write out stuff about the meeting beforehand, so the conversation is open before we got there. The hardest part is starting the conversation. This makes it easier. "
Reduce Barriers
" This could give people access to help 24/7 without having to go through a supervisor, which is embarrassing if you want to talk about mental issues, and takes forever."
​
"This would be a less stressful and easier way for someone to reach out and talk about a touchy topic."
Moving Forward...
1
Test the effectiveness for this format of conversations, which involves in-person or video conferencing with three participants, having a structured timeline, etc.
2
Test for the ideal number of people there can be in a conversation intended for learning and understanding
3
If the tests indicate difficulty in conversations, test another concept which involves having a third experienced party to facilitate the conversation
4
Start researching and designing the platform that civilians and civilians in professional fields use to connect with the officers.
Thank You!
Thank you! I am very excited to show you this work and I would love any criticism and welcome any feedback from you!