Research Focus

Jason Yip’s research is interdisciplinary and collaborative, intersecting human-computer interaction, learning sciences, and information sciences. His research focuses on how digital technologies support collaborative learning in children and families. Prior research demonstrates that children’s joint media engagement around digital technologies with their families can play a positive role in learning. He investigates how to build family-based collaborative learning technologies, develop new methods for design, and examining how families utilize current technologies for learning together.

The COVID-19 pandemic has forced the world of learning out of our traditional classrooms or campuses. Instead, our teachers, children, and parents are in front of screens, struggling with online tools that were not designed for all learners of all ages. Specifically, children’s play, collaboration, and designing and playing with physical objects with other children can hardly be done online. A new genre of technology tools, activities, and playful interactions needs to be conceptualized.

My plans for the fellowship period

My team proposes to investigate how children physically co-design across online spaces. As an alternative, we propose to investigate remote collaborative physical design through Augmented Reality (AR) where two or more people create, share and experience physical objects directly in their physical environment. An interface to support such activities would enable design partners like children to (1) capture physical objects they create or find in their environment (2) sketch, annotate and share 3D designs in real-time, and (3) modify and arrange them remotely. Key to this experience is that these experiences should support improvised activities: rapid iteration and sketches over precision, fast turn-taking and simultaneous interaction, and use of the body and objects in the environment, rather than a slow design process requiring many steps.

Formative Study. My team will conduct a formative study of how children design together with current remote collaboration tools like video chat and shared docs. We will conduct a survey of educators that had to move their classroom experiences online recently. A source of insights leading to this work is KidsTeam UW, an intergenerational co-design team of children (ages 7 – 12) and adults who shifted from physically co-located co-design to an online video chat tool.

We have already conducted numerous pilot studies (Summer and Fall 2020) with online co-design to determine the needs, challenges, and design guidelines to transition RealitySketch (Daniel Leithinger) into a tool to help children connect physical prototypes together. Based on the lessons learned by our collaborators, we will design a survey questionnaire for educators to learn about current remote design practices and challenges.

Prototype Tool. Based on the insights of the study, we propose a prototype mobile AR application that scans the shape of physical objects, tracks spatial drawing tools, and transmits the geometry, sketches and video between connected mobile devices. The tool will be based on a mobile handheld device and use video see-through AR. A key consideration in addition to the scanning and sketching tools is video and voice for communication, and to what extent a remote design partner’s viewing position and orientation should be visualized for increased presence. Based on preliminary experiments and prior work, the role of the physical objects may vary in the design process: They can provide a template for geometry to be scanned, become a storytelling device, or a tangible token to manipulate virtual objects.

We will test this tool with spatially distributed groups of both children (ages 7 – 12), public school students, and college-level design and engineering students, leading to insights on how this AR tool can support remote connectedness during physical co-design together. My team already committed to integrating KidsTeam UW towards this project to conduct numerous co-design sessions and testing of the prototype tool over the course of the year.

How will my work change children’s and youth’s lives?

If we are able to get children to successfully physically co-design with our prototype tool, we will be able to scale this tool outwards to other design contexts that have been limited by remote online setups. Working with Pratt Institute collaborators (Dr. Allison Druin, Professor Amanda Huynh), we will be able to integrate this prototype tool into different online schools and learning environments.

The key aspect of this research is a focus on the improvised, embodied experience of shared ideation, sketching, and creating with physical objects. Unlike existing AR remote collaboration tools, supporting remote physical co-design requires new interfaces that focus on similar, improvised interactions and combine spatial drawing tools, physical object capture and video chat. We propose to study how children design together, create a prototype AR interface, and test it in remote design sessions with children. The outcomes of this work will provide insights on how AR can remove barriers for remote design collaboration, physical play, and learning.