Elevating knowledge discovery through AI integration

Go to the app

My role

Product design, Strategy

TEAM

Asaf (PM)

Johannes (Front-End Engineer)

Vahid (Back-End Engineer)

Salih (ML Engineer)

Nina (User Researcher)

Steffen (UX Writer)

TARGET AUDIENCE

Secondary school students and University students. (Germany, UK and France)

TIMELINE

January 2023 to Ongoing

The Opportunity

BACKGROUND

In November 2022, OpenAI Introduced API for ChatGPT, it offers companies a seamless way to incorporate the powerful capabilities of GPT into their applications.

WHY IS IT IMPORTANT?

Integrating AI and natural language processing capabilities to enhance the learning features will provide early adopters with a competitive edge in the market.

OPPORTUNITY SPACE

Improved Content Discovery & Generation: The API can generate high-quality educational content, including lesson summaries, flashcards, and study guides.

The Approach

Identifying existing problems with the 'Search' feature

Review past research and user feedback surveys to identify potential issues users face with content discovery and generation.

Brainstorming the ways to improve search experience with AI

We conducted a company-wide GPT hackathon to identify groundbreaking ideas for embedding AI within our search features.

Prioritise ideas based on business impact and launch an MVP

Identify the most impactful concepts through impact mapping and craft a strategic roadmap for MVP development.

Existing Problems

Current search functionality: Provides results based on keyword matches only.
A lot of users are churning because they couldn't find the relevant materials after performing the search.
No additional context or explanation in the search results.
Users may struggle with complex content comprehension.
Lack of customization options for better content understanding.

The Goals

Business goals

Improve search/browse activity
Increase 30mins activity on the app (for first-time users).
Increase day 1 retention.
Increase flashcards and note creation.

UX goals

Make content discovery easier for users.
Provide users with more options to customize content for better understanding.
Simplify the flashcards and notes creation for users.

Impact mapping

Utilizing insights from previous user research and ideas generated during the GPT hackathon, we conducted an impact mapping exercise to define the priority order. We have chosen to adopt an MVP approach in order to expedite the rollout of this feature.

MVP

Embedding short AI-generated explanations with search.
Allowing regeneration of explanations and versioning.
Simple customization techniques to modify explanations.
Flashcards and note creation from the AI-generated explanations.

Milestone 1

Dedicated space for the Explain AI feature in the explore page.
Smart topic suggestions based on user's onboarding data.
Keeping the recent browsing history and generated explanations.

Milestone 2

Advanced customization techniques.
Generating mind map from explanations.
Speech-to-text and text-to-speech explorations.

The Solution

Defining the basic architecture for MVP

Hey Team, We need to build an initial architecture for this MVP. Let's share our ideas first and then we can discuss their technical feasibility.

Sounds good! 👍 It would be nice if we could do it as a flow diagram and divide it into different segments. And, what are we planning to call this feature?

Salih (ML Engineer)

Vahid (Back-End Engineer)

Johannes (Front-End Engineer)

Steffen (UX Writer)

Let's call it "Explain AI" ✨

Since this topic was highly technical, I collaborated with the engineers at an early stage. We shared our ideas, discussed their feasibility, and together, we developed an initial architecture. We also named the feature 'Explain AI.'

Formulating query based on user profile & search keyword

To offer users personalized results, we enhance their queries using the profile data collected during onboarding. This approach ensures that users receive more relevant information tailored to their preferences.

Sanitizing the response from ChatGPT

Occasionally, ChatGPT produces results that may not be suitable for students. Hence, an additional layer of filtering is required to differentiate between appropriate and inappropriate content.

Customize explanations for a better understanding

Users perceive information differently, and to facilitate their understanding of complex topics, we incorporated various memory techniques based on previous research conducted, named 'How Our Users Study.'

Generating flashcards using AI

Due to the extended loading time, we decided to restrict the number of flashcards generated to '15' for the MVP.

Validation

Hey Nina, How about we conduct a usability test to validate our concept?

Awesome idea! I've already got 7 users lined up. I'll handle the interviews, and you're more than welcome to join me to observe and help with note-taking.

Nina (User Researcher)

I would love to! Let's do this together!🥳

Building upon the initial architecture, I developed a prototype for user testing. In collaboration with the user researcher, we jointly prepared the interview guide and conducted 7 user interviews to validate the prototype. During these sessions, participants interacted with the prototype on their mobile devices, and we closely observed their activities.

Task 1: Getting an initial understanding of the feature

Task 2: Exploring different customization techniques

Research conducted on Feb 2023

PAIN POINTS

University students have expressed that the depth of the explanations provided is insufficient for their needs.
Concerns about trust in AI-generated content persist, with some skepticism regarding the accuracy of facts generated by AI and whether it meets the required level for examinations.

"As a scientist, I hold the belief that AI lacks a robust peer-review process, which makes me personally hesitant to place complete trust in it."

— Elena

WHAT WORKS WELL

Short explanations prove highly beneficial when users seek a quick definition.
Younger users particularly find memory techniques extremely useful for rapidly grasping concepts.

"Would look at Explain AI explanation for a quick definition. Study notes for more in-depth"

— Farzana

Final solution

Hey Team, Considering what we've learned from the user interviews, we should refine the search query to offer more in-depth explanations tailored specifically for university students.

Absolutely, we can do that. Did you encounter any issues with the design aspect?

Salih (ML Engineer)

Vahid (Back-End Engineer)

Johannes (Front-End Engineer)

No, actually, on the design side, everything seems to be working well. Users are engaging with the feature without any significant problems.

Drawing from insights gained during user interviews, we refined the search query to provide more in-depth explanations tailored for university students. On the design front, there were no critical issues impeding user engagement with the feature.

Search → Explain AI preview → Detailed screen

Different customization techniques

Testing Strategy

We implemented the A/B test by introducing the Explain AI feature to 50% of our user base, with the existing search feature serving as the control group. The entire testing phase spanned 6 weeks, a duration chosen strategically to mitigate the influence of seasonal fluctuations and other external factors that could skew our experimental data. Given the critical importance of this feature to our business metrics, we exercised caution throughout the testing period to minimize the introduction of additional elements or modifications that could contaminate the integrity of our test results.

The Impact

67.4% increase in search activity was detected within one month after releasing the Explain AI feature.

Other significant improvements:

24.2%

Increase in 30mins activity.

We've successfully hooked users with our AI feature, and analyzing their search inputs revealed extended app engagement while exploring various keywords.

38.2%

Increase in day 1 retention.

User engagement skyrocketed after the launch of this feature and a large majority of new users came back to the app on day2 and frequently used Explain AI.

11.1%

Increase in flashcard creation

There was a huge increase in the number of Flashcards created after the launch of Explain AI. It helped users to avoid the friction of creating flashcards manually.

8.2%

Increase in note creation

Same as flashcards, a lot of users saved the AI generated explanations as notes and because of this the note creation metrics also improved.

Features that didn't work quite well..

Memory techniques

Initially, memory techniques were well-received; however, over time, user engagement declined. After the first few weeks, only a few users continued to use it.

Based on the data and interviews, it became evident that among all the customization techniques, 'elaborate' and 'simplify' were the most frequently used. In terms of memory techniques, users primarily relied on 'explaining with real-life examples.

Iterating based on the knowledge gained

Users used 'elaborate' and 'simplify' more frequently, and as a result, making these options more accessible to them became my top priority. According to the data and interview findings, it was evident that there was some friction for users in repeatedly opening the bottom sheet. Therefore, in the new version, I relocated the most frequently used options to the explanation screen, which turned out to be the users' favorite version in user testing.

Next steps

Explain AI as a separate tab

Given that users are more frequently utilizing Explain AI in comparison to the existing search feature, introducing a dedicated tab would likely result in improved engagement.
We are planning to incorporate intelligent topic suggestions based on users' onboarding data within this tab. This feature will serve as a viable alternative to conventional search.

Generating mind map from explanation

Throughout the interviews, the term "mind map" surfaced frequently.
It is a proven method employed by many of our users to memorize complex concepts.
The generation of mind maps from explanations is entirely feasible with AI, but further technical research is required to achieve this.

Conclusion

In summary, our efforts to create and launch the 'Explain AI' MVP have yielded significant achievements alongside notable challenges. Collaborative teamwork resulted in remarkable impacts, including a 67.4% surge in search activity within a month of launch, a substantial 38.2% increase in day 1 retention, and heightened engagement in flashcards and note creation.

However, we encountered setbacks, particularly with the initial adoption of memory techniques, leading to a decline in user engagement. This underscored the importance of adaptability and iterative development, prompting us to prioritize user-favored options, 'elaborate' and 'simplify,' based on valuable feedback.

Our journey underscores the effectiveness of teamwork and continuous improvement. Looking ahead, we are committed to refining 'Explain AI' by introducing it as a separate tab due to its popularity and exploring the potential to generate mind maps from explanations. This project reflects our dedication to enhancing the learning experience for our target audience and leveraging AI to make education more accessible and effective.

Let's Connect!

Feel free to get in touch with me here ↓

manumohanan.de@gmail.com

LinkedIn →

Bento →