Members: #

1. Teamlead – Elena Tesmeeva - @elenatesm, e.tesmeeva@innopolis.university
2. ML – Gleb Pavlov - @venikhl, g.pavlov@innopolis.university
3. ML – Aleksandra Kuzmich - @pferdlexxie, a.kuzmich@innopolis.university
4. 3d + gamedesigner – Nika Lobanova - @weieh, n.lobanova@innopolis.university
5. 3d + techlead – Nikita Rashkin - @Ishikawwwa, n.rashkin@innopolis.university

Value Proposition: #

Identify the Problem: #

Our team initially had several ideas for the summer semester project. We decided to choose the one that seemed to us the most suitable for our technical background and also the most interesting for the team. Our project is to create a game for learning sign language using computer vision and 3d modelling (during the course we plan to create a prototype of the game for the most part, and also at the moment we are counting our strength to understand how much data we can teach in our game). We decided to stop there to comply with all the conditions of the project in this course. Additionally, with further development after the course ends, we will have the opportunity to attract more people to the problem of the lack of visual aids for learning sign language in a game format.

Solution Description: #

As it was mentioned earlier, we have not found similar sign language learning games. So, we decided to try to plunge into this area in order to further create a real game that will teach people the alphabet, basic words and much more in sign language from scratch. During the course of the summer semester, we plan to create a game in which you can learn gestures for the alphabet of the sign language, as well as use them in a game format. We will also use computer vision to make it very simple and easy to read user movements. At the moment, we have settled on the idea of a game in the form of creating food / drinks using gestures (the user can spell out the word he has in mind, and then create a magic drink from several ingredients).

Benefits to Users: #

Our team felt that it would be interesting to learn anything, and even more so from an area where there is quite little material on the Internet, in a game format, which is why the game format was chosen. Also, for implementation, we do not need any equipment other than a laptop with a camera (you can connect a phone camera to control hand movement). One of the problems that may arise is hand recognition if there are several people on the camera or if the person is far enough away from the camera /gestures at an angle inconvenient for the camera.

Differentiation: #

Our main advantage is that there is no need to purchase special devices to control the user’s movement (this may be necessary only with further development of the game). Another advantage is the ability to project the player’s actions on any screen, nothing special is needed. Regarding competitiveness, our team was looking for similar games and did not find anything similar to our idea. We plan to work on the project with American Sign Language. However, if we switch to Russian Sign Language, there will be even less competition due to the lack of available material for its study.

User Impact: #

Our project is designed to increase interest in learning sign language, especially among teenagers and young people. The game makes sign language learning accessible to a wider audience, including those who live in remote areas or do not have the opportunity to attend specialized courses. Successful implementation of the project can attract attention to the educational technology market and stimulate investment in the development of new products and solutions. The recognition and use of sign language in a playful way contributes to its popularization and recognition at a broader level, which can lead to wider use of sign language in various spheres of life.

User Testimonials or Use Cases: #

Use Cases: Interactive learning in schools: Teachers note that thanks to the game format, children remember any information better (in this case, gestures) and show a high interest in learning new material. As a result, students’ proficiency in sign language can increase significantly with minimal difficulty in learning Corporate training: In a large company with several hearing impaired employees, the HR department has implemented a game to teach sign language to all employees. This has helped create a more inclusive work environment where employees can communicate freely with each other. Thanks to the game, the team has become more cohesive, and the level of employee satisfaction has increased significantly.

Lean Questionnaire #

1. What problem or need does your software project address? #

The software project aims to find an exciting and effective way to learn sign language. Traditional methods of learning sign language can be dry and devoid of interactivity, which can lead to reduced motivation and slow learning progress. In addition, there is a shortage of available resources for people who want to learn sign language on their own or in areas where formal education is not available. The goal of this project is to make learning sign language fun, interactive and accessible through a game format that uses computer vision to provide real-time feedback.

2. Who are your target users or customers? #

For the most part, our target audience is teenagers and young people - those who will be interested in the game format. As well as any people interested in sign language.

3. How will you validate and test your assumptions about the project? #

Conduct surveys and interviews with first-time users, including students, teachers, and families, to gather feedback on the effectiveness and engagement of the game. Consult with sign language teachers, linguistics students to ensure the accuracy and pedagogical value of the content

4. What metrics will you use to measure the success of your project? #

Learning Outcomes: Improvement in sign language proficiency among users, measured through pre- and post-tests or assessments. Engagement Levels: Time spent on the game, frequency of use, and user retention rates. User Satisfaction: Feedback scores and reviews from users regarding their learning experience and satisfaction with the game.

5. How do you plan to iterate and pivot if necessary based on user feedback? #

Adopt an agile development methodology that allows for rapid iterations and quick pivots based on user feedback and testing outcomes. Leveraging AI, Open-Source, and Experts. Our team plans to use openal and opensource models to speed up work and find references in the design of the game and 3D models.

Defining the Vision for Your Project #

Overview: #

At the end of these 7 weeks, we will present an mvp of the ALS training model that will recognize basic human gestures (fist, opening palm, etc.), letters of the English alphabet (additions are possible in the future) using computer vision.

Schematic Drawings: #

Could not upoload any images (folder structure error). At the moment, we have created several sketches of our game, and also selected the style of all the necessary elements.

Tech Stack: #

Python, 3ds max, Unity, С#, blender, topogun, 3dcoat, substance painter - we decided to use them because they seemed to us the most suitable to use, and the team members also understand them.