About
Mathventure was a master project, where I voluntarily participated as a Programmer, during my seventh semester at h_da.
Mathventure is a cooperative AR fantasy adventure educational game for mobile devices such as smartphones and tablets. The game was specifically designed to counteract math anxiety by enabling children to associate mathematics with positive emotions and experiences. Through a story and various quests, children are introduced to math problems that they can solve by placing number cards in front of the camera, all while being supported by helpful tips and fun characters.
Project
Role: Programmer
Team Size: 6
Time Frame: 3 Months
Engine: Unity
Introduction
My focus with this game was everything AI and AR related, the other Programmer on the project worked on the UI and story related tasks.
The team consisted of two Programmers, one 3D Artist and Animator, another 3D Artist and two directors who did the game design, planning, management and everything in between that wasn’t covered by the assigned roles, since everyone, apart from the two directors, worked on it voluntarily alongside an internship or a regular semester.
AR
The key features of the game are the AR environment and the visual number recognition. Before the system got what it is now, i experimented with the whole feature set of Unity’s AR package. I tried Image Tracking, Object Tracking, Point Clouds and Plane Detection to get the environment to stick to a fixed point in the real world. Eventhough almost all of those methods worked to fixate the environment, none of them gave us a stable enough result, that also made sure the players would always see the environment the way we intended. As we use a stand for the tablets and the angle of the camera never changes nonetheless, we decided to leave out a real time placement method.
I also tried Image Tracking as a way to recognize the numbers on the cards. As there are only 10 different number cards this would be the fastest way to make number recognition work. But, as with the real time placement, the results were not stable enough to ensure that players would not get frustrated with faulty recognition. I tried many different designs for the number cards to make them as unique an recognizable as possible, but the results did not improve enough. Which meant the only possible solution would be AI Image Recognition.
AI - Image and Object Recognition
As with the AR Features I tried a variety of different methods to make mobile on device inference work with Unity. There are a lot of paid services that can make it possible, but we did not have any budget. Nonetheless i experimented with a test version of Open CV, to try out my models and test if and with what constraints on device inference would be possible. I ended up writing my own script that uses Unitys own Inference Engine package.
For the Number Recognition I trained my own Yolo Model using a number plate dataset from Roboflow. In the beginning of the project a dice detection quest was planned, where I also tried out different datasets and trained Yolo models to get them to recognize the different dice faces. Both models worked great, but the dice quest got cut, due to scope reasons.
What I learned
Besides learning a lot about AR, training my own AI models and implementing them in Unity, I also designed a clearly laid out Quest System for one of the directors that did not have any programming experience. That way she could create new Quests and all the necessary inputs without needing any help. That teached me a lot about how serializable objects can be customized inside of Unity. I also implemented the help mechanic, that tries to find out what probably went wrong in the calculation steps by looking at the result and recognizing common mistakes.