Week #5 #

Feedback #

Testing session with TA #

Suggestions:

• Add visual preview
• Think about the possibilities of moving to a specific text on the page

Testing session inside the team #

• Fix the format of md files for the correct text preview of search answer cards
• Fix the applicability of filters to search

Testing session test sessions with potential users #

Testing sessions were conducted in person, when users used our product for their needs and commented out loud on their intentions, actions and difficulties.

Session #1:

Search

• The user found the page a bit empty when first visiting it
• The user was confused by the search result for an empty string - unclear state of the website

Problem queries in search

• A search containing “108” or “one-zero-eight” does not return the desired club
• User queries that are not covered by our resources:
  • Coffee
  • Cafe
  • any information about course instructors and schedule
  • List of electives

Ask

• Consistency of the query and response language of LLM does not work
• The user expressed the opinion that a good feature would be to implement clarifying questions from the user
• The user tried to ask LLM what resources it uses

Problem queries in ask

• The user tried to find information about the heads of clubs, not all names gave a relevant answer

Session #2:

Search

• Our resources do not cover the query “schedule”
• The filter does not work if you remove all types of answers from it (pdf, website)

By design search

• Remove the dash in the “Web-site” text in the filters
• The text in the “Residents” filters is not self-explanatory - it is not clear what kind of resource this is
• The user said that he wants a longer description of the cards, otherwise it does not make sense
• During loading, he suggested making a blinking skeleton or a moving loading icon at least to make the state clear

Ask

• Error 502 was thrown for the request “calculate my scholarship for GPA 4.5”
• For the request “Please calculate my scholarship for GPA 4.0” LLM could not calculate a specific value because the minimum and maximum thresholds are unknown

Session #3:

Search

• For the request “campus” the website campuslife is returned, although information on dorms was expected

Ask

• For requests unrelated to Innopolis like “do you understand Russian?” AI gives an answer but puts a random resource of ours as a reference

Session #4:

Search

• For a request for which we cannot provide an answer, irrelevant answers are given
• Request not covered by our resources - “How to see my GPA”

Design

• On a mobile device, it is unclear what the search button is responsible for
• The filtering window needs improvement - the user noted that the window overlapping the page is not a typical look for this element

Ask

• If you ask something in binary code, it throws an error
• For the request “What is my name?” LLM answers with the names of professors
• A request not fully covered by our resources - “How to get innopoint”

Session #5:

Ask

• Is the Ashkazan canteen open? Answers that yes, although this is not the case
• Is it possible to bring a PC to the dorm? LLM answers that yes, but does not recommend registering it, which would be an important part of the answer

Analyze #

During the discussion, it was determined that we should focus on the UX/UI of our web page, the completeness of the data and its presentation in the search pipeline. Namely, we prioritized our tasks from the most important to the least.

• Add a visual preview of search sites (see closed issue)
• Fix a text preview in answer cards (see issue and commits #1, #2, #3)
• Fix LLM for answers in the same language as the request (see commits #1 and #2)
• Expand the pool of resources used by the search (see closed issue)
• Give the user answers and resources only if they are relevant to him (see issue)
• Make minor adjustments to the frontend (labels, filter fixes) (see commit)

Iteration & Refinement #

Implemented features based on feedback #

As you can see from the task artifacts, our team has closed/made progress on almost all tasks. In addition, our team has been working on general advancement of the project functionality, see the “Project development” section.

Performance & Stability #

For our project, it is important to provide answers to users’ search queries relatively quickly. Therefore, we mainly measure our performance in two metrics:

• Response time for a search query. Most search queries do not exceed 1 sec, the average time to receive a response from the backend = 0.98 sec.
• LLM response time. At the moment, almost all queries are processed in less than 10 seconds, while the average time for a request to LLM = 4.25 sec.

We are satisfied with the current state of the metrics and our task is to monitor them so that the results do not worsen with an increase in the number of sources and the addition of new features.

Documentation #

In our project we have documentation in the form of:

• README - a typical way of documentation for team members and potential contributors, includes:
  • Project description
  • Technology stack
  • Project launch instructions
  • Contribution guide
• Swagger - uses openapi scheme to facilitate the work of frontenders (automate their routine), synchronize developers
• Code comments - will allow developers who are not currently involved in the development to quickly understand the project if they wish.

ML Model Refinement #

This week we checked user testing, as a result of which we found problems with:

Search:

• Queries containing “one-zero-eight” do not give the expected results (there is no club among the answers)
• Not all queries for university room numbers give the expected results
• When there are no relevant answers to a question, the result is still given

Ask:

• The language of the query and the answer does not match (fixed)
• When the answer does not use our knowledge base (the assistant answers that there is no data for the answer), our resources are still indicated in the list of sources

Some of the tasks were fixed or we started the process, some are left for future weeks. At the moment, we also have a dataset for testing the operation of our models and can use it if desired. In the future, we are considering the possibility of automating model testing.

Project development #

Testing #

Vladimir was writing integration and ml tests to ensure ml service works correctly and interacts with backend service properly (see commits: #1, #2, #3, #4, #5, #6). More precisely, Vladimir wrote the code to preparse test data from sources and save it locally to use it in test mongo database for integration tests. Then implemented (partially , main part) core code(fixtures, test case) for integration tests. Spent many time to overcame problems with wrong fastapi initialisation for tests. Since ml database can be used locally, in the first prototype it processed again and again for each test running. However in reality it takes incredibly much time to index all documents from test mongodb, so it was decided that we should write pipeline to prepare test data - parse it from sources, load in temporary mongodb, then index all documents used powerful hardware and save this data for lancedb locally, so there will be fast initialisation in integration tests.

Anna conducted testing sessions with TA, within the team, with five potential users (see feedback section). From the feedback received, Anna identified relevant notes and formed issues, prioritized them and assigned them to team members (see Kanban board, search project).

Backend #

To correctly display the preview text of search results, it was necessary to provide MD files that the frontend could process. To do this, Anna adjusted the parser hotel, campuslife, eduwiki (see commits #1, #2). For a more meaningful search, the parser of the residents’ site was refactored. To improve the visual presentation, the map processing pipeline was refactored. During site testing, we caught the 502 error several times, so Anna debugged it (see #1, #2) and refactored the code for a more pleasant presentation in swagger.

Azaliia implemented ask statistics saving to MongoDB (see PR), enabling the tracking of user queries to support analytics and in future response optimization. Azaliia also implemented static sources processing (see PR), ensuring efficient handling of predefined data ( InNoHassle and My University).

ML #

Sofia implemented:

• One of the modules for act - booking module for “music-room” (in progress, not fully correctly implemented): added Pydantic schemas, implemented ActRepository and FastAPI routes (see branch)
• Use-cases for act: documented core scenarios (see document)
• Map-based search improvements (in progress): attempted to align numerical room identifiers with database entries by full-text search, adjusting parsing and query logic, but the issue remains unresolved (see branch)
• Bilingual Q/A support: enhanced language layer so that, even if database resources exist only in Russian, users can query in English and receive English responses and the same way on Russian (see commits #1 and #2)

Frontend #

Aliia made:

• First PR: Connected a new endpoint /ask/ and implemented fetchAsk with the POST method. Added the download and display of the AI response on the AskPage page: the “Thinking…” status is displayed while waiting for a response and the result is displayed with source. Refactored the AskPage and AskResult to match the response structure from the API. Added the TruncatableMarkdown component for the correct display of markdown content
• Second PR: Added a new IframePreviewCard component to display website content inside an iframe. Updated the logic of displaying previews on SearchPage: depending on the type of source, either PreviewCard or IframePreviewCard is now rendered. Improved the TruncatableMarkdown component: added logic for more correct text cropping, depending on the type of source. Added the display of several unique sources on the Ask
• Other minor changes (see #1, #2, #3, #4).

Anna corrected the request sent to the backend (if you do not pass the source parameters to the search query, then a search will be performed on all sources, see the commit) and enabled new sources for display (see the commit).

Overall progress #

Check our progress here.

The most notable points of our progress this week are adding a visual preview for the search functionality, adding new sources (InNoHassle, My University). We also added a new ask section where you can get answers from the AI ​​assistant.

Weekly commitments #

Individual contribution of each participant #

Plan for Next Week #

• Rewrite the project to uv (faster than poetry, better resolves dependencies, more convenient CI/CD)
• Continue writing the act functionality
• Redesign the page
• Correctly display static resources on the frontend (divide them into cards of different types)
• Provide search results only if they are relevant
• Fix the search and ask for problematic queries

Confirmation of the code’s operability #

!!! The working code in the backend repository is in the main branch, and in the frontend repository in the capstone branch (difficulties due to automatic deployment)

• In working condition.
• Run via docker-compose (or another alternative described in the README.md).