Right To Know (MLH Winner - Social Impact)
Major League Hackathon (24 hrs), NCSU 2024 : Winner - Social Impact
The Situation:
- When you are stopped by a Law Enforcement agent, do you know your rights?
- If you are asked to do something that is against your rights, would you know?
How did we solve it?
When the user opens our application and taps the record button and permits access to location and audio,
- It streams audio from the fronend to the backend as chunks of audio
- The audio is converted from speech to text
- The text is then sent to Google Gemini, along with the user’s location information
- We prompt engineered Gemini to output the rights the person holds in that area according to the conversation upto that point
- It also provides the sources, which are government/univeristy websites
- The next incoming audio chunks are processed and the cumulative converation is passed on to Gemini which further informs your rights with more context
- The process repeats until the recording is stopped
What did we achieve?
- An informed person, who knows their rights
- This solution can be extended to work for different purposes, where recording audio is allowed
What more can we do?
- Read out the rights to the user during the conversation, instead of prompting the user
- Save a copy of the recording to the cloud, accounting for possible future circumstances
Technologies:
- Google AI Services: Gemini (Vertex AI)
- Backend: Python (Flask, SpeechRecognition, socketio, pydub)
- Frontend: Javascript, HTML, CSS
The repository can be found here.
Team:
- Abhimanyu Bellam, Arun Parthsarathy, Pradhyumna Khawas, Swara Gaurkar
