Covered Face Alerts & Precious Baby Moments

The Requirement:

Hubble Connected is a baby monitoring camera manufacturer with a mobile app to access camera and offers a lot of features. They wanted to add AI features into their services to potentially reduce the risk of Sudden Infant Death Syndrome by detecting when a baby’s face is covered.

Parallely, they wanted to automatically capture beautiful moments of the user’s baby and send it to the user. Well, who doesn’t love to see their baby’s pictures when they are left unattended?

How did we solve it? (contributions in paranthesis)

• (30%) Motion videos from the camera are stored in AWS S3 buckets. We implemented a server client setup that communites with eachother via AWS SQS messages. The following pipeline is triggered whenever motion videos are produced
• (100%) Set up a computer vision annotation tool (CVAT) on AWS for remote access to annotate images by annotators
• (90%) Created a pipeline to extract key frames from the motion videos and used You Only Look Once based object detector to detect the baby and identify their expressions among: Happy, Eyes-closed (sleep also), Sad (cry, sulk), Normal
• (80%) Also detected covered face, baby boundary, and roll over using the same object detector.
• (100%) Executed multiple requests using multi-processing and multi-threading on EC2 instances
• (100%) Sent back the processed outputs back to server which updates the S3 buckets
• (100%) Facilitated docker container option to run this as well

What did we achieve?

• Customer Satisfaction
• Increased subscriptions
• Baby safety

Technologies:

• AWS Services: EC2, SQS
• Programming Languages: Python
• Data Manipulation: OpenCV
• ML Models: Pytorch
• MLOps: Customer/Management Feedback based model tuning and redeployment

Contribution for future work:

• Cry and sleep detection from Object detector for insights, that can be coupled with audio for better performance.

Team: Abhimanyu Bellam, Ritesh Mishra (Manager), Gaurav Kumar Sharma, Rahul Kumar

Part of Hubble AI Features, found here.