Knock_Knock_Opening_Edit.jpg

Knock Knock

Knock Knock

Machine Learning // UIUX // Agriculture

KnockKnock

Concept

Do you remember when your grandma knocked on watermelons to see if they were ripe? The subtle art and skill of being able to pick out fresh produce in a busy market place is a dying art. Exploring sound diagnostics through everyday experience, Knock Knock is a mobile application that uses machine learning algorithms to re-discover this skill for the next generation.

Work completed with Bahareh Saboktakin Rizi, Fay Feng, Hugo Richardson, and WuQing Hipsh.

 
KnockKnockApp.jpg

App Flow

Based on a Knock Knock joke, a simple app flow was developed to ease the user journey. The app includes 4 main components: image detection, sound detection, ripeness evaluation and feedback.

App UX.jpg
FunctionofRipeness.jpg

Image Data

Fruits type are classified by color, shape and size. Ripeness is not binary but a scale. As the correlation between how a fruit looks and its ripeness is quite intuitive and visual, we would create an initial data set of labeled image data to then train the sound machine learning algorithm We validated our idea by training a convolutional neural network model for real time object detection using Tensorflow.

AppleDetection.jpg
 
Final_Presentation_knock_knock_Page_19.jpg

Sound Data

To validate the feasibility of detecting ripeness of watermelons from knocking sounds. We gathered a library of audio files from knocking on objects. The Fast Fourier Transform (FFT) converts knocks of the collected audio files from time to the frequency domain. This show a promising pattern that could be used for further classification.

 
SystemFlow.jpg

Together, three separate machine learning algorithms are proposed. Supervised learning is used to first classify the type of fruit from visual data. Next a regression model is used to determine the level of ripeness. This data is combined with a CNN processing the Fourier transform of fruit knocks to give an overall ripeness prediction. The user feedback allows classification of new data to retrain all the algorithms involved.

Banner.jpg