THE
MAIN MENU
Contains three main prototype under development –
1. Speech recognition & Image Annotation-> SPEAK & RECOGNIZE
2. Image and its interpretation -> TALKING ICON
3. Understanding real world objects -> OBJECT RECOGNITION
SPEAK
& RECOGNIZE
Two main options-
1. Tag Images – A normal person preferably one that assist older people can use this option to tag images with keywords and make other configuration. These are only one time configuration.
2. Recognize – Older people (actual user) of the system would use the part of the system. It’s as easy as saying a word and its related image appearing on the screen.
After selected tag image from aforementioned screen user would land on this screen where they have the option to select which images to tag. Remember that a user can tag more than one keyword for an image.
User can say the word or type it and it would appear in the drop down box below (Saved). If user wants to delete any word from the drop down box they can do that simply by selecting particular keyword and hitting the X mark.
This would be the screen after a word spoken by user would be recognized. In above case user had spoken church and its image appeared on the screen.
Users here have the option to create sentence too by selecting noun, adjective or verb. On selecting a particular word it would appear in speak textbox and to actually speak the word just hit the SPEAK button.
ICON TO SPEECH
Overview
Prototype consists of Icon-to-Speech
conversion of Human Activities. Various activities that a person does are
represented using Image icons. When an Icon is tapped, voice saying the activity
associated with that icon is generated. This uses the Text-to-Speech Engine
available in Android. This application would help people with speaking
disabilities to use their phone speak for them.
Detailed
Description
There are four main categories of activities in the
application
I.
Greetings
II.
Daily
Living Activities
III.
Emergency
IV.
Miscellaneous
Activities
MAIN CATEGORIES
Each category has sub-categories of icons.
Subcategories of
I) Greetings:
1) Hi
2) Bye
3) Thank you
4) Sorry
ICONS IN GREETINGS
CATEGORY
II) Daily Living Activities
1) Bed-Time activities
2) Eat
3) Walk
4) Restroom activities
5) Changing clothes
6) Transferring
SUB-CATEGORIES OF
DAILY LIVING ACTIVITIES
Taping on the Sleep icon takes you to a
screen where user can select from the following icons
a) Place – Bed, Sofa etc (Request for
blanket, pillow are also included)
b) Time to Sleep
c) Time to Wake up
PLACE TO SLEEP
SAY TIME TO SLEEP OR
WAKE UP
When user selects the “Sleep Time”
or “Wake up” icon he gets an option to set the time so that he can tell at what
time he wants to sleep or wake up.
II) Emergency
1) Doctor
2) Fire
3) Ambulance
When the User taps one or more icons,
the voice will be a meaningful sentence with the activities selected.
Eg: When
the user selects “Fire” and”Ambulance” the voice will be “There is a fire
emergency, can
somebody call the fire dept and ambulance”
EMERGENCY ICONS
IV) Miscellaneous
1) ordering a Burger
2) Banking
3) Church
4) Pharmacy
for ordering a burger, user
will be provided with the following options
a) Place – McDonald, Wendy’s, Burger
King
b) Kind of burger- Menu list will be
displayed based on the Place Selected
c) Quantity
This will be useful when the User
wants to Order a burger. Depending on the kind of burger and the quantity
selected, voice will be generated saying the order.
PLACE SELECTION SCREEN
BURGER SELECTION
SAY THE COMPLETE ORDER
Banking
activity can be useful when the user is at a bank. Basic banking activities are
included.
DEPOSIT MONEY AT THE BANK
When
the user is at a Pharmacy, he can tap the pharmacy icon and select a medicine
based on the illness. This would tell the pharmacist what medicine he wants.
SELECT MEDICINE BASED ON ILLNESS
TELL THE PHARMACIST WHAT MEDICINE
HE NEEDS
Future Enhancements
1) A feature can be added which gives the user
flexibility to add his own gesture by drawing it on the mobile screen and
associating it with some text which he wants to be converted to voice.
For example: The user draws the symbol “T”
on the screen and associates it with the text “Thank you” by typing it. The
next time he opens the application and draws “T”, voice saying “Thank you”
should be generated.
2) Categorization can be made better
using “Wordnet”.
3) Improvement to the User Interface
can be done.
OBJECT RECOGNITION
Overview
Prototype consists of image detection and image recognition. Face
detection consists of two parts: Static Detection and Dynamic Detection. User
can take image of his choice with the android phone using static detection
mode, in dynamic detection phone will show a square once it detects any face in
the preview. On clicking face recognition button application match the image
detected with its dataset and user will get result if image matched or not via
speech. This can assist older people or low vision people who problem in
recognizing the people
Detailed Description
There are three main categories of activities in the application
I. Static Face Detection
II. Dynamic Face Detection
III. Face Recognition
MAIN CATEGORIES
COMMON STEPS:
ENTER IP OF LOCAL SERVER
User should enter local IP address of the machine to connect to it and click on button “Click OK to proceed”.
CAMERA STARTS
Now user will see that camera has started.
SELECTION OF MODE
Click button “Static Detection” for selecting static face detection & if you want to go in dynamic face detection mode click on button named as “Dynamic Detection”
I)
Static Face Detection:
When the user taps “Static Detection button” after focusing on the image preview in camera he will see a bulls-eye mark in red confirming that static face has been detected & image has been captured. (See image below)
STATIC FACE DETECTION
II)
Dynamic
Face Detection:
If user taps “Dynamic Detection button” after focusing on the image preview in camera he will see a rectangle box in middle of the screen which will be moving as the face preview in camera changes position. (See image below)
DYNAMIC FACE DETECTION
III) Image Recognition
NEW OPTIONS
After Dynamic Detection by clicking on “menu” button user will see new buttons, here for Face Recognition user needs to click on button named “Recognize” to activate Image Recognition.
FACE RECOGNITION ALGORITHIM WORKING
Now after matching the faces (depends on the dataset & the current image taken) user will hear the sound whether image has been recognized or not.
.