[orca-list] orca general audio support (sound icons, etc)

[Rui Batista <ruiandrebatista gmail com>]

Hi all,

My name is Rui Batista and I'm from Portugal (some of you may already
know me from this and other lists).
Now, due to an university course I'm taking (my university is Instituto
Superior TÃcnico here in Lisbon) I can work more seriously in orca, in
particular I'm taking the opportunity to develop orca general audio
support and based on this, use sound (sometimes they call it sound icons
and things like that) to improve the user experience in orca.

There are already some screen readers using sounds to provide
information to users, cases are NVDA, JAWS (MSN messenger script for
example) and emacspeak. I'd like to know what use cases of sounds you
like in other screen readers, what do you would like to have implemented
in orca related to sound, what new great ideas you have no one add
implemented yet in other screen reader but you think will be great for
orca?
Here are some ideas I collected for now:

=== Beep on Progress bar Updates ===

 * Description: on progress bar updates, inform the user of the relative
percentage using beeps of various frequencies.
 * Details: One great idea implemented in the nVDA screen reader is the
beep on progress bar updates functionality. When the current application
as a progress bar control and that control changes the value NVDA beeps
informing the user of the completed percentage on the progress bar.
Higher the beep frequency, higher the progress bar value. This is really
useful when copying or downloading files, installing packages, etc.

=== Provide Mouse Localization using Sound ===

 * Description: when the user moves the mouse, provide mouse
localization in screen using sound.
 * Details: The current mouse review feature in orca provides a way to
inform the user of the object under the mouse pointer. Using sound, it
is also possible to inform the location of the object on screen using
pan (stereo) and pitch, pan provides the horizontal coordinate and pitch
the vertical one. For example, if the user moves the mouse to the screen
top left corner, orca plays a high pitch beep on the left speaker, on
the bottom right corner it plays a low frequency beep on the right
speaker. This feature is also present on the NVDA screen reader.

=== Audio cues when Crossing line boundaries ===

 * Description: When in flat review, when navigating a text area, and in
"say all", Orca should provide an option to present an audio cue to the
user when they cross line boundaries.
 * Details: I don't recall if I saw this feature on emacspeak but it
seems really interesting from an usability standpoint
    * Notes: bug #354474 filled on
http://bugzilla.gnome.org/show_bug.cgi?id=354474

=== Report miss-spelled Words using Sound ===

    * Description: use a sound icon to report miss-spelled words instead
of using speech
    * Details: Currently in firefox for example, miss-spelled words are
reported using speech (the synth inform the user "miss-spelled word").
Using a sound icon the user can be informed of the mistake without
interrupting the normal text flow. This must be synchronized with the
speech flow.

=== Sound on bodytyping in pidgin ===

    * Description: When a body is typing in pdigin (or some other IM
client) inform this fact using sound instead of saying "body is typing"
    * Details: This is a feature heavily used by the JAWS screen reader
users and seems good from an usability perspective.
  

=== Audio support configuration ===

 * Description: The user should be able to configure audio support in
orca
 * Details: some users don't like sounds going around, some other pref
ere them to play lower, some must turn the off on some occasions. There
must be a way to turn sounds off, control there overall volume, possibly
change the sound card where they are played, Etc. There must be also a
good default configuration for most users, for example use gnome
settings for Te sound card output.


And there are really many more good ways to use sound, I know you all
must have things to say :)

I created a page to talk about audio support in orca:

HTTP://live.gnome.org/Orca/audio

For now it is just a draft, I'm asking here for contributions so I can
gather a good base and organize the information there.

With your feedback I plan to write the requirements (mor or less
formally), designing  and then start coding.

For the tech people here this will be implemented using the gstreamer
multimedia framework.

With kind regards,

Rui Batista


Rui Batista
E-mail/googletalk: ruiandrebatista (at) gmail (dot) com
MSN/WLM: ruiandrebatista (at) hotmail (dot) com (don't send mail to this
on)
Skype: ruiandrebatista
twitter: http://twitter.com/ragb
weblog: http://outputstream.wordpress.com