PDA

View Full Version : Summary: Trust and "Black Box" Technology



Drew Smith, Phd
01-13-2005, 09:23 AM
Colleagues,

Back in October 2004, I initiated a discussion on the Biomechanics Symposium
titled, 'Trust and "Black Box" Technology'. The main point I raised was
that in earlier times, researchers often had to write their own software
code and even build the electronics and thus had an intimate understanding
of the strengths and weaknesses of the tools they employed. Over the years
as equipment and software for biomechanics and movement science research has
become more sophisticated, the inner workings of these valuable tools can be
somewhat mysterious to the end user. Yet the results are often taken at
face value, trusted almost implicitly. I was curious whether other people
had any concerns or experiences along these lines.

This initial posting generated quite a few replies such as the following
(from Mitch Maltenfort):

"My own preference is for Matlab code where I can read the M-file or for GPL
code where I know the "black box" is open for correction. I was burned once
and I haven't forgotten the experience. I was working on C code which
included an FFT routine and I thought I should check my FFT routine against
Matlab's own. After a week when I couldn't figure out *why* I couldn't make
my fft code match Matlab's result, I took my code to a colleague for him to
look at. He asked for the data I was using, ran the FFT on Matlab on his
machine, and it agreed with my code's output! It seems Matlab had released a
version with a buggy FFT routine. My colleague knew about it and had the
patch. Ever since, my policy is that if I can't see the code, I don't use
it."

Another comment from Enrico said, "This is a very interesting argument. Too
often "black box" systems are also used in biomechanics without checking the
validity of their measures, especially by students. "Do I consider
uncertainty in processing raw data?" "There are elements in my measure-chain
that may affect my results?" These are two important points that could
influence the reliability of raw data collection and consequently the
elaboration of results. It may be of some help an approach more "aware"
about what one is measuring and about the instruments used in data
collection."

I think the most interesting comments were made by Andrew Dainis, who said,
"As the originator of "black box" software used routinely in biomechanics
motion measurement systems, I would like to add my comments to this
discussion. I do not think that we can escape the tendency for measurement
and analysis systems to become more sophisticated and intelligent. Both
characteristics imply increasing complexity, the comprehension of which can
quickly overwhelm the background and resources of most users...Rather than
spurn the benefits of "black box" tools that are increasingly becoming
available in biomechanics and all other disciplines, the emphasis should be
on evaluation, and ultimately acceptance (or rejection) of the tools'
capabilities. Unfortunately, attempts at evaluation (especially by
independent parties) of tools used in biomechanics are rare, due largely to
their disparate nature and small numbers in use. Also, there are at present
few standards for performing such evaluations. In my area of expertise,
camera system calibration and marker tracking for 3D motion measurements, I
see a very strong need for establishing testing standards, and the task
should be quite straight forwards. Perhaps some of the readership might be
interested in moving in this direction..."

"[The Biomechanics Symposium] might be a good place to initiate something. I
have been interested in the matter for many years, and certainly would be
willing to contribute my time and expertise. Many people who purchase
expensive motion measurement systems have limited experience and knowledge
of the intricacies of getting data from the video streams to 3D trajectories
which form the bases of all analyses (that is a big "black box"). And of
course, the application software is another equally important area. In my
opinion the systems manufacturers have typically not been forthcoming with
performance measures for their systems because the consumer body has not
pressed them for it. In fact, quite often the information is considered to
be "proprietary". The application of standard testing and evaluation of
systems would go a considerable way towards making manufacturers more open
in this matter, and base competition more on technical merit rather than
hype and salesmanship. I have recently been shopping for a digital still
camera, and have been impressed by the quantity of technical and evaluation
data available for just about any camera in current production. At present
there is nothing remotely like this available for 3D measurement systems,
which is a shame..."

Andrew went on to summarise:

"To summarize my position for establishing testing standards:
1) A 3D motion measurement system is a tool.
2) In order to get the best out of a tool we must understand its
capabilities, limitations, and how it functions.
3) Presently, little information is available for just about any system.
4) System testing and evaluation based on standard protocols would provide
very useful information to users, and foster more competition among systems'
manufacturers."

Although the discussion continued for a few more postings, the discussion
stalled. I felt that what was lacking was more input, especially in reply
to the Andrew's comments about the need for standardised testing and his
personal interest in contributing. Perhaps there are others who are
interested as well. I would especially like to hear from manufacturers
and/or programmers. I also wonder if this may be something that the ISB may
have an interest in supporting?

Please feel free to either reply to this email and (with your permission) I
will post replies at the URL listed below. Or easier still, feel free to
add your comments to the discussion yourself by directing your browser to
the URL.

[For the complete discussion, please see Biomechanics Symposium :: View
Forum - Open Discussions =>
http://www.movement-analysis.com/biomechanics/viewforum.php?f=34]

Cheers,

Drew Smith, PhD

Department of Sport & Exercise Science
http://www2.auckland.ac.nz/tmk/ses/
University of Auckland, Tamaki Campus
Private Bag 92019, Auckland, New Zealand
Office: 09 373-7599 x86849 [Int'l: +64 9 373-7599 x86849]
Fax: 09 373-7043 [Int'l: +64 9 373-7043]
Mobile: 021 264-9790 [Int'l: +64 21 264-9790]
************************************************** **********************
This message may contain information that is confidential and/or protected
by law. If the reader of this message is not the intended recipient, you are
hereby notified that any dissemination, distribution, copying or
communication of this message is strictly prohibited. If you have received
this communication in error, please contact the sender immediately and
delete the message. Please note that although we will take all commercially
reasonable efforts to prevent viruses from being transmitted from our
systems, it is the responsibility of the recipient to check for and prevent
adverse action by viruses on its own systems.
************************************************** **********************




-----------------------------------------------------------------
To unsubscribe send SIGNOFF BIOMCH-L to LISTSERV@nic.surfnet.nl
For information and archives: http://isb.ri.ccf.org/biomch-l
-----------------------------------------------------------------