Tweet
Hello all,
I would like to apologize for the tardiness of this summary and I'd like to
thank all of you who took the time to help me. The information was very
helpful.
First, here is the message that I had sent:
> I'm a maters student who is working in the area of gait. I have
> access to data from a multi-camera vision system. I am looking
> into the problem of missing markers and the ability to track them and
> predict their path while they are out of view of the vision systems. I
> have come across some interesting techniques.
>
> Unfortunately, I haven't been able to find out how big this missing
> marker problem really is. I understand that as long as the markers
> placed in the anatomical sites are viewed by any two cameras the 3D
> positions of these markers can be calculated. However, in case of
> pathalogical gait, the use of canes, walkers, walking pattern, etc., can
> cause the marker to be obstructed from view of the cameras. I haven't
> been able to find any information to show if this really is a problem
> other than the words of a few people who work in gait laboratories.
>
> In many of the vision systems, when the vision system loses sight of the
> markers the tracks of the marker paths become disjointed. Generally the
> user has had to connect the disjointed tracks manually. Is there softwar=
e
> that makes these connections automatically? If so how do they do it?
>
> Finally, in the prediction of motion, I have narrowed my search to using
> motion models of those limbs. Are there better techniques, esp. for
> pathological gait?
>
MISSING MARKERS:
Pertaining to the problem of missing markers, it seems that almost
everyone agreed that it was a problem (to varying degrees of
seriousness). The best solution, as one person said, is to make sure the
problem does not occur and this could be done by: increasing the number=20
of cameras viewing the tracking region; using non-camera methods of
tracking such as magnetic tracking[1]; carefully choosing a marker set
that would stay in view despite obstructions, rotations, etc,; and using
special assistive devices that minimize intereference with the=20
cameras [2].
PREDICTING MARKERS AND CONNECTING DISJOINTED TRACKS:
As for actually filling in the gaps using software, I have broken this
down into two separate sections: the first one deals with comments from
individuals and the second would be the companies that specialize in=20
motion analysis' hardware and software such as VICON, Ariel Dyanmics,
Inc., QUALYSIS, Peak Performance Technologies, and Elite (BTS).
The techniques are listed in point form and the reference number attached
to it is based on the order of the email responses which are concatenated=
=20
to the end of this summary. The following is a list of techniques and
algorithms:
- Join missing gaps using spline functions (cubic or quintic) [1]. It
may not be very accurate over large gaps. Cheng cautions that some
kind of human involvment is usually necessary when using visual systems.
=20
- Using a proxy marker to provide an offset-recording [2,3] of the marker
that consistently disappears from view. This proxy marker can then be=20
used to infer the position of the missing marker.=20
- For short isolated gaps, using a straight line or polynomial algorithm
works well [3]
- Extrapolate along a straight line between two markers [3]. For example,
if you have trouble seeing an ankle marker, put two markers on the lower=
=20
leg, so that the ankle is further along the straight line drawn between
the twp markers.
- For constructing analytical models for analysis [3] some papers to=20
consult are:
Quantitative analysis of human movement synergies: constructive pattern
analysis for gait, C.D. Mah, M. Hulliger, R.G. Lee and I.O. Callaghan,
Journal of Motor Behaviour, 26, 83-102, 1994.
Quantitative analysis techniques for human movement: finding multivriate
patterns in large data sets, in M. Whitten and D.J. Vincent, eds.=20
Computational Medicine, Public Health, and Biotechnology: Building a man
in the machine, Part II, pp. 1056-1069. World Scientific Press, 1996
- Curve fitting alogrithms are fine for small gap sizes (10 frames) [4].
- If the general forms of a specific movement are known, then a tracking
algorithm can be written for a this specific problem [5].
- 3 markers uniquely define a segment's location and orientation in space.
If a 4th marker is used, the redundancy could be used to calculate a miss=
ing
marker. Some pros and cons of this technique are discussed in email [6].
- If the distances between three markers on a segment are known and the
location of two of the marker are known, the location of the third marker
can be constrained to a small area. This might be used in conjunction=20
with a interpolative procedure to give a better estimate of location [6].
- If the data is cyclical, it may be possible to "guess" where a reasonable
location of the marker might be based on where it was one period before
or after the instant [6]. However, in gait analysis cases where an
assistive device obstructs the camera view they tend to happen in the=20
same potion of each stride [8].
- For path matching, sometimes paths can be easily identified by some uniqu=
e
characteristic of their location [6] such as a marker on the upper part o=
f
the body would have greater values in the height coordinates.
- Using direction and velocity to help identify possible path matches [6].=
=20
This will work well if the gaps are short and there are no sudden reversa=
ls
in direction of the marker.
- Again for path matching, using a marker path reconstruction technique to=
=20
reconstruct a path and match it to an observed path [6].
- Other useful papers[9]:=20
Ball, K.A. and Pierrynowski, M.R. (1991), WalkTrak: Automated analysis=20
of 3D kinematic data from video systems. Proceedings of International
Symposium on 3D Analysis of Human Movement, First Meeting, 28-31 July,
Montreal, Quebec, Canada, 6-9.
Papers on rigid body kinematics:
Ball, K.A. and Pierrynowski, M.R. (1995), Estimation of siz degree of=20
freedom rigid body segment motion from two dimensional data, Human=20
Movement Science, 14, 139-154
Ball, K.A. and Pierrynowski, M.R. (1996), Classification of errors in
locating a rigid body, Journal of Biomechanics, Any day now.
The following is a list of the companies and a brief comment on their=20
software.
Qualysis (Sweden & USA) [10] who manufactures the MacReflex Motion Measurem=
ent
Systems for the Macintosh or PC computers gave me information on how they
track the markers and what they do for prediction of the missing markers.
In handling the tracking of markers, the Qualysis software extrapolation
of the marker movements are carried out to approximate the next position
of the marker. This extrapolation helps the tracker to look in the
right place when looking for the next position of the marker. In
situations where the adding of extra cameras will not help solve the
marker out-of-view-of-cameras problems, the operator can run the tracking
manually or run the tracking separately for different segments of motion
stored in a file. Since the tracking can be run both forwards and
backwards in the file, a point with missing markers can be approached and
tracked from both directions. A user of this software (E-mail Corresponden=
ce,
Zhang, 1996) commented that when the discontinuities were too large the
filling in of gaps did not work well.
Peak Performance Technologies, Inc. (Englewood, CO, USA), says that their
latest software [11] allows the user to connect the gaps manually by
connecting points, semi-automatically by allowing the user to run different
algorithms, or automatically by allowing the user to set several tracking
parameters. According to the engineer, while much has been done using
brute force mathematical extrapolation and interpolation, along with more
cameras, to accurately fill in missing data, there is still plenty of
room for improvement. Especially in the areas of using fewer cameras
with software or hardware that is smart enough to fill in bigger and
bigger gaps in the data.
The ELITE system by Bioengineering Technology & Systems (Milano, Italy)
primary products include: ELITEplus (Three-dimensional motion package),
ELICLINIC (Clinical gait analysis), TELEMG (Dynamic electromyography with
no limitations on subject motion), GAITemg (Identification of muscle
activity in walking), PcVect (Analysis of ground reaction forces), etc.
The Elite software provides interpolation to handle relatively short gaps.
According to the founder and Senior Vice President of Motion Analysis
Corporation, [12] pathological gait motion is difficult to track. They have
software, the Track Mender, which looks at the continuity of 3D path segmen=
ts
and automatically joins them if they are no more than a few frames apart.=
=20
Another technique used by their software is to allow the user to manually
spline across the gaps.
VICON Motion Systems (Oxford, England and Santa Fe Springs, CA, USA)=20
Bodybuilder software [13] can automatically fill gaps, interpolate=20
and smooth data. It also allows for the user to manually edit and model
the data. The software uses kinematic equations and/or some physical=20
modelling of the body to connect disjointed marker tracks.
Ariel Dynamics,Inc. (Englewood, CA, USA) software [14] does provide
some gap filling algorithms. They use some complex search algorithms and
linear extrapolation to predict marker motion over the missing gaps. The
software also does some smoothing where interpolation is done to fill in=20
the smaller gaps. Their software does allow the user to estimate the marke=
r
locations based on their observations of the whole range.
I will now list the emails that I did receive. The first emails are from
indivduals from various labs and departments around the world and the
later ones are from the companies. I preface each email with a short
summary of what they contain (for the shorter emails this will seem
redundant) I hope that this summary has been useful and not long winded
or boring.
***************** Recieved emails *************************************
[1] Email from Cheng Cao:
He mentions that a small gap can be filled using spline functions. He
says the the larger the gap, the less accurate the 'filled-in' data would
be and states that some kind of human involvement is needed when using
visual systems.
>Date: Wed, 28 Aug 1996 14:12:51 -0230
>From: Cheng Cao
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hi,
>
>The missing makrer thing is a big problem most of time. You can always
>join a missing gap or gaps usin spline functions (cubic or quintic)
>"automatically". But, the danger is how real or accurate your
>data would be without considering the size of the gap and when the gap
>appears. Thus, some kinds of human involvement are mostly needed when
>you using visual systems. But, if you use a magnetic system, there is no
>such problem, of course, there are other problems.
>
>Cheng Cao. Department of Mechanical Engineering and Applied Mechanics
> University of Michigan, Ann Arbor. -*-
--------------------------------------------------------------------------
[2] Email from Dr. Manuel Hulliger:
In this email Dr. Hulliger comments on the the problems with mssing
markers and briefly comments on one technique being used by his department.
>Date: Wed, 28 Aug 1996 14:44:17 -0230
>From: Manuel Hulliger
>To: joshua@engr.mun.ca, mhullige@acs.UCalgary.CA
>Subject: Marker obstruction - gait anlaysis
>
>Dear Joshua,
>
>Chris Mah from our gait lab will send you a more comprehensive reply.
>There is therefore no need to include this one in your summary. At
>any rate, when you compose your summary, you would do the community
>a service by submitting a genuine digest, summarizing main points
>thematically and regrouping the various elements from different replies
>accordingly, rather than merely appending poorly correlated and hetero-
>genous replies in incoming order, as we see so often in the debates
>going on in this forum.
>
>Marker obstruction often is a genuine problem, even in normal gait:
>with a two camera system arm movements obscure hip markers quite
>regularly and predictably. The Elite software we are using provides
>interpolation software to handle relatively short gaps. In addition,
>we are using a "tail" attached to the subject's lower back, to get
>an offset-recording of hip marker coordinates (in crude approximation).
>You may want to consult Mah et al., Journal of Motor Behaviour 26,
>83-102, 1994.
>
>With multi-camera systems (Vicon is going up to 7, I understand) this
>problem appears to be reduced in magnitude.
>
>The above problems are exacerbated in the conditions you list. Canes
>and walkers indeed cause additional loss of marker information. For
>our work with cerebral palsy patients we have constructed a special
>walker to minimize such interference. For patients using canes, we use
>parallel bars to provide support (on the side remote to the cameras).
>
>I hope this helps a bit. Chris Mah will reply more extensively.
Yes this was very useful.
>
>You might want to visit the lab to get some first hand experience.
That would be very interesting. I hope to do this sometime in the future.
>
>Yours,
>Manuel
>
>Manuel Hulliger, D.Phil.
>Department of Clinical Neurosciences
>Faculty of Medicine
>University of Calgary
>3330 Hospital Drive N.W.
>Calgary, Alberta, Canada T2n 4N1
>Phone: 403-220-6216 [Ellen Wong Phone: 403-220-8389]
>Fax: 403-283-8770 [Secretary Fax: 403-283-8731]
>E-mail:manuel@cns.ucalgary.ca or mhullige@acs.ucalgary.ca
>www: http://www.cns.ucalgary.ca/people/hulliger.html
--------------------------------------------------------------------------
[3] Email from Chris Mah:
This is an informative email about the seriousness of the problem.
Christ mentions that the best solution for this problem is not to have
the problem. But in cases that this cannot be avoided he dicusses some
techniques that can be used. He also provided two useful references in
discussion of pathological gait.
>From cmah@cns.ucalgary.caMon Sep 9 11:30:12 1996
>Date: Wed, 28 Aug 1996 17:47:27 -0230
>From: Chris Mah
>To: joshua@engr.mun.ca, manuel@cns15.cns.ucalgary.ca,
> cmah@cns15.cns.ucalgary.ca
>Subject: missing markers
>
> Dear Joshua:
>
> Despite what some people may tell you, the problem of
> missing markers is a serious one. When data are missing, there is no
> good way to rectify the problem, because any method you use involves
> constructing estimates of data you dont have. For highly pathological
> gaits, it may occur unpredictably and often. Clever methods to replace
> missing data are usually not worth the effort, and only make matters
> worse.
>
> So the first line of defence against missing markers is not
> to have the problem. This means having more cameras if possible, to
> increase the probability that at least two can see them all. In additi=
on,
> you need to think about simplifying your data collection protocol (if
> that is acceptable) so that the missing marker problem does not occur.
> Having said this, there are different degrees of seriousness
> for this problem. When there is a short (say < 100ms), isolated
> gap, in the sense that there is a stretch of good data on either
> size of it, there is no problem filling in the gap with a straight
> line or polynomial algorithm. When there are longer gaps, or
> stuttering intervals of missing data it is helpful to have some
> kind of proxy marker which might allow you to infer the position of
> the missing marker.
>
> We often have this problem with the hip during sagittal
> view recordings of normal walking, because it is covered by the hand
> during normal arm swing. To deal with the problem we have our subjects
> wear a tail piece of 10 - 15 cm in length attached rigidly to a belt.
> This remains in view when the hip is covered. Assuming that the subjec=
ts
> are walking in a straight line, there is a fixed spatial relationship
> between the hip marker and the tail, and the position of the hip durin=
g
> relatively short gaps (say 500ms) can be inferred.
>
> A slightly different, but related strategy (which I can sugges=
t
> but have not actually employed) is to extrapolate along a straight li=
ne
> between two markers. For example, if you have trouble seeing an ankle
> marker, put two markers on the lower leg, so that the ankle is (say
> 10cm) further along a straight line drawn between the two markers.
>
> As far as your analytical problem goes, the best technique
> depends on what scientific question you want to ask. However, if you
> are interested in unpredictable pathological gait, you may
> want to take a look at our methods papers.
>
> 1. Quantitative analysis of human movement synergies: constructive
> pattern analysis for gait (1994). C.D. Mah, M. Hulliger, R.G. Lee and
> I. O'Callaghan. Journal of motor behavior. 26, 83-102.
>
> 2. Quantitative Analysis Techniques for Human Movement: Finding
>Multivariate Patterns in Large Data Sets C.D. Mah, M. Hulliger, R.G. Lee
>and I. O'Callaghan. In M. Witten and D. J. Vincent, eds. Computational
>Medicine, Public Health, and Biotechnology: Building a Man in the
>Machine, Part II, pp. 1056-1069. World Scientific Press, 1996.
>
> I hope you find this helpful.
> Chris Mah
--------------------------------------------------------------------------
[4] Email from Bill Keezel:
Mr. Keezel mentions some packages that will interpolate missing gaps in
data. He says that will small gaps and slow moving subjects curve
fitting is an acceptable method for approximating the gaps.
>From tsrhgait@ix.netcom.comMon Sep 9 11:30:09 1996
>Date: Wed, 28 Aug 1996 17:22:03 -0230
>From: TSRH Gait Lab
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hello Joshua,
> Yes, there are a few software packages out there that will interpolate
>over those missing gaps in your trajectory information. However, the
>accuracy of interpolation is limited by the gap size and the subject's
>walking velocity. Also, the applications I'm thinking of utilize C3D file=
s
>for evaluating a subject's kinetics and kinematics. So, what software are
>you using for your data reduction and what is the range of your gap
>sizes of missing data?
>
>If your gap size is within 10 frames for a camera system collecting at 60h=
z
>and a slow moving (walking) subject, then curve fitting is an acceptable
>method for approximating that gap. There are three applications that we
>utilize in our lab for interpolating over gaps: AMASS(Adtech Motion
>Analysis Software System), Vicon Clinical Manager (Oxford Metrics), and
>EVENTS (developed at the NIH, contact Steven Stanhope at
>sstanhop@cc.nih.gov for more details).
>Looking forward to your reply,
>Bill Keezel
>tsrhgait@ix.netcom.com
>Texas Scottish Rite Hospital Gait Lab
>ph (214)559-7580
>TSRHGAIT@ix.netcom.com
-------------------------------------------------------------------------
[5] Email from Xudong Zhang:
He mentions that if one has an idea of specific motion profiles then, one
can develop curve fitting methods to fill in gaps.
>Date: Wed, 28 Aug 1996 13:20:28 -0230
>From: Xudong Zhang
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hi Joshua,
>I have some experience with the MacReflex system. That system provides
>some capability of filling the drop-outs. However, when the discontinuity
>is too severe, (i.e., missing for a large number of consecutive frames),
>the filling does not work well. Also, it does not extrapolate. I would
>imagine these are problems for most motion analysis systems with filling
>mechanisms.
>If you have a good idea about the general forms of your specific movement
>profiles and you wish to develop your own tracking algorithm, it is
>possible to come up with a curve fitting method to fill your missing data
>better.
>Hope this helps. I look forward to seeing your summary.
>
>_________________________________________________ ______ ___ ___
>Xudong Zhang [ \ / ]
>Dept of Industrial & Operations Engineering |MICHIGAN|
>The University of Michigan Internet: xudong@umich.edu | |\ /| |
>_________________________________________________ ______ [___]\/[___]
--------------------------------------------------------------------------
[6] Email from Ken Olree:
Mr. Olree also agrees that missing markers and disjointed tracks are a=20
problem but the size of the problem can be contained by careful setup.
He also provides som good ideas for recreating markers that have been
missing for an extended period of time. His reply is divided into two=20
sections: (i) Recreating markers and (ii) Path Matching
>From kolree@roadrunner.carenet.orgMon Sep 9 11:30:15 1996
>Date: Wed, 28 Aug 1996 20:05:05 -0230
>From: Ken Olree
>To: joshua@tera.engr.mun.ca
>Subject: Missing Markers
>
>Joshua,
> Missing markers and disjointed marker paths are indeed a problem. =
How
>large a problem it is depends on many factors. Some of these are the
>location and number of cameras used, the number, size, and location of
>markers studied, the activity investigated, the use of walking aids or
>other devices (e.g. EMG equipment) that may block a marker from view
>and the type of equipment and software to do the tracking. With a
>proper set up, it has been my experience that missing markers are not as
>large a problem as the time it takes to manually piece together
>disjointed marker paths. In some cases a 4 to 6 second gait trial can
>take 5 minutes to track and 45 minutes to manually piece the disjointed
>marker paths together, especially if a large number of markers are used,
>the subject is very small and walking aids are used. The software I've
>seen to "recreate" missing markers is usually some sort of interpolation
>procedure to join two known marker paths. This works fine when the marker
>was missing for only a short period of time, but if the marker was
>missing for an extended period of time it is virtually useless.
>Although I haven't implemented any new methods of recreating missing
>markers, I have given it considerable thought. I've outlined some of my
>ideas below.
>
>Recreating Markers
>
>1. 3 markers on a segment uniquely define a segment's location and
>orientation in space. If a 4th marker was used, the redundancy could be
>used to calculate a missing marker. The pros of this are that the
>redundancy allows you to calculate a missing marker (or the segment's
>position and orientation for that matter) when the other 3 are known.
>The cons are that the additional marker will make the tracking and
>editing harder. It may also be difficult to do the calculation in
>practice since a small error in the other 3 locations may result in a
>larger error in the calculated position. This is similar to problems
>calculating centers of rotation from marker data.
>
>2. If the distances between three markers on a segment are known, and the
>location of 2 of these markers are known, the location of the 3rd marker i=
s
>constrained to a circle. This might be used in conjunction with an
>interpolative procedure to provide a better estimate of the marker's
>location. It could also be used where the marker is only seen by one
>camera. One camera view constrains the marker to be on a line. The
>intersection of the line and circle could provide the location.
>
>3. If the data is cyclical it may be possible to "guess" where a reasonab=
le
>location for the marker might be based on where it and the other two marke=
rs
>were at one period before or after the instant the marker is missing.
>
>I think the joining of marker segments is a much more complex and time
>consuming problem and well deserves addressing. My guess is that there wi=
ll
>not be a single algorithm that will work well in matching disjointed marke=
r
>paths. I've noticed that in doing it manually I use a variety of techniqu=
es
>to solve the problem quickly. I think the best approach would combine the=
se
>techniques and then use some sort of artificial intelligence to make a "be=
st
>guess" at which path belongs to which. Some of the techniques you might
>want to consider for path matching are below.
>
>Path Matching
>
>1. Sometimes paths can be easily identified by some unique
>characteristic of their location. For instance, in our lab, superior
>markers on a subject will always have a greater Z coordinate than
>inferior markers. Left sided markers will always have greater Y values.
>Anterior segment markers will have greater X values, although this may
>not be true for markers on the lower extremities as one foot passes the
>other. This will in general be true for all of our gait analyses,
>however, in other activities it may not be true at all (e.g. high jump or
>back flip). Ideally, your algorithm will be more useful if it is
>applicable to any activity, not just gait.
>
>2. Generally when a marker is lost and then picked up again the drop
>out will be for a short period of time. The direction and velocity of
>the two paths will tend to "point" to each other in each of the 3
>dimensions if they are really one path. This may not be true however if
>there are sudden reversals in direction of the marker.
>
>3. If you can reconstruct the marker position using some of the technique=
s
>above, you should be able to use that information to find where a marker
>should be. If you have a data segment that is similar to the
>reconstruction, it may be a match.
>
>My opinion is that for the marker reconstruction algorithm and the data
>matching to be really useful it should be applicable to many activities an=
d
>make as little assumptions about lab coordinates and how the data is
>"supposed" to look as possible. Ideally the algorithm would be smart enou=
gh
>to make some generalizations about the data from the data itself and not
>depend on a human telling it what assumptions to make. I also think there
>will be a need for some human oversight of the algorithm's actions to
>correct mistakes it will make. This is a project that I think is very
>worthwhile and would save a lot of time should you develop an algorithm
>that worked well. I would be very interested in hearing of other
>responses and hearing of any progress you make.
>
>Good Luck,
>
>
>Kenneth S. Olree, MS
>BJC Human Performance Laboratory phone: (314) 454-7592
>Barnes-Jewish and St. Louis fax : (314) 454-5500
>Children's Hospitals
>4555 Forest Park Parkway kolree@roadrunner.carenet.org
>St. Louis, MO 63108
>USA
--------------------------------------------------------------------------
[7] Email from At Hof:
Mr. Hof also confirms that with his ELITE system the missing markers are a =
big
problem.
>Date: Thu, 29 Aug 1996 06:09:57 -0230
>From: "A.L.HOF"
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Dear Joshua,
>
>> cause the marker to be obstructed from view of the cameras. I haven't
>> been able to find any information to show if this really is a problem
>> other than the words of a few people who work in gait laboratories.
>
> I am working, sometimes, with the ELITE system, a version of some=20
>5 years old. Missing markers are really a big problem there: when=20
>they are missing for more than a few frames, the processing does not=20
>go further. Manual correction is even not possible.=20
> With two camera's all the time both cameras should view all=20
>markers. This is really a big problem in any kind of real-world=20
>movement. Some people here tried to get data for the arm movements =20
>from crawling babies. This was very discouraging.
>
>> In many of the vision systems, when the vision system loses sight of the
>> markers the tracks of the marker paths become disjointed. Generally the
>> user has had to connect the disjointed tracks manually. Is there softwa=
re
>> that makes these connections automatically? If so how do they do it?
>Part of the solution would be a set-up with more than two cameras,=20
>in which for each marker data from those two cameras are used that=20
>have the best view on that particular marker.
>
>> Finally, in the prediction of motion, I have narrowed my search to using
>> motion models of those limbs. Are there better techniques, esp. for
>> pathological gait?
>>=20
>> I would greatly appreciate any help in this area. Also, I will summariz=
e
>> all my replies and post them to this newsgroup.
>
> I am very interested in the responses. Particularly I would like=20
>to know whether any of the manufacturers has solutions for this=20
>problem. Free software would be even more handy, of course..
>Greetings,=20
>
>At Hof
>Department of Medical Physiology =20
>University of Groningen
>Bloemsingel 10
>NL-9712 KZ GRONINGEN
>The Netherlands
>Phone: (31) 50 3632645
>Fax: (31) 50 3632751
--------------------------------------------------------------------------
[8] Email from Mark Geil:
He comments that interpolation is made easier when the missing marker
information can be copied from a different stride (cycle) of the same=20
trial. However when using assistive devices in pathological gait usually
the marker is missing for the same portion of each gait cycle.
>Date: Thu, 29 Aug 1996 11:02:59 -0230
>From: "Mark D. Geil"
>To: joshua@tera.engr.mun.ca
>Subject: Missing Markers
>
>Regarding your question on Biomch-L,
>You are correct in your thought that assistive devices common with
>pathological gait can result in missing markers. This presents a problem,
>because the marker in question is usually missing for the same portion of
>each gait cycle. Interpolation is made easier when the missing marker
>information can be copied from a different stride of the same trial.
>However, when a crutch or walker obstructs a camera's view of a marker, it
>typically does this at the same portion of each stride.
>VICON's software will perform interpolation as part of the AMASS package
>upon marker identification or as part of the Vicon Clinical Manager packag=
e
>upon processing gait cycles.
>Best wishes,
>Mark Geil
>Ohio State University Gait Analysis Laboratory, Columbus, Ohio
>(614) 293-4832
--------------------------------------------------------------------------
[9] Email from Kevin Arthur Ball:
Mr. Ball writes that the missing marker problem is pretty serious. He
provides some references
for the automated analysis of gait and for
rigid body kinematics (which I found useful).
>Date: Fri, 30 Aug 1996 11:25:21 -0230
>From: Kevin Arthur Ball
>To: Joshua Swamidas , joshua@engr.mun.ca
>Subject: Re: Some Help please - gait anlaysis
>
>Hello Joshua
>
>First some comments about your email. You have written that "you are=20
>a maters student", I assume by this that you are concerned with all=20
>that matters! In the future I would suggest that you drop this prefix=20
>altogether. Your question can stand on its own, and it is a good one.
>
>"I am looking into the problem of missing markers and the ability to=20
>track them and predict their path while they are out of view of the=20
>vision systems. I have come across some interesting techniques.
>Unfortunately, I haven't been able to find out how big this missing
>marker problem really is."
>
>In my estimation this problem is far more pervasive than has even=20
>been realized. Have you ever wondered why there are virtually no=20
>complete 3D kinematic studies of gait or running that includes run-and-
>cut manoeuvres, turns or pirouettes. Each would certainly be=20
>interesting to study but given the present state of our technology=20
>very few researchers would ever be willing to attempt them. Instead we=20
>tend to restrict gait analysis to straight line walking and use as few=20
>markers as possible so as to avoid "marker collisions".=20
>
>"In many of the vision systems, when the vision system loses sight of=20
>the markers the tracks of the marker paths become disjointed. =20
>Generally the user has had to connect the disjointed tracks manually.=20
>Is there software that makes these connections automatically? If so=20
>how do they do it?"
>
>Some years ago Michael Pierrynowski and I were working with a vision=20
>system similar to that which you have described. At that time we=20
>decided to circumvent the vendors software, so we designed our own=20
>system for data processing. This took the better part of a year to=20
>write. It is described in the following:
>
>Ball, K.A. and Pierrynowski, M.R. (1991). WalkTrak: Automated=20
>analysis of 3D kinematic data from video systems. Proceedings of the=20
>International Symposium on 3D Analysis of Human Movement, First=20
>Meeting, 28-31 July. Montreal, Quebec, Canada, 6-9.
>
>More recently, we have switched to an activer marker system, thus=20
>our previous efforts have been shelved as of late.
>
>If you are willing to explore the use of rigid body methods for=20
>kinematic measurements, then I think you will may find the answer for=20
>the marker drop-out problem. I could provide a long list of=20
>references for you but to save myself some time you will find=20
>many of these in the following:
>
>Ball, K.A. and Pierrynowski, M.R. (1995). Estimation of six degree of=20
>freedom rigid body segment motion from two dimensional data. Human=20
>Movement Science, 14, 139-154
>
>Ball, K.A. and Pierrynowski, M.R. (1996). Classification of errors in=20
>locating a rigid body. Journal of Biomechanics. Any Day Now.
>
>I hope this helps!
>
>Bye for now, Kevin.
>
>P.S. I hope all of you at Memorial have a good 500th anniversary of=20
>Cabot's sailing, eh!
>
>Kevin Arthur Ball, B.P.H.E., M.Sc.
>Director, Biomechanics Laboratory
>Sport Medicine / Biomechanics Group=20
>School of Physical and Health Education
>University of Toronto
>>320 Huron St.
>Toronto, Ontario M5S 1A1
>CANADA
>
>Voice (416) 978-3196
>Fax (416) 978-4384
>ball@phe.utoronto.ca
__________________________________________________ ________________________
[10] Email from Hakan Johnsson, QUALYSIS:
He commented on the complications of the problem of tracking and discusses
some of the techniques used by their software.
>Date: Wed, 28 Aug 1996 12:12:20 -0230
>From: =3D?iso-8859-1?Q?H=3DE5kan_Johnsson?=3D
>To: joshua
>Subject: QUALISYS
>
>Dear Mr. Swamidas,
>
>Thanks a lot for your interest in the MacReflex system. Indeed QUALISYS
>has a wide experience in the field of motion analysis, and tracking
>missing markers is a part of our system.
>
>Furthermore the tracking is one of the more complicated parts of the
>system. This is because of the fact that there is a variety of effects
>disturbing the optical signals from the markers. As you have pointed out
>one of them is that markers are hidden ( or partly hidden ) behind the
>measurement object. Another problem is that the paths of two or more
>markers coincide in one point and are complicated to tell from each
>other automatically by the tracking program.
>Over to your questions:
>1)
>It is obvious that it is easier to track the motions the more cameras
>you use. More cameras will generate more registered 3-D points.
>Unfortunately we don't have any statistics on the number of markers
>missing for different camera setups. But considering measurement
>frequency, field of view and walking speed I suppose that you can make
>an approximation of such a number.
>In our system AutoGait 3D a minimum of four cameras is recommended.
>2)
>In our tracking system an interpolation and extrapolation of the marker
>movements are carried out in order to approximate the next position of a
>marker. The extrapolation 'helps' the tracker to look in the right place
>when looking for the next position of a marker.
>3)
>In correctly measured files with appropriate setups the tracker is able
>to fully automatically join the tracks of the three - dimensional
>movements. There are, however, measurement situations when this is not
>possible ( e.g. complex movements including fu ll body rotations ).
>In such cases the operator can run the tracking manually or, preferably,
>run the tracking separately for different segments in the file. Since
>the tracking can be run both forwards and backwards in the file, a point
>with missing markers can be approached and tracked from two directions.
>
>Best Regards,
>
>H=E5kan Johnsson
>QUALISYS AB
>SWEDEN
>
>hakan.johnsson@qualisys.se
------------------------------------------------------------------------
[11] Email from Mr. Elm Strukol (Peak Performance):
This email was very useful because it provided specific numbers for the
15 point Helen Hayes marker set. As was mentioned in the commercial
software section above, their software will fill in missing gaps either
automatically or manually.
>Date: Wed, 28 Aug 1996 13:19:05 -0230
>From: Elm Sturkol
>To: joshua@engr.mun.ca
>Subject: Missing Marker Interpolation
>
>Thank you for your inquiry regarding missing marker interpolation issues. =
I
>referred the questions to one of our engineers and he responded as follows=
:
>
>Your first question is very broad. There are many different answers
>depending on how many markers are used and on what parts of the body they
>are placed. Not to mention the image acquisition rate of the camera system
>being used. Because of this, I'll limit my answer to a standard clinical
>gait situation using what is called in the industry, a 15 point Helen Haye=
s
>marker set. I don't know how familiar you with this "gait analysis" field,
>so if my descriptions are too basic or not in-depth enough let me know. Th=
is
>"marker set" consists of 15 spherical markers placed on the seven lower bo=
dy
>segments. Because the markers are placed on the left and right sides of th=
e
>body as well as the front and back, a two camera system is not able to dea=
l
>with the situation at all. Unless you use tricks with mirrors to allow the
>markers not directly visible to be seen. Also, three cameras are not enoug=
h.
>You'll need at least four cameras to even have a chance of tracking all 15
>markers. With four cameras, our software can fill in all the "gaps" caused
>by markers that are not in clear view of at least two cameras. It can also
>fill in the gaps in the 2-D image space data and create 3-D coordinates
>where the marker is not in view of any camera. However, this "filled in"
>data is not very accurate because with four cameras, the gaps are frequent
>and very large. Out of the 15 points there may be only six or seven of the=
m
>that can be reconstructed to 3-D without filling in any gaps. The others m=
ay
>have up to 50 percent of their image space data missing. Using five camera=
s,
>this percentage may decrease to 30 percent. And, using six cameras or more=
,
>this is reduced to about 10 percent or less in all but the most difficult
>cases. To accurately fill in the gaps, the number of cameras is increased =
so
>that each marker is clearly visible by at least two cameras all the time.
>
>The marker motion is predicted in several ways both in 2-D and 3-D.
>Extrapolation and interpolation algorithms based on human motion are used =
in
>different combinations to predict not only which path segment may join to
>another, but also, what direction the point is moving in the camera's imag=
e
>space so that it can be self correcting. The marker prediction is very
>complex and not easy to describe briefly.
>By setting several tracking parameters used by the software, the marker
>paths can be automatically connected or the user can connect them manually=
>The user is also allowed to break apart path fragments already connected a=
nd
>reconnect them differently. Our Motus software has a whole interface
>specifically devoted to this process.
>
>A lot has been done already using brute force mathematical extrapolation a=
nd
>interpolation along with more cameras to accurately fill in missing data.
>However, there is still plenty of room for improvement. Especially in the
>areas of using fewer cameras with software or hardware that is smart enoug=
h
>to fill in bigger and bigger gaps in the data.
>
>I hope this has been helpful.
>
>The above was written by Steve Risenhoover. Please contact us again if we
>can be of help.
>
>Mr. Elm Sturkol
>National Sales Representative
---------------------------------------------------------------------------=
-
[12] Email from John Greaves (Motion Analysis Corporation):
Dr. Greaves also acknowledges the problem of tracking several markers.
He mentions that their ExpertVision HiRES system does provide tools for
filling in the gaps in marker tracks. With short gaps of no more than a
few frames a software (Track Mender) looks at the continuity of the 3D
math segments (and a few other things) and automatically joins the
segments. Another method is one that involves human intervention to help
draw a curve over the gap.
>Date: Wed, 28 Aug 1996 14:44:10 -0230
>From: John Greaves
>To: Joshua Swamidas
>Cc: Rebecca Stine
>Subject: Re: Some Help please - gait analysis
>Hi Joshua,
>
> I'm John Greaves, founder and Sr. VP at Motion Analysis Corporation in
>California. Here are some ways that our ExpertVision HiRES video motion
>capture system deals with the missing marker problem you mentioned. First
>of all, gait is not too difficult to "track" (which means reduction to 3D
>coordinates from the raw camera data in our parlance) in the grand scheme
>of what people are doing with motion capture systems. What is more
>difficult is collecting data from two or three persons, each with 30 or 40
>markers to accurately represent their limb segment motions. Anyway,
>pathological gait we also have a lot of experience with (the OrthoTrak gai=
t
>analysis software has been on the market since about 1987) and pathologica=
l
>gait is more difficult to track than normal subjects, as you know.
>
>The best solution is to have redundant cameras for picking up more marker
>positions from more camera locations. A minimum of 4 cameras (two for eac=
h
>side) is required for markers placed around the body, although the folks a=
t
>Newington Children's hospital used 3 cameras for a long time by designing =
a
>special marker set with markers only on the front of the body. As you add
>more cameras, the more redundancy you have in your data collection
>mechanism and the fewer holes you will have in your data. We sell mostly =
6
>camera systems (roughly 3 cameras per side) for gait analysis now, and the
>data can be tracked with almost no "holes" in the 3D data set. BUT if
>there are holes, the mechanisms we have are:
>
>1-The Track Mender, which looks at the continuity of 3D path segments and
>automatically joins them if they are no more than a few frames apart and
>meet some equations about continuity in the XYZ components and their first
>derivatives. If you don't like the results, you can Undo it.
>
>2-The Add button in the XYZ time-series editor. Here you can position the
>mouse in a hole in each component (X, Y and Z) and place a point. The
>software splines across the gap with your input and you confirm each
>component separately. It turns out that the human is very good at seeing
>what "looks right" from this XYZ editor view--much better than what you ca=
n
>"see" from a stick figure or 3D spatial view of your data.
>
>Anyway, I hope this helps. We have spent years refining the tools for
>accurate data collection, tracking and editing and we were just at the
>Canadian Society of Biomechanics meeting, but that was on the opposite end
>of Canada from you.
>
>Sincerely,
>John Greaves, Ph.D.
>Sr. Vice-President
>Motion Analysis Corporation
-------------------------------------------------------------------------
[13] Email from Georger Roundanez (VICON):
>Date: Fri, 23 Aug 1996 12:37:08 -0230
>From: Georges Roudanez
>To: joshua@engr.mun.ca
>Subject: Re: mailto:georges@viconsys.com
>
>>1) How frequently do these holes (missing data) occur say, with a 2, 3=20
>> and 5 camera system and in studying pathalogical walking? The system
>> that will be purcahsed will be used maily wth CF kids and for research
>>
>Joshua,
>
>This is a difficult question to answer. The lost trajectories occur as a
>result of a marker not being seen by at least 2 cameras (in order to provi=
de
>xyz coordinates). The more cameras, the placement of those cameras and th=
e
>type of movement being measured determine the opportunity to avoid
>occlusions (not seen markers). A simple move with two cameras being
>constantly in view of the markers will most likely result in no occlusions=
>
>
>> purposes.
>>2) Do you have a way of predicting the marker motion?
>
>The way the Vicon operates it is possible to determine the anitcicipated
>marker path (trajectory).
>
>>3) Does the software automatically connect disjointed marker tracks or do=
=20
>> you let the operators handle the connecting of incomplete tracks.
>
>The answer is yes to both. You may choose to have the software (BodyBuild=
er
>- a special package that has editing and modelling features) automatically
>fill gaps, delete and fill noise spikes and interpolate or smooth the data=
-------------------------------------------------------------------------
[14] Email from Jermy Wise (Ariel):
Dr. Wise comments on the general problem of missing markers and makes=20
comments on how their software handles the problem. They seem to use
linear extrapolation of some kind and also some smoothing functions to
fill in the shorter gaps.
>Subject: Re: missing marker solution=20
> Date: Mon, 9 Sep 1996 18:25:52 -0230=20
> From: Jeremy Wise
> To: joshua@tera.engr.mun.ca
> CC: ariel1@ix.netcom.com
>
>Joshua-
>
>Dr Ariel has asked me to respond to your questions regarding handling of
>incomplete tracks with the APAS system. I'll do my best.
>
> >I understand that you develop state of the art vision systems and the
> >software. There are three questions I'd like to ask:
> >1) How frequently do these holes (missing data) occur say, with a 2, 3
> > and 5 camera system and in studying pathalogical walking? The system
> > that will be purcahsed will be used maily wth CF kids and for researc=
h
> > purposes.
>
>It's impossible to say how often this occurs because it depends entirely
>on the activity under consideration. Generally the more rotation about a
>vertical axis the more often a marker will become obscured. The more
>cameras one has the more likely atleast 2 cameras will see a marker for
>all frames. If 2 or more cameras "see" a marker then the software can
>calculate the 3D coordinates.
>
> >2) Do you have a way of predicting the marker motion?
>
>The program predicts marker motion using a rather complex search
>algorithm. The location the software starts searching at is an
>extrapolation of the previous 2 frames or previous frame, a software
>selectable switch. The reason a simple linear extrapolation is used is
>because higher order extrapolations are more error prone.
>
> >3) Does the software automatically connect disjointed marker tracks or d=
o
> > you let the operators handle the connecting of incomplete tracks.
>
>I'm not sure if you are referring to tracks in 2D or reconstructed 3D.
>If you are talking about the plane of the video, the software can be set
>to skip and flag as MISSING any markers which the software can't find, or
>the software can be configured to pause & let the user estimate marker
>location based on his/her observation of the whole image. If you are
>talking about 3D, then the program flags any point that cannot be seen by
>atleast 2 cameras as MISSING and will be considered as missing RAW data.
>However, when it is smoothed, and incomplete sections of data will be
>interpolated then smoothed with the selected algorithm. What you are
>asking the software to do is to make data where it can't be seen be
>atleast 2 cameras. This is risky business.
>
>I hope this helps,
> Dr. Jeremy Wise
> Dir R&D
-------------------------------------------------------------------------
[15] Email from Mandy Sketch:
Date: Wed, 28 Aug 1996 12:26:31 -0230
From: Mandy Sketch
To: joshua@engr.mun.ca
Subject: Some Help please - gait anlaysis -Reply
I am out of the office until September 2nd.
If you need to contact me urgently please either call or fax my office (tel=
:
44 1865 843658, fax:
44 1865 843965) and ask for a message to be passed on. Alternatively I will
respond to your
message on my return.
Mandy Sketch
Publishing Editor
----------------------------------------------------------------------
*******************End of Email Replies*******************************
Thanks again,
Joshua Swamidas
--
joshua@tera.engr.mun.ca
Box 30, Faculty of Engineering and Applied Science,
Memorial University of Newfoundland
St. John's, NF A1B-3X5
(709) 737-8809
I would like to apologize for the tardiness of this summary and I'd like to
thank all of you who took the time to help me. The information was very
helpful.
First, here is the message that I had sent:
> I'm a maters student who is working in the area of gait. I have
> access to data from a multi-camera vision system. I am looking
> into the problem of missing markers and the ability to track them and
> predict their path while they are out of view of the vision systems. I
> have come across some interesting techniques.
>
> Unfortunately, I haven't been able to find out how big this missing
> marker problem really is. I understand that as long as the markers
> placed in the anatomical sites are viewed by any two cameras the 3D
> positions of these markers can be calculated. However, in case of
> pathalogical gait, the use of canes, walkers, walking pattern, etc., can
> cause the marker to be obstructed from view of the cameras. I haven't
> been able to find any information to show if this really is a problem
> other than the words of a few people who work in gait laboratories.
>
> In many of the vision systems, when the vision system loses sight of the
> markers the tracks of the marker paths become disjointed. Generally the
> user has had to connect the disjointed tracks manually. Is there softwar=
e
> that makes these connections automatically? If so how do they do it?
>
> Finally, in the prediction of motion, I have narrowed my search to using
> motion models of those limbs. Are there better techniques, esp. for
> pathological gait?
>
MISSING MARKERS:
Pertaining to the problem of missing markers, it seems that almost
everyone agreed that it was a problem (to varying degrees of
seriousness). The best solution, as one person said, is to make sure the
problem does not occur and this could be done by: increasing the number=20
of cameras viewing the tracking region; using non-camera methods of
tracking such as magnetic tracking[1]; carefully choosing a marker set
that would stay in view despite obstructions, rotations, etc,; and using
special assistive devices that minimize intereference with the=20
cameras [2].
PREDICTING MARKERS AND CONNECTING DISJOINTED TRACKS:
As for actually filling in the gaps using software, I have broken this
down into two separate sections: the first one deals with comments from
individuals and the second would be the companies that specialize in=20
motion analysis' hardware and software such as VICON, Ariel Dyanmics,
Inc., QUALYSIS, Peak Performance Technologies, and Elite (BTS).
The techniques are listed in point form and the reference number attached
to it is based on the order of the email responses which are concatenated=
=20
to the end of this summary. The following is a list of techniques and
algorithms:
- Join missing gaps using spline functions (cubic or quintic) [1]. It
may not be very accurate over large gaps. Cheng cautions that some
kind of human involvment is usually necessary when using visual systems.
=20
- Using a proxy marker to provide an offset-recording [2,3] of the marker
that consistently disappears from view. This proxy marker can then be=20
used to infer the position of the missing marker.=20
- For short isolated gaps, using a straight line or polynomial algorithm
works well [3]
- Extrapolate along a straight line between two markers [3]. For example,
if you have trouble seeing an ankle marker, put two markers on the lower=
=20
leg, so that the ankle is further along the straight line drawn between
the twp markers.
- For constructing analytical models for analysis [3] some papers to=20
consult are:
Quantitative analysis of human movement synergies: constructive pattern
analysis for gait, C.D. Mah, M. Hulliger, R.G. Lee and I.O. Callaghan,
Journal of Motor Behaviour, 26, 83-102, 1994.
Quantitative analysis techniques for human movement: finding multivriate
patterns in large data sets, in M. Whitten and D.J. Vincent, eds.=20
Computational Medicine, Public Health, and Biotechnology: Building a man
in the machine, Part II, pp. 1056-1069. World Scientific Press, 1996
- Curve fitting alogrithms are fine for small gap sizes (10 frames) [4].
- If the general forms of a specific movement are known, then a tracking
algorithm can be written for a this specific problem [5].
- 3 markers uniquely define a segment's location and orientation in space.
If a 4th marker is used, the redundancy could be used to calculate a miss=
ing
marker. Some pros and cons of this technique are discussed in email [6].
- If the distances between three markers on a segment are known and the
location of two of the marker are known, the location of the third marker
can be constrained to a small area. This might be used in conjunction=20
with a interpolative procedure to give a better estimate of location [6].
- If the data is cyclical, it may be possible to "guess" where a reasonable
location of the marker might be based on where it was one period before
or after the instant [6]. However, in gait analysis cases where an
assistive device obstructs the camera view they tend to happen in the=20
same potion of each stride [8].
- For path matching, sometimes paths can be easily identified by some uniqu=
e
characteristic of their location [6] such as a marker on the upper part o=
f
the body would have greater values in the height coordinates.
- Using direction and velocity to help identify possible path matches [6].=
=20
This will work well if the gaps are short and there are no sudden reversa=
ls
in direction of the marker.
- Again for path matching, using a marker path reconstruction technique to=
=20
reconstruct a path and match it to an observed path [6].
- Other useful papers[9]:=20
Ball, K.A. and Pierrynowski, M.R. (1991), WalkTrak: Automated analysis=20
of 3D kinematic data from video systems. Proceedings of International
Symposium on 3D Analysis of Human Movement, First Meeting, 28-31 July,
Montreal, Quebec, Canada, 6-9.
Papers on rigid body kinematics:
Ball, K.A. and Pierrynowski, M.R. (1995), Estimation of siz degree of=20
freedom rigid body segment motion from two dimensional data, Human=20
Movement Science, 14, 139-154
Ball, K.A. and Pierrynowski, M.R. (1996), Classification of errors in
locating a rigid body, Journal of Biomechanics, Any day now.
The following is a list of the companies and a brief comment on their=20
software.
Qualysis (Sweden & USA) [10] who manufactures the MacReflex Motion Measurem=
ent
Systems for the Macintosh or PC computers gave me information on how they
track the markers and what they do for prediction of the missing markers.
In handling the tracking of markers, the Qualysis software extrapolation
of the marker movements are carried out to approximate the next position
of the marker. This extrapolation helps the tracker to look in the
right place when looking for the next position of the marker. In
situations where the adding of extra cameras will not help solve the
marker out-of-view-of-cameras problems, the operator can run the tracking
manually or run the tracking separately for different segments of motion
stored in a file. Since the tracking can be run both forwards and
backwards in the file, a point with missing markers can be approached and
tracked from both directions. A user of this software (E-mail Corresponden=
ce,
Zhang, 1996) commented that when the discontinuities were too large the
filling in of gaps did not work well.
Peak Performance Technologies, Inc. (Englewood, CO, USA), says that their
latest software [11] allows the user to connect the gaps manually by
connecting points, semi-automatically by allowing the user to run different
algorithms, or automatically by allowing the user to set several tracking
parameters. According to the engineer, while much has been done using
brute force mathematical extrapolation and interpolation, along with more
cameras, to accurately fill in missing data, there is still plenty of
room for improvement. Especially in the areas of using fewer cameras
with software or hardware that is smart enough to fill in bigger and
bigger gaps in the data.
The ELITE system by Bioengineering Technology & Systems (Milano, Italy)
primary products include: ELITEplus (Three-dimensional motion package),
ELICLINIC (Clinical gait analysis), TELEMG (Dynamic electromyography with
no limitations on subject motion), GAITemg (Identification of muscle
activity in walking), PcVect (Analysis of ground reaction forces), etc.
The Elite software provides interpolation to handle relatively short gaps.
According to the founder and Senior Vice President of Motion Analysis
Corporation, [12] pathological gait motion is difficult to track. They have
software, the Track Mender, which looks at the continuity of 3D path segmen=
ts
and automatically joins them if they are no more than a few frames apart.=
=20
Another technique used by their software is to allow the user to manually
spline across the gaps.
VICON Motion Systems (Oxford, England and Santa Fe Springs, CA, USA)=20
Bodybuilder software [13] can automatically fill gaps, interpolate=20
and smooth data. It also allows for the user to manually edit and model
the data. The software uses kinematic equations and/or some physical=20
modelling of the body to connect disjointed marker tracks.
Ariel Dynamics,Inc. (Englewood, CA, USA) software [14] does provide
some gap filling algorithms. They use some complex search algorithms and
linear extrapolation to predict marker motion over the missing gaps. The
software also does some smoothing where interpolation is done to fill in=20
the smaller gaps. Their software does allow the user to estimate the marke=
r
locations based on their observations of the whole range.
I will now list the emails that I did receive. The first emails are from
indivduals from various labs and departments around the world and the
later ones are from the companies. I preface each email with a short
summary of what they contain (for the shorter emails this will seem
redundant) I hope that this summary has been useful and not long winded
or boring.
***************** Recieved emails *************************************
[1] Email from Cheng Cao:
He mentions that a small gap can be filled using spline functions. He
says the the larger the gap, the less accurate the 'filled-in' data would
be and states that some kind of human involvement is needed when using
visual systems.
>Date: Wed, 28 Aug 1996 14:12:51 -0230
>From: Cheng Cao
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hi,
>
>The missing makrer thing is a big problem most of time. You can always
>join a missing gap or gaps usin spline functions (cubic or quintic)
>"automatically". But, the danger is how real or accurate your
>data would be without considering the size of the gap and when the gap
>appears. Thus, some kinds of human involvement are mostly needed when
>you using visual systems. But, if you use a magnetic system, there is no
>such problem, of course, there are other problems.
>
>Cheng Cao. Department of Mechanical Engineering and Applied Mechanics
> University of Michigan, Ann Arbor. -*-
--------------------------------------------------------------------------
[2] Email from Dr. Manuel Hulliger:
In this email Dr. Hulliger comments on the the problems with mssing
markers and briefly comments on one technique being used by his department.
>Date: Wed, 28 Aug 1996 14:44:17 -0230
>From: Manuel Hulliger
>To: joshua@engr.mun.ca, mhullige@acs.UCalgary.CA
>Subject: Marker obstruction - gait anlaysis
>
>Dear Joshua,
>
>Chris Mah from our gait lab will send you a more comprehensive reply.
>There is therefore no need to include this one in your summary. At
>any rate, when you compose your summary, you would do the community
>a service by submitting a genuine digest, summarizing main points
>thematically and regrouping the various elements from different replies
>accordingly, rather than merely appending poorly correlated and hetero-
>genous replies in incoming order, as we see so often in the debates
>going on in this forum.
>
>Marker obstruction often is a genuine problem, even in normal gait:
>with a two camera system arm movements obscure hip markers quite
>regularly and predictably. The Elite software we are using provides
>interpolation software to handle relatively short gaps. In addition,
>we are using a "tail" attached to the subject's lower back, to get
>an offset-recording of hip marker coordinates (in crude approximation).
>You may want to consult Mah et al., Journal of Motor Behaviour 26,
>83-102, 1994.
>
>With multi-camera systems (Vicon is going up to 7, I understand) this
>problem appears to be reduced in magnitude.
>
>The above problems are exacerbated in the conditions you list. Canes
>and walkers indeed cause additional loss of marker information. For
>our work with cerebral palsy patients we have constructed a special
>walker to minimize such interference. For patients using canes, we use
>parallel bars to provide support (on the side remote to the cameras).
>
>I hope this helps a bit. Chris Mah will reply more extensively.
Yes this was very useful.
>
>You might want to visit the lab to get some first hand experience.
That would be very interesting. I hope to do this sometime in the future.
>
>Yours,
>Manuel
>
>Manuel Hulliger, D.Phil.
>Department of Clinical Neurosciences
>Faculty of Medicine
>University of Calgary
>3330 Hospital Drive N.W.
>Calgary, Alberta, Canada T2n 4N1
>Phone: 403-220-6216 [Ellen Wong Phone: 403-220-8389]
>Fax: 403-283-8770 [Secretary Fax: 403-283-8731]
>E-mail:manuel@cns.ucalgary.ca or mhullige@acs.ucalgary.ca
>www: http://www.cns.ucalgary.ca/people/hulliger.html
--------------------------------------------------------------------------
[3] Email from Chris Mah:
This is an informative email about the seriousness of the problem.
Christ mentions that the best solution for this problem is not to have
the problem. But in cases that this cannot be avoided he dicusses some
techniques that can be used. He also provided two useful references in
discussion of pathological gait.
>From cmah@cns.ucalgary.caMon Sep 9 11:30:12 1996
>Date: Wed, 28 Aug 1996 17:47:27 -0230
>From: Chris Mah
>To: joshua@engr.mun.ca, manuel@cns15.cns.ucalgary.ca,
> cmah@cns15.cns.ucalgary.ca
>Subject: missing markers
>
> Dear Joshua:
>
> Despite what some people may tell you, the problem of
> missing markers is a serious one. When data are missing, there is no
> good way to rectify the problem, because any method you use involves
> constructing estimates of data you dont have. For highly pathological
> gaits, it may occur unpredictably and often. Clever methods to replace
> missing data are usually not worth the effort, and only make matters
> worse.
>
> So the first line of defence against missing markers is not
> to have the problem. This means having more cameras if possible, to
> increase the probability that at least two can see them all. In additi=
on,
> you need to think about simplifying your data collection protocol (if
> that is acceptable) so that the missing marker problem does not occur.
> Having said this, there are different degrees of seriousness
> for this problem. When there is a short (say < 100ms), isolated
> gap, in the sense that there is a stretch of good data on either
> size of it, there is no problem filling in the gap with a straight
> line or polynomial algorithm. When there are longer gaps, or
> stuttering intervals of missing data it is helpful to have some
> kind of proxy marker which might allow you to infer the position of
> the missing marker.
>
> We often have this problem with the hip during sagittal
> view recordings of normal walking, because it is covered by the hand
> during normal arm swing. To deal with the problem we have our subjects
> wear a tail piece of 10 - 15 cm in length attached rigidly to a belt.
> This remains in view when the hip is covered. Assuming that the subjec=
ts
> are walking in a straight line, there is a fixed spatial relationship
> between the hip marker and the tail, and the position of the hip durin=
g
> relatively short gaps (say 500ms) can be inferred.
>
> A slightly different, but related strategy (which I can sugges=
t
> but have not actually employed) is to extrapolate along a straight li=
ne
> between two markers. For example, if you have trouble seeing an ankle
> marker, put two markers on the lower leg, so that the ankle is (say
> 10cm) further along a straight line drawn between the two markers.
>
> As far as your analytical problem goes, the best technique
> depends on what scientific question you want to ask. However, if you
> are interested in unpredictable pathological gait, you may
> want to take a look at our methods papers.
>
> 1. Quantitative analysis of human movement synergies: constructive
> pattern analysis for gait (1994). C.D. Mah, M. Hulliger, R.G. Lee and
> I. O'Callaghan. Journal of motor behavior. 26, 83-102.
>
> 2. Quantitative Analysis Techniques for Human Movement: Finding
>Multivariate Patterns in Large Data Sets C.D. Mah, M. Hulliger, R.G. Lee
>and I. O'Callaghan. In M. Witten and D. J. Vincent, eds. Computational
>Medicine, Public Health, and Biotechnology: Building a Man in the
>Machine, Part II, pp. 1056-1069. World Scientific Press, 1996.
>
> I hope you find this helpful.
> Chris Mah
--------------------------------------------------------------------------
[4] Email from Bill Keezel:
Mr. Keezel mentions some packages that will interpolate missing gaps in
data. He says that will small gaps and slow moving subjects curve
fitting is an acceptable method for approximating the gaps.
>From tsrhgait@ix.netcom.comMon Sep 9 11:30:09 1996
>Date: Wed, 28 Aug 1996 17:22:03 -0230
>From: TSRH Gait Lab
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hello Joshua,
> Yes, there are a few software packages out there that will interpolate
>over those missing gaps in your trajectory information. However, the
>accuracy of interpolation is limited by the gap size and the subject's
>walking velocity. Also, the applications I'm thinking of utilize C3D file=
s
>for evaluating a subject's kinetics and kinematics. So, what software are
>you using for your data reduction and what is the range of your gap
>sizes of missing data?
>
>If your gap size is within 10 frames for a camera system collecting at 60h=
z
>and a slow moving (walking) subject, then curve fitting is an acceptable
>method for approximating that gap. There are three applications that we
>utilize in our lab for interpolating over gaps: AMASS(Adtech Motion
>Analysis Software System), Vicon Clinical Manager (Oxford Metrics), and
>EVENTS (developed at the NIH, contact Steven Stanhope at
>sstanhop@cc.nih.gov for more details).
>Looking forward to your reply,
>Bill Keezel
>tsrhgait@ix.netcom.com
>Texas Scottish Rite Hospital Gait Lab
>ph (214)559-7580
>TSRHGAIT@ix.netcom.com
-------------------------------------------------------------------------
[5] Email from Xudong Zhang:
He mentions that if one has an idea of specific motion profiles then, one
can develop curve fitting methods to fill in gaps.
>Date: Wed, 28 Aug 1996 13:20:28 -0230
>From: Xudong Zhang
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Hi Joshua,
>I have some experience with the MacReflex system. That system provides
>some capability of filling the drop-outs. However, when the discontinuity
>is too severe, (i.e., missing for a large number of consecutive frames),
>the filling does not work well. Also, it does not extrapolate. I would
>imagine these are problems for most motion analysis systems with filling
>mechanisms.
>If you have a good idea about the general forms of your specific movement
>profiles and you wish to develop your own tracking algorithm, it is
>possible to come up with a curve fitting method to fill your missing data
>better.
>Hope this helps. I look forward to seeing your summary.
>
>_________________________________________________ ______ ___ ___
>Xudong Zhang [ \ / ]
>Dept of Industrial & Operations Engineering |MICHIGAN|
>The University of Michigan Internet: xudong@umich.edu | |\ /| |
>_________________________________________________ ______ [___]\/[___]
--------------------------------------------------------------------------
[6] Email from Ken Olree:
Mr. Olree also agrees that missing markers and disjointed tracks are a=20
problem but the size of the problem can be contained by careful setup.
He also provides som good ideas for recreating markers that have been
missing for an extended period of time. His reply is divided into two=20
sections: (i) Recreating markers and (ii) Path Matching
>From kolree@roadrunner.carenet.orgMon Sep 9 11:30:15 1996
>Date: Wed, 28 Aug 1996 20:05:05 -0230
>From: Ken Olree
>To: joshua@tera.engr.mun.ca
>Subject: Missing Markers
>
>Joshua,
> Missing markers and disjointed marker paths are indeed a problem. =
How
>large a problem it is depends on many factors. Some of these are the
>location and number of cameras used, the number, size, and location of
>markers studied, the activity investigated, the use of walking aids or
>other devices (e.g. EMG equipment) that may block a marker from view
>and the type of equipment and software to do the tracking. With a
>proper set up, it has been my experience that missing markers are not as
>large a problem as the time it takes to manually piece together
>disjointed marker paths. In some cases a 4 to 6 second gait trial can
>take 5 minutes to track and 45 minutes to manually piece the disjointed
>marker paths together, especially if a large number of markers are used,
>the subject is very small and walking aids are used. The software I've
>seen to "recreate" missing markers is usually some sort of interpolation
>procedure to join two known marker paths. This works fine when the marker
>was missing for only a short period of time, but if the marker was
>missing for an extended period of time it is virtually useless.
>Although I haven't implemented any new methods of recreating missing
>markers, I have given it considerable thought. I've outlined some of my
>ideas below.
>
>Recreating Markers
>
>1. 3 markers on a segment uniquely define a segment's location and
>orientation in space. If a 4th marker was used, the redundancy could be
>used to calculate a missing marker. The pros of this are that the
>redundancy allows you to calculate a missing marker (or the segment's
>position and orientation for that matter) when the other 3 are known.
>The cons are that the additional marker will make the tracking and
>editing harder. It may also be difficult to do the calculation in
>practice since a small error in the other 3 locations may result in a
>larger error in the calculated position. This is similar to problems
>calculating centers of rotation from marker data.
>
>2. If the distances between three markers on a segment are known, and the
>location of 2 of these markers are known, the location of the 3rd marker i=
s
>constrained to a circle. This might be used in conjunction with an
>interpolative procedure to provide a better estimate of the marker's
>location. It could also be used where the marker is only seen by one
>camera. One camera view constrains the marker to be on a line. The
>intersection of the line and circle could provide the location.
>
>3. If the data is cyclical it may be possible to "guess" where a reasonab=
le
>location for the marker might be based on where it and the other two marke=
rs
>were at one period before or after the instant the marker is missing.
>
>I think the joining of marker segments is a much more complex and time
>consuming problem and well deserves addressing. My guess is that there wi=
ll
>not be a single algorithm that will work well in matching disjointed marke=
r
>paths. I've noticed that in doing it manually I use a variety of techniqu=
es
>to solve the problem quickly. I think the best approach would combine the=
se
>techniques and then use some sort of artificial intelligence to make a "be=
st
>guess" at which path belongs to which. Some of the techniques you might
>want to consider for path matching are below.
>
>Path Matching
>
>1. Sometimes paths can be easily identified by some unique
>characteristic of their location. For instance, in our lab, superior
>markers on a subject will always have a greater Z coordinate than
>inferior markers. Left sided markers will always have greater Y values.
>Anterior segment markers will have greater X values, although this may
>not be true for markers on the lower extremities as one foot passes the
>other. This will in general be true for all of our gait analyses,
>however, in other activities it may not be true at all (e.g. high jump or
>back flip). Ideally, your algorithm will be more useful if it is
>applicable to any activity, not just gait.
>
>2. Generally when a marker is lost and then picked up again the drop
>out will be for a short period of time. The direction and velocity of
>the two paths will tend to "point" to each other in each of the 3
>dimensions if they are really one path. This may not be true however if
>there are sudden reversals in direction of the marker.
>
>3. If you can reconstruct the marker position using some of the technique=
s
>above, you should be able to use that information to find where a marker
>should be. If you have a data segment that is similar to the
>reconstruction, it may be a match.
>
>My opinion is that for the marker reconstruction algorithm and the data
>matching to be really useful it should be applicable to many activities an=
d
>make as little assumptions about lab coordinates and how the data is
>"supposed" to look as possible. Ideally the algorithm would be smart enou=
gh
>to make some generalizations about the data from the data itself and not
>depend on a human telling it what assumptions to make. I also think there
>will be a need for some human oversight of the algorithm's actions to
>correct mistakes it will make. This is a project that I think is very
>worthwhile and would save a lot of time should you develop an algorithm
>that worked well. I would be very interested in hearing of other
>responses and hearing of any progress you make.
>
>Good Luck,
>
>
>Kenneth S. Olree, MS
>BJC Human Performance Laboratory phone: (314) 454-7592
>Barnes-Jewish and St. Louis fax : (314) 454-5500
>Children's Hospitals
>4555 Forest Park Parkway kolree@roadrunner.carenet.org
>St. Louis, MO 63108
>USA
--------------------------------------------------------------------------
[7] Email from At Hof:
Mr. Hof also confirms that with his ELITE system the missing markers are a =
big
problem.
>Date: Thu, 29 Aug 1996 06:09:57 -0230
>From: "A.L.HOF"
>To: Joshua Swamidas
>Subject: Re: Some Help please - gait anlaysis
>
>Dear Joshua,
>
>> cause the marker to be obstructed from view of the cameras. I haven't
>> been able to find any information to show if this really is a problem
>> other than the words of a few people who work in gait laboratories.
>
> I am working, sometimes, with the ELITE system, a version of some=20
>5 years old. Missing markers are really a big problem there: when=20
>they are missing for more than a few frames, the processing does not=20
>go further. Manual correction is even not possible.=20
> With two camera's all the time both cameras should view all=20
>markers. This is really a big problem in any kind of real-world=20
>movement. Some people here tried to get data for the arm movements =20
>from crawling babies. This was very discouraging.
>
>> In many of the vision systems, when the vision system loses sight of the
>> markers the tracks of the marker paths become disjointed. Generally the
>> user has had to connect the disjointed tracks manually. Is there softwa=
re
>> that makes these connections automatically? If so how do they do it?
>Part of the solution would be a set-up with more than two cameras,=20
>in which for each marker data from those two cameras are used that=20
>have the best view on that particular marker.
>
>> Finally, in the prediction of motion, I have narrowed my search to using
>> motion models of those limbs. Are there better techniques, esp. for
>> pathological gait?
>>=20
>> I would greatly appreciate any help in this area. Also, I will summariz=
e
>> all my replies and post them to this newsgroup.
>
> I am very interested in the responses. Particularly I would like=20
>to know whether any of the manufacturers has solutions for this=20
>problem. Free software would be even more handy, of course..
>Greetings,=20
>
>At Hof
>Department of Medical Physiology =20
>University of Groningen
>Bloemsingel 10
>NL-9712 KZ GRONINGEN
>The Netherlands
>Phone: (31) 50 3632645
>Fax: (31) 50 3632751
--------------------------------------------------------------------------
[8] Email from Mark Geil:
He comments that interpolation is made easier when the missing marker
information can be copied from a different stride (cycle) of the same=20
trial. However when using assistive devices in pathological gait usually
the marker is missing for the same portion of each gait cycle.
>Date: Thu, 29 Aug 1996 11:02:59 -0230
>From: "Mark D. Geil"
>To: joshua@tera.engr.mun.ca
>Subject: Missing Markers
>
>Regarding your question on Biomch-L,
>You are correct in your thought that assistive devices common with
>pathological gait can result in missing markers. This presents a problem,
>because the marker in question is usually missing for the same portion of
>each gait cycle. Interpolation is made easier when the missing marker
>information can be copied from a different stride of the same trial.
>However, when a crutch or walker obstructs a camera's view of a marker, it
>typically does this at the same portion of each stride.
>VICON's software will perform interpolation as part of the AMASS package
>upon marker identification or as part of the Vicon Clinical Manager packag=
e
>upon processing gait cycles.
>Best wishes,
>Mark Geil
>Ohio State University Gait Analysis Laboratory, Columbus, Ohio
>(614) 293-4832
--------------------------------------------------------------------------
[9] Email from Kevin Arthur Ball:
Mr. Ball writes that the missing marker problem is pretty serious. He
provides some references
for the automated analysis of gait and for
rigid body kinematics (which I found useful).
>Date: Fri, 30 Aug 1996 11:25:21 -0230
>From: Kevin Arthur Ball
>To: Joshua Swamidas , joshua@engr.mun.ca
>Subject: Re: Some Help please - gait anlaysis
>
>Hello Joshua
>
>First some comments about your email. You have written that "you are=20
>a maters student", I assume by this that you are concerned with all=20
>that matters! In the future I would suggest that you drop this prefix=20
>altogether. Your question can stand on its own, and it is a good one.
>
>"I am looking into the problem of missing markers and the ability to=20
>track them and predict their path while they are out of view of the=20
>vision systems. I have come across some interesting techniques.
>Unfortunately, I haven't been able to find out how big this missing
>marker problem really is."
>
>In my estimation this problem is far more pervasive than has even=20
>been realized. Have you ever wondered why there are virtually no=20
>complete 3D kinematic studies of gait or running that includes run-and-
>cut manoeuvres, turns or pirouettes. Each would certainly be=20
>interesting to study but given the present state of our technology=20
>very few researchers would ever be willing to attempt them. Instead we=20
>tend to restrict gait analysis to straight line walking and use as few=20
>markers as possible so as to avoid "marker collisions".=20
>
>"In many of the vision systems, when the vision system loses sight of=20
>the markers the tracks of the marker paths become disjointed. =20
>Generally the user has had to connect the disjointed tracks manually.=20
>Is there software that makes these connections automatically? If so=20
>how do they do it?"
>
>Some years ago Michael Pierrynowski and I were working with a vision=20
>system similar to that which you have described. At that time we=20
>decided to circumvent the vendors software, so we designed our own=20
>system for data processing. This took the better part of a year to=20
>write. It is described in the following:
>
>Ball, K.A. and Pierrynowski, M.R. (1991). WalkTrak: Automated=20
>analysis of 3D kinematic data from video systems. Proceedings of the=20
>International Symposium on 3D Analysis of Human Movement, First=20
>Meeting, 28-31 July. Montreal, Quebec, Canada, 6-9.
>
>More recently, we have switched to an activer marker system, thus=20
>our previous efforts have been shelved as of late.
>
>If you are willing to explore the use of rigid body methods for=20
>kinematic measurements, then I think you will may find the answer for=20
>the marker drop-out problem. I could provide a long list of=20
>references for you but to save myself some time you will find=20
>many of these in the following:
>
>Ball, K.A. and Pierrynowski, M.R. (1995). Estimation of six degree of=20
>freedom rigid body segment motion from two dimensional data. Human=20
>Movement Science, 14, 139-154
>
>Ball, K.A. and Pierrynowski, M.R. (1996). Classification of errors in=20
>locating a rigid body. Journal of Biomechanics. Any Day Now.
>
>I hope this helps!
>
>Bye for now, Kevin.
>
>P.S. I hope all of you at Memorial have a good 500th anniversary of=20
>Cabot's sailing, eh!
>
>Kevin Arthur Ball, B.P.H.E., M.Sc.
>Director, Biomechanics Laboratory
>Sport Medicine / Biomechanics Group=20
>School of Physical and Health Education
>University of Toronto
>>320 Huron St.
>Toronto, Ontario M5S 1A1
>CANADA
>
>Voice (416) 978-3196
>Fax (416) 978-4384
>ball@phe.utoronto.ca
__________________________________________________ ________________________
[10] Email from Hakan Johnsson, QUALYSIS:
He commented on the complications of the problem of tracking and discusses
some of the techniques used by their software.
>Date: Wed, 28 Aug 1996 12:12:20 -0230
>From: =3D?iso-8859-1?Q?H=3DE5kan_Johnsson?=3D
>To: joshua
>Subject: QUALISYS
>
>Dear Mr. Swamidas,
>
>Thanks a lot for your interest in the MacReflex system. Indeed QUALISYS
>has a wide experience in the field of motion analysis, and tracking
>missing markers is a part of our system.
>
>Furthermore the tracking is one of the more complicated parts of the
>system. This is because of the fact that there is a variety of effects
>disturbing the optical signals from the markers. As you have pointed out
>one of them is that markers are hidden ( or partly hidden ) behind the
>measurement object. Another problem is that the paths of two or more
>markers coincide in one point and are complicated to tell from each
>other automatically by the tracking program.
>Over to your questions:
>1)
>It is obvious that it is easier to track the motions the more cameras
>you use. More cameras will generate more registered 3-D points.
>Unfortunately we don't have any statistics on the number of markers
>missing for different camera setups. But considering measurement
>frequency, field of view and walking speed I suppose that you can make
>an approximation of such a number.
>In our system AutoGait 3D a minimum of four cameras is recommended.
>2)
>In our tracking system an interpolation and extrapolation of the marker
>movements are carried out in order to approximate the next position of a
>marker. The extrapolation 'helps' the tracker to look in the right place
>when looking for the next position of a marker.
>3)
>In correctly measured files with appropriate setups the tracker is able
>to fully automatically join the tracks of the three - dimensional
>movements. There are, however, measurement situations when this is not
>possible ( e.g. complex movements including fu ll body rotations ).
>In such cases the operator can run the tracking manually or, preferably,
>run the tracking separately for different segments in the file. Since
>the tracking can be run both forwards and backwards in the file, a point
>with missing markers can be approached and tracked from two directions.
>
>Best Regards,
>
>H=E5kan Johnsson
>QUALISYS AB
>SWEDEN
>
>hakan.johnsson@qualisys.se
------------------------------------------------------------------------
[11] Email from Mr. Elm Strukol (Peak Performance):
This email was very useful because it provided specific numbers for the
15 point Helen Hayes marker set. As was mentioned in the commercial
software section above, their software will fill in missing gaps either
automatically or manually.
>Date: Wed, 28 Aug 1996 13:19:05 -0230
>From: Elm Sturkol
>To: joshua@engr.mun.ca
>Subject: Missing Marker Interpolation
>
>Thank you for your inquiry regarding missing marker interpolation issues. =
I
>referred the questions to one of our engineers and he responded as follows=
:
>
>Your first question is very broad. There are many different answers
>depending on how many markers are used and on what parts of the body they
>are placed. Not to mention the image acquisition rate of the camera system
>being used. Because of this, I'll limit my answer to a standard clinical
>gait situation using what is called in the industry, a 15 point Helen Haye=
s
>marker set. I don't know how familiar you with this "gait analysis" field,
>so if my descriptions are too basic or not in-depth enough let me know. Th=
is
>"marker set" consists of 15 spherical markers placed on the seven lower bo=
dy
>segments. Because the markers are placed on the left and right sides of th=
e
>body as well as the front and back, a two camera system is not able to dea=
l
>with the situation at all. Unless you use tricks with mirrors to allow the
>markers not directly visible to be seen. Also, three cameras are not enoug=
h.
>You'll need at least four cameras to even have a chance of tracking all 15
>markers. With four cameras, our software can fill in all the "gaps" caused
>by markers that are not in clear view of at least two cameras. It can also
>fill in the gaps in the 2-D image space data and create 3-D coordinates
>where the marker is not in view of any camera. However, this "filled in"
>data is not very accurate because with four cameras, the gaps are frequent
>and very large. Out of the 15 points there may be only six or seven of the=
m
>that can be reconstructed to 3-D without filling in any gaps. The others m=
ay
>have up to 50 percent of their image space data missing. Using five camera=
s,
>this percentage may decrease to 30 percent. And, using six cameras or more=
,
>this is reduced to about 10 percent or less in all but the most difficult
>cases. To accurately fill in the gaps, the number of cameras is increased =
so
>that each marker is clearly visible by at least two cameras all the time.
>
>The marker motion is predicted in several ways both in 2-D and 3-D.
>Extrapolation and interpolation algorithms based on human motion are used =
in
>different combinations to predict not only which path segment may join to
>another, but also, what direction the point is moving in the camera's imag=
e
>space so that it can be self correcting. The marker prediction is very
>complex and not easy to describe briefly.
>By setting several tracking parameters used by the software, the marker
>paths can be automatically connected or the user can connect them manually=
>The user is also allowed to break apart path fragments already connected a=
nd
>reconnect them differently. Our Motus software has a whole interface
>specifically devoted to this process.
>
>A lot has been done already using brute force mathematical extrapolation a=
nd
>interpolation along with more cameras to accurately fill in missing data.
>However, there is still plenty of room for improvement. Especially in the
>areas of using fewer cameras with software or hardware that is smart enoug=
h
>to fill in bigger and bigger gaps in the data.
>
>I hope this has been helpful.
>
>The above was written by Steve Risenhoover. Please contact us again if we
>can be of help.
>
>Mr. Elm Sturkol
>National Sales Representative
---------------------------------------------------------------------------=
-
[12] Email from John Greaves (Motion Analysis Corporation):
Dr. Greaves also acknowledges the problem of tracking several markers.
He mentions that their ExpertVision HiRES system does provide tools for
filling in the gaps in marker tracks. With short gaps of no more than a
few frames a software (Track Mender) looks at the continuity of the 3D
math segments (and a few other things) and automatically joins the
segments. Another method is one that involves human intervention to help
draw a curve over the gap.
>Date: Wed, 28 Aug 1996 14:44:10 -0230
>From: John Greaves
>To: Joshua Swamidas
>Cc: Rebecca Stine
>Subject: Re: Some Help please - gait analysis
>Hi Joshua,
>
> I'm John Greaves, founder and Sr. VP at Motion Analysis Corporation in
>California. Here are some ways that our ExpertVision HiRES video motion
>capture system deals with the missing marker problem you mentioned. First
>of all, gait is not too difficult to "track" (which means reduction to 3D
>coordinates from the raw camera data in our parlance) in the grand scheme
>of what people are doing with motion capture systems. What is more
>difficult is collecting data from two or three persons, each with 30 or 40
>markers to accurately represent their limb segment motions. Anyway,
>pathological gait we also have a lot of experience with (the OrthoTrak gai=
t
>analysis software has been on the market since about 1987) and pathologica=
l
>gait is more difficult to track than normal subjects, as you know.
>
>The best solution is to have redundant cameras for picking up more marker
>positions from more camera locations. A minimum of 4 cameras (two for eac=
h
>side) is required for markers placed around the body, although the folks a=
t
>Newington Children's hospital used 3 cameras for a long time by designing =
a
>special marker set with markers only on the front of the body. As you add
>more cameras, the more redundancy you have in your data collection
>mechanism and the fewer holes you will have in your data. We sell mostly =
6
>camera systems (roughly 3 cameras per side) for gait analysis now, and the
>data can be tracked with almost no "holes" in the 3D data set. BUT if
>there are holes, the mechanisms we have are:
>
>1-The Track Mender, which looks at the continuity of 3D path segments and
>automatically joins them if they are no more than a few frames apart and
>meet some equations about continuity in the XYZ components and their first
>derivatives. If you don't like the results, you can Undo it.
>
>2-The Add button in the XYZ time-series editor. Here you can position the
>mouse in a hole in each component (X, Y and Z) and place a point. The
>software splines across the gap with your input and you confirm each
>component separately. It turns out that the human is very good at seeing
>what "looks right" from this XYZ editor view--much better than what you ca=
n
>"see" from a stick figure or 3D spatial view of your data.
>
>Anyway, I hope this helps. We have spent years refining the tools for
>accurate data collection, tracking and editing and we were just at the
>Canadian Society of Biomechanics meeting, but that was on the opposite end
>of Canada from you.
>
>Sincerely,
>John Greaves, Ph.D.
>Sr. Vice-President
>Motion Analysis Corporation
-------------------------------------------------------------------------
[13] Email from Georger Roundanez (VICON):
>Date: Fri, 23 Aug 1996 12:37:08 -0230
>From: Georges Roudanez
>To: joshua@engr.mun.ca
>Subject: Re: mailto:georges@viconsys.com
>
>>1) How frequently do these holes (missing data) occur say, with a 2, 3=20
>> and 5 camera system and in studying pathalogical walking? The system
>> that will be purcahsed will be used maily wth CF kids and for research
>>
>Joshua,
>
>This is a difficult question to answer. The lost trajectories occur as a
>result of a marker not being seen by at least 2 cameras (in order to provi=
de
>xyz coordinates). The more cameras, the placement of those cameras and th=
e
>type of movement being measured determine the opportunity to avoid
>occlusions (not seen markers). A simple move with two cameras being
>constantly in view of the markers will most likely result in no occlusions=
>
>
>> purposes.
>>2) Do you have a way of predicting the marker motion?
>
>The way the Vicon operates it is possible to determine the anitcicipated
>marker path (trajectory).
>
>>3) Does the software automatically connect disjointed marker tracks or do=
=20
>> you let the operators handle the connecting of incomplete tracks.
>
>The answer is yes to both. You may choose to have the software (BodyBuild=
er
>- a special package that has editing and modelling features) automatically
>fill gaps, delete and fill noise spikes and interpolate or smooth the data=
-------------------------------------------------------------------------
[14] Email from Jermy Wise (Ariel):
Dr. Wise comments on the general problem of missing markers and makes=20
comments on how their software handles the problem. They seem to use
linear extrapolation of some kind and also some smoothing functions to
fill in the shorter gaps.
>Subject: Re: missing marker solution=20
> Date: Mon, 9 Sep 1996 18:25:52 -0230=20
> From: Jeremy Wise
> To: joshua@tera.engr.mun.ca
> CC: ariel1@ix.netcom.com
>
>Joshua-
>
>Dr Ariel has asked me to respond to your questions regarding handling of
>incomplete tracks with the APAS system. I'll do my best.
>
> >I understand that you develop state of the art vision systems and the
> >software. There are three questions I'd like to ask:
> >1) How frequently do these holes (missing data) occur say, with a 2, 3
> > and 5 camera system and in studying pathalogical walking? The system
> > that will be purcahsed will be used maily wth CF kids and for researc=
h
> > purposes.
>
>It's impossible to say how often this occurs because it depends entirely
>on the activity under consideration. Generally the more rotation about a
>vertical axis the more often a marker will become obscured. The more
>cameras one has the more likely atleast 2 cameras will see a marker for
>all frames. If 2 or more cameras "see" a marker then the software can
>calculate the 3D coordinates.
>
> >2) Do you have a way of predicting the marker motion?
>
>The program predicts marker motion using a rather complex search
>algorithm. The location the software starts searching at is an
>extrapolation of the previous 2 frames or previous frame, a software
>selectable switch. The reason a simple linear extrapolation is used is
>because higher order extrapolations are more error prone.
>
> >3) Does the software automatically connect disjointed marker tracks or d=
o
> > you let the operators handle the connecting of incomplete tracks.
>
>I'm not sure if you are referring to tracks in 2D or reconstructed 3D.
>If you are talking about the plane of the video, the software can be set
>to skip and flag as MISSING any markers which the software can't find, or
>the software can be configured to pause & let the user estimate marker
>location based on his/her observation of the whole image. If you are
>talking about 3D, then the program flags any point that cannot be seen by
>atleast 2 cameras as MISSING and will be considered as missing RAW data.
>However, when it is smoothed, and incomplete sections of data will be
>interpolated then smoothed with the selected algorithm. What you are
>asking the software to do is to make data where it can't be seen be
>atleast 2 cameras. This is risky business.
>
>I hope this helps,
> Dr. Jeremy Wise
> Dir R&D
-------------------------------------------------------------------------
[15] Email from Mandy Sketch:
Date: Wed, 28 Aug 1996 12:26:31 -0230
From: Mandy Sketch
To: joshua@engr.mun.ca
Subject: Some Help please - gait anlaysis -Reply
I am out of the office until September 2nd.
If you need to contact me urgently please either call or fax my office (tel=
:
44 1865 843658, fax:
44 1865 843965) and ask for a message to be passed on. Alternatively I will
respond to your
message on my return.
Mandy Sketch
Publishing Editor
----------------------------------------------------------------------
*******************End of Email Replies*******************************
Thanks again,
Joshua Swamidas
--
joshua@tera.engr.mun.ca
Box 30, Faculty of Engineering and Applied Science,
Memorial University of Newfoundland
St. John's, NF A1B-3X5
(709) 737-8809