mviceconti63

07-11-2004, 05:42 PM

Dear Colleagues:

What follows may be one of the longest

messages ever posted on BIOMCH-L. Thus, before I

challenge your patience, let me summarise the

content of this essay. Some time ago I posted a

question on which criteria we should use when

reviewing papers that draw clinically relevant

conclusions using the results of numerical

models. My original question is reported below

in the last section, where also all the replies I

got are reported.

On the basis of the replies I collected, plus

some further reading, plus some useful

discussions I had during the ESB meeting in Den

Bosch, I wrote a brief essay entitled 'The use of

finite element analysis to produce

recommendations for the clinical practice', that

you find here below. Such essay can be

summarised as follow. I recommend four levels of

validation for a numerical model, and I suggest

for each type of journal which of them should be

considered mandatory or publication:

- Stage #1: verification and proper parameters

identification. This stage only ensures that the

model is numerically correct. This level should

be mandatory for any published paper.

- Stage #2: sensitivity analysis and

inter-subject variability. This level of

validation should be present in papers published

in applied biomechanics journals.

- Stage #3: validation against in vitro

controlled experiments: This level of validation

should be present in papers published in those

journals aiming at the space between biomechanics

research and clinical research.

- Stage #4: risk-benefit analysis and

retrospective clinical studies: This level of

validation should be present in papers published

in clinical research journals and should be

available before clinical trials are started.

This stage requires that the whole research line

is mature enough that we can identify a clear

clinical question to which we can answer using

our numerical model.

- Stage #5: prospective studies: before any

technology can be used routinely in the clinical

practice its validity should be proved

conclusively by a prospective trial.

I hope this draft will create a lively discussion

so that we can translate this draft essay into a

consensus paper on this important topic.

Regards

Marco Viceconti

----

Essay: The use of finite element analysis to

produce recommendations for the clinical practice

What is finite element model?

There was consensus among those who responded to

my original query that a finite element model is

conceptually equivalent to a theory. Thus,

presenting predictions of a non-validated finite

element model is very much like report personal

opinions. Any finite element model should be

verified and validated before we can consider its

predictions of any scientific value.

Verification and validation

We usually indicate the term verification to

indicate the process that ensures that a

numerical model accurately predicts the results

of the theoretical model it is based on. In

other words we verify a model by assessing its

numerical accuracy. On the contrary, the term

validation indicated the process that ensures

that our numerical model is accurately predicting

the physical phenomenon it was designed to

replicate. Thus, we validate a model by assessing

its accuracy tout court, i.e. the accuracy with

which it predicts the reality.

However, as for a theory, also for a numerical

model it is in general impossible to prove its

validity in a complete way. This is why the

formation of a scientific truth, which is a fully

validated theory, is a slow and sedimentary

process. In science we frequently use the

expression 'it is generally believed' to indicate

this consensus process.

It is evident that while the verification is a

process fully internal to the work of the single

scientist, and thus we can consider it mandatory

for publication, the validation process involves

the whole scientific community and in this sense

there is a sort of relativism in it.

Validation of numerical models to be used for clinical purposes

In my opinion the biggest difference between

physicists and engineers is that the physicists

can wait. Newton kept it universal gravitation

theory 30 years in a drawer, until he was able to

clarify all mathematical implications of his

work. Obviously, Newton was not in the need to

use that theory to do something practical, such

as building a house or an engine. Clinical

medicine faces a similar problem. The medical

professional must do something when is faced with

a suffering patient, even if he or she is not

100% sure that what is planning to do will truly

help the patient.

In contemporary medicine one way to formalise

this decisional process is the so-called

risk-benefit analysis. I propose that we should

start to consider this instrument for the

evaluation of the degree of validation of a

numerical model to be used in the clinical

practice. We accept the fact that no numerical

model can be totally validated. Thus, there is

always a risk associated to the use of the

predictions made by this model in the clinical

practice. So the first question we should try to

answer is: what is the level of risk to which we

are exposing the patient when we include the

results of the numerical model into the clinical

process? This is not an easy process. Firstly,

the more the model has been validated by means of

controlled experiments and/or clinical efficacy

studies, the lower is the risk that it provides

results that a re completely wrong. But the

other question we need to ask ourselves is: what

is the risk for the patient if the model is wrong

(within the limits of its validation)?

Once we have an idea of the risk associated with

the uncertainty of our model results, we have to

balance it with the benefits that the use of this

numerical model would bring to the patient, in

order to decide whether this is worth or not.

Obviously the researcher alone cannot conduct the

risk-benefit analysis; the whole

research-clinical team that is considering using

some numerical results into the clinical practice

should be responsible for it.

So, what we do on our journals?

The original motivation of this work was related

to the doubts we have to accept certain papers

entirely or mostly based on numerical models in

journals with a clinical target. On the basis of

the comments I received to my original query, and

because of the line of thoughts I reported above,

which derived form these comments, my conclusions

are:

- zero tolerance for verification: in no case we

should allow the publication of a paper where the

numerical model in use has not been fully

verified. For linear models we need to have

information of the convergence of the mesh

refinement, or even better on any post-hoc

indicator related to mesh refinement. If an

iterative solver is used, this should be made

clear and the convergence tolerance used should

be reported. When the model is non-linear the

verification should be specific for the type of

non-linearity present. If you are including

non-linear frictional contact it is appropriate

to report the peak penetration or equivalent

tolerance, and so on. Also, convergence

tolerance for the Newton-Raphson or other similar

schemes needs to be reported. Please notice that

the proper way to report these convergence

details is to write the convergence variable

(i.e. the force), the norm on which we check the

convergence, and the tolerance itself.

- Proper parameter identification: another

mandatory required for any numerical paper in

order to be published is the proper

identification of its parameters. Parameters

should be associated to physical/physiological

quantities and be independent by the time or the

repetition in the experiment used to identify

them. When one pretends to validate a model by

showing its ability to fit experimental results

over an interval of a given parameter, i.e. time,

we should check that the number of independent

parameters in the model is significantly lower

than the 'order' of the of the experimental

event. In this stage it may be reasonable to

investigate the sensitivity of a few critical

parameters, and eventually perform a 'design of

experiment' study in order to explore the

sensitivity of the solution to the uncertainty of

the parameters.

When a paper presents these features it should be

published in those journals that are interested

to the theoretical speculation, and the deductive

reasoning. In practice with a fully verified

model, we put forward a theory, and we can use

the numerical model to explore all its

implications. At this stage, when deductions are

made with respect to the clinical practice, it

should be made clear that these are theoretical

speculations, in need of further support.

- Sensitivity analysis: the second vital step in

the verification and validation process should be

the sensitivity analysis. Recent statistical

finite element analysis tools made this process

much simpler than in the past. Sensitivity

studies are essential in biomedical research,

where frequently the few available experimental

measurements are affected by large uncertainties.

Rather than 'using' these uncertainty to say that

the predictions of our model fall with the range

of the predicted experiment, it is necessary to

do a full sensitivity study that shows how this

uncertainty impacts on our deductions.

- Inter-subject variability: this is another type

of sensitivity analysis. We know that human been

are very different one to each other, from

anatomical and functional point of views. On the

other hand we usually create our models targeting

a specific subject or an idealistic average

subject. Thus, in principle, we should assess

how much our conclusions are affected by

inter-subject variability. It is very difficult

to provide strict guidelines here. In some cases

the inter-subject variability can be

parameterised, and thus included in the

sensitivity analysis. In these cases I would

consider this mandatory. In other cases you

simply need to build a new model for each new

subject you intend to describe, and this prevents

you from doing a systematic exploration on the

effects of inter-subject variability. In any case

the authors should consider this issue somehow.

One option may be to investigate a few subjects

that are representative of the extremes of

variability of the population of interest. This

is very similar to a design of experiments

approach, and it gives you a gross indication of

the level of variability you may expect in your

results.

At this stage of the validation process, we have

theoretical model that is robustly linked to the

experiments that are used to identify it. I

suggest that this second level of validation

should be mandatory to publish in those journals

aiming to applied biomechanics research.

- Validation against in vitro controlled

experiments: this is the first step in the true

validation process. It is usually very difficult

to perform them, and when you succeed they

usually show you a lot of unexpected weaknesses

in your model. Thus, I consider this a highly

valuable form of scientific result. One word

about how to report the results of this

comparison between numerical and experimental

results. Most authors use a linear regression

between measured and predicted value, and report

the regression parameters and coefficients.

However, I propose that we should always ask also

to report on the residuals of such regression.

One way I like is provide a root mean square

error as an indication of the average residual,

and the peak error, as indication of the maximum

residual.

My suggestion is that we should consider this

third level mandatory for all those journals

aiming at the space between biomechanics research

and clinical research.

- Risk-benefit analysis: as I wrote at the

beginning of this report, I believe that before

we can use results obtained from numerical models

in the clinical practice we need to report the

results of a risk-benefit analysis. To do so we

need a fully verified model on which a complete

sensitivity study and an in vitro validation

study has been conducted. This would provide us a

quantitative basis to estimate the risk

associated with the use of the model. Of course

these studies should be conducted in

collaboration with experts in clinical research,

which should have the necessary knowledge to

estimate the expected benefits.

- Retrospective studies: along the same lines,

but with greater level of confidence, there are

the retrospective studies. If you can use your

model to answer a clinical question over a

population for which the answer to this question

is known, you may get a good insight on the

validity of your model. In many cases the best

way to report these results is in term of

specificity and sensitivity, using the R.O.C.

curves. A key issue here is that the clinical

question must accept a yes-no answer.

When this type of validation studies is available

my suggestion is that we should allow the

publication of numerical studies also on the most

clinical journals. The clinical audience would

have in general clear enough the difference

between a retrospective and a prospective study.

Numerical models at this stage of validation

could also, in my opinion, be used in controlled

clinical trials.

- Prospective studies: as for any other method,

whenever possible the conclusive word on the

clinical use of a numerical model comes from

prospective clinical trials.

Code reliability

Some among those who replied to my original query

pointed out the issue of reliability of

commercial numerical analysis software, which is

a black box that may hide some problems. Some

others reversed the issue warning about the

danger of using software developed in-house.

After some thinking I decided that this has to do

with the general quality control we should apply

to all our laboratory instruments, including

software. Nobody is reporting in a scientific

paper when he calibrated last time the load cell

used in the study. This is left to our

consciences. For sure in our group we shall

start soon to develop a quality control system

for numerical analyses, very much like the one we

have already in place for the experimental

biomechanics unit. We expect to rely a lot on the

NAFEMS independent benchmarks for finite element

analysis codes.

----------------------------------------------------------------------------------------------------

Original posting:

I serve as a reviewer for a few biomechanics or orthopaedics

journals, some of which have a clear clinical perspective. In

particular I frequently revise papers where finite element models are

used as the main research tool.

These models are becoming more and more effective and powerful, and

it is not rare to find papers where the authors, on the basis on the

results obtained with the model, draw conclusions that may have a

clinical relevance, i.e. affect clinical decisions.

The problem I have is methodological: are we allowed to draw

clinically relevant conclusions from the predictions obtained by a

numerical model? Or, more appropriately, what are the conditions a

numerical model must fulfill in order to considered so reliable that

we can reasonably use it to draw clinically relevant conclusions?

In order to foster the discussion let me bring in my two cents.

In my understanding a numerical model is a particular instance of a

theoretical model. Once we are sure that the theoretical model has

been solved with sufficient numerical accuracy (and this is in

general possible with post-hoc indicators) what remains to be proved

is the adequacy of the theoretical model.

A theoretical model (a theory) in principle can be assumed to be true

(in the sense of a scientific truth, i.e. as far as we, as a

community we know, and within the limits of validity of the theory),

if with this model we can predict the outcomes of independent

experiments (independent in the sense that they are not those

measurements that were used to identify the model) and/or if starting

from the model we can deduct derivative conditions that are proved to

be true.

How do we translate these general rules in the specific of our

research domain, biomechanics? Can we say that once a model predicts

with sufficient accuracy the results of an in vitro experiment, we

can consider it valid and use it to draw clinically relevant

conclusions? Or is it sufficient to prove that the model is

numerically accurate and all the model assumptions and parameters are

well supported by experimental observations?

As usual I am ready to post a summery of the comments I shall receive.

----------------------------------------------------------------------------------------------------

Replies:

From: "Daniel P. Nicolella"

Marco,

You have raised an excellent point. Model simulation verification and

validation (V&V - essentially what you have eloquently described) is an

ongoing research topic within several disciplines. My colleague, Ben

Thacker, is actively involved in this area and has published several papers

on the topic of V&V. I have attached one such paper for your information

that should give you some background on the current thinking in this area.

I personally believe this is an important issue that should be addressed

within the biomechanics community and would like to see a consensus

developed on this topic.

Best Regards,

Dan Nicolella

----------------------------------------------------------------------------------------------------

From: "Mahar, Andrew"

Hi Marco,

You raise a very important issue in the world of orthopedic biomechanics.

We are conducting in-vitro experiments and numerical tests in parallel

across a variety of applications (spine/trauma) to optimize the numerical

theories, with the understanding that even in-vitro experiments can't

duplicate the in-vivo situation. At this point, the surgeons I work with

are hesitant to accept surgical interpretation based on the numerical data

and the engineers are hesitant to provide it. At this point, for our

research group, the greatest value numerical tests have are to better

understand implant behavior under a wide variety of testing

scenarios/positions/materials. These data may be used to better understand

the clinical situation, but that is after the fact. Anyway, that is where

our group stands on the issue.

I look forward to your summary.

Andrew

----------------------------------------------------------------------------------------------------

From: "Warren G. Darling"

Dear Dr. Viceconti, I would say that a model must

predict results of in vivo experiments (in a

human or in a close animal model) before it

should be used to draw any clinically relevant

conclusions. In vitro experiments often produce

quite different results from in vivo experiments.

I definitely do not think it is sufficient to

simply show a model is numerically accurate and

that model assumptions and parameters are well

supported by experimental observations. A

numerical model with valid assumptions and

parameters can be used as a guide to design

experiments to test clinically relevant issues,

but should not guide treatment until its

predictive value has been tested.

Sincerely,

Warren Darling

----------------------------------------------------------------------------------------------------

From: Bjorn.Olsen@MEMcenter.unibe.ch

Dear Dr. Viceconti,

Like yourself, I am often faced with the same

questions regarding numerical simulation; from

both surgeons and editors.

Like many on this list I have used various

numerical models in biomechanics, and I find it

useful to answer specific research questions.

However, in my opinion numerical modeling is

*just* a method. There is nothing special about

numerical simulation, and the manner in which

this particular method is employed must be

scrutinized like any other (eg. experimental)

method. Here I think the biomechanics community

has failed to some degree. When compared to most

other engineering disciplines, numerical

simulation is in its infancy when employed in

bio-engineering. The range of published

(modeling) material range from absolutely cutting

edge, to the - to say the least - not so great.

Very often, in the papers I review, I find a lot

of *very* basic mistakes (boundary conditions

applied incorrectly, linear analysis for large

deformation, material property/units wrong,

etc.). Furthermore, often the models do not

correspond to the problem the authors are trying

to solve.

The rise of numerical simulation over the last 10

years has seen a drop in experimental

biomechanics (which I personally find a bit

disturbing). This is most likely also related to

cost. Simulation is, most often, a cheaper and

quicker solution. Whether or not experimental

methods should always be used depends on the

specific problem at hand. As all of you would

know, quite often simulation is the only method

available to study certain problems in our field.

The engineer should always review the existing

methods (experimental/numerical) and chose the

one which is best suited to the answer the

research question.

It is mistaken to think that numerical analysis

(like FEA) is a simple, out-of-the-box method. It

is not. This is an engineering discipline, and

great care should be taken when using it. My

message is: We, as authors and reviewers, must

ensure that these methods are used correctly.

This is the only way it is possible to maintain

and transfer the credibility of numerical

simulation.

With kind regards,

Björn Olsen

----------------------------------------------------------------------------------------------------

From: "Anders Eriksson"

Dear Marco,

I think you have raised a very important issue, which deserves

an extensive discussion in the ongoing quick

development of numerical modelling and

simulations in biomechanics. My viewpoint, being

primarily in computational mechanics but with

several applications to biomechanical problems,

is that there is often/always a too high

credibility given to numerical, e.g. FEM,

calculations -- "the experiments were verified by

FEM calculations" -- without stating anything

about the used elements: nuermical, geometric,

kinematic and kinetic assumptions.

As you say, numerical simulations are always

based on some underlying assumptions and theory:

the 'model'. They can therefore, at best, be as

good as these theories and assumptions. But, and

this is important, they also have their own sets

of assumptions, approximations and numerical

error sources. These must be kept under strict

control, for any conclusions of practical

importance to be drawn. This is as well known by

people within the computational research areas as

is the fact that physical theories are only the

best possible explanation, within a set of

assumptions, is to the physics people. The

general science philosophical conclusion is

probably that we always have a tendency to

consider everyone else's scientific area as

simple and well-defined, whereas all complexities

occur in our own area.

Regarding your underlying question on the

applicability of numerical simulations, I am,

however, not only pessimistic. But, care is

needed. I think that the basic requirement on a

numerical study, aiming to have application

relevance, is a careful documentation (by you or

someone reliable) of the underlying assumption

and theories used in the modelling. And, this is

perhaps even more the case when black box

software is used. In particular, this is true

when the general FEM software is used for

biological problems, which are very often far off

from the parametric ranges where the basic

numerics have been developed and verified. As

strict theories seldom exist for general

problems, testing of limited setups should always

be documented, as a basis for the larger models.

It is also, as you mention, important to state

the problem in fundamental principles, and not

just base parameters and assumptions on

regressions from very similar experiments. This

done, I think that conclusions drawn from

numerical modelling and simulations can be

helpful in the clinical understanding.

In the same context, I would also like to point

to a common type of problem statement in

engineering, and maybe as useful in other

scientific branches, namely the 'inverse problem'

solving, or perhaps a very systematic 'what-if'

research. This can be used for understanding many

types of behaviour qualitatively and (at least

relatively) quantitatively. Assuming that you

have good knowledge of the basic building blocks

inccluded in your simulation package, you can

very easily vary assumptions and parameters in

the simulation model, to see how results are

affected by these assumptions, thereby allowing

some conclusions to be drawn, when these

simulations are compared to experiments (or

general knowledge of behaviour). This is, but

only when you have the underlying knowledge about

the modelling assumptions, something very

different and much more powerful than regression

of parameters in a fixed basic model, as it

allows a much wider space of assumptions to be

tested. Good basic knowledge and a critical view

on the interpretation of the results are the key

ingredients in this method, and the possibility

for independent critical assessment of

assumptions the strength.

As I said, I think that this area needs an

extensive discussion to avoid that many incorrect

clinical conclusions are drawn, based on bad

numerical models, using irrelevant theories and

incompletely known parameters.

Best regards

Anders Eriksson

----------------------------------------------------------------------------------------------------

From: "linping zhao"

Dr. Venceconti:

You brought a very important topic for

discussing. I am expecting such discussion for a

long time. It seems it's time now. I'd like to

see various responses from people with either

engineering or clinical background or from other

point of views.

Here are more concerns regarding to the validation of a FE model:

If a FE model is validated with a mechanical

model or even a cadaver test, is it clinical

relevant? In what sense and what degree?

When we build up a FE model based upon in vivo

CT/MRI data, is it possible to validate a FE

model in vivo? If so, how? If not, how can we say

the FE model is validated?

If the prediction of a FE model is in agreement

with clinical observation, the model can be

considered as validated in some sense. Is this

statement true? If so, in what sense the model is

validated?

Looking forward to reading more.

Linping Zhao, Ph.D.

Plastic and CranioMaxillofacial Surgery

Shriners Hospitals for Children, Chicago

----------------------------------------------------------------------------------------------------

From: Apache

Here is my comment: I think it would be too rigorous to require

every modeller to do the whole job at once, i.e. to find a parameter

fit to an experimental outcome (identifying all unknown parameters

of the numerical model), use the predictive power of the model by parameter

variation and perform an adequate testing experiment.

Now, here, by trying to define what "adequate" means, we find that there

are some prerequisites to be fulfilled in order for a model to increase

the probability of its validity. The probability can never really be 1 (valid)

but 0 (invalid).

1. The parameters of the theoretical model must be independent of time

or experiment (i.e. they may in no way be fitted along time or across

experiments!) AND can be mapped to physical / physiological quantities

which can at least in principle be determined experimentally.

2. The more parameters the model has (in the fitting process) the lower is

its value for identifying fundamental, underlying principles.

In my opinion this statement should be equivalent to saying that for every

structure or phenomenon that can be identified in nature you only need

a finite number of essential parameters describing it. It's just a personal

belief that this number is always small enough to get the chance

of understanding. It is the art of research to identify these essential

parameters.

In FEM models there are heaps of parameters that additionally may be

hidden to the normal user. I.e. even requiring tests of sensitivity

with respect to these parameters (physical or numerical) seems to be

an endless job. I think providers of these software packages should be

urged much more by the scientific community to pass fundamental physical

tests e.g. such as conservation of energy or of moment of momentum in simple

test models that are claimed to be conservative or free of external

forces / torques.

3. More generally: The lower the ratio between the number of fitted parameters

and the number of independent experimental findings that may be used for the

fitting procedure the better the model meaning that the probability

of the model to have predictive power is higher.

The lower the number of model parameters the higher its potential for

gaining insight. Thus, the effort of researches to REDUCE the number of

parameters should be favoured by referees over the apparent success

to provide a perfect multi-parameter fit. Here, it is very useful to

have an estimation of the accuracy of the EXPERIMENTAL findings. Simpler

models (and there potential of gaining deep insight) are discredited

by an inappropriate trust into measured data.

4. Additionally to 2. good referees ask the researches for the sensitivity of

their models with respect to predictions. Here, 'sensitivity' means that

the FITTED parameters should be varied and the deviation of the results

with respect to the a priori experiments should be shown. On the other hand,

a good referee also identifies very few but CRITICAL parameters but does

not make the researcher do a complete new study.

> How do we translate these general rules in the specific of our

> research domain, biomechanics? Can we say that once a model predicts

> with sufficient accuracy the results of an in vitro experiment, we

> can consider it valid and use it to draw clinically relevant

> conclusions?

I tried to speak out explicitly some criteria (1. to 4. above) that I

find essential when dealing with the interplay of theory and experiment.

I personally 'love' extremely reduced models more and more. My very personal

view is that an FEM-model can be considered a little bit more valid if it has

passed one predictive test. Full validity in biomechanics can not be shown

within one study as long as the MINIMAL number of parameters that might be

necessary for predicting underlying processes is not at least estimated.

If one compares to the many years that e.g. the

spring-mass model for locomotion

was and still is tested and developed around the world one can get a feeling

for the long way to high validity of a model. I personally would not draw

clinically relevant conclusions based on an FEM-model before knowing many

things about the set of parameters, sensitivities etc. (see above).

> Or is it sufficient to prove that the model is

> numerically accurate and all the model assumptions and parameters are

> well supported by experimental observations?

As a summary: I think, validity can only be given in terms of probability.

Unfortunately, one can only prove a model to be INVALID if it clearly fails

with a predection. Otherwise, we can only check accuracies, minimize the number

of parameters and do as many tests (some mentioned in 1. to 4.) as possible to

enhance the probability of model validity.

Cheers,

Michael.

--

Michael Gunther

----------------------------------------------------------------------------------------------------

From: "Ben Thacker"

Marco,

The points you raise are exactly those that must

be addressed if numerical simulation is

to play an active role in design, i.e., produce

credible predictions with quantified

accuracy. Requiring this of numerical models

means the stakes have been raised -- over the

next few years it will be interesting (and

exciting) to see if the modeling community can

address these questions to the satisfaction of

the decision makers and customers. It will

not be easy.

I'd like to refer you to our ASME V&V Standards Committee website at

www.usacm.org/vnvcsm

There are several reports there, etc. that may be

of interest to you. You can also

subscribe to our email list from the web site.

I hope this helps.

--Ben

----------------------------------------------------------------------------------------------------

From: Ton van den Bogert

Dear Marco,

This is a great question, and very important.

I had hoped for some public replies, maybe when

you post your summary you can ask if people want

to post additional public responses.

>Or, more appropriately, what are the conditions a

>numerical model must fulfill in order to considered so reliable that

>we can reasonably use it to draw clinically relevant conclusions?

It depends, I suppose. If a model depends only

on known laws of physics, and not on assumptions,

one should have no problem with clinical

conclusions. But usually there are assumptions

and/or simplifications, and this is where your

question applies. Here are my ideas on how to

alleviate such concerns.

First, it is often not known to what extent the

results depend on assumptions. Usually it is

possible to do sensitivity analysis or

perturbation analysis. Simply change the assumption and see how much

difference it makes. If there is any question on the influence of

assumptions, such a sensitivity analysis should always be attempted.

Second, computatonal studies often use just one model, and if you think

of it, this is absurd. You would never use just one human subject, one

animal, or one cadaver specimen. There is biological variation among

humans and when a model represents a sample of N=1, conclusions may be

quite unreliable. Why not make a group of models to reflect variability

within the human population, do the model study

with all of them, and analyze the results

statistically? This also takes care of the

sensitivity analysis to some extent. If a result

is obtained with one model, and not with another,

based on a different human subject, the result is

not reproducible and should not be reported. If

you have N=1, how do you know?

Computers get faster all the time. N=1 with no sensitivity analysis may

have been OK ten years ago. Now, computers are 20 times faster, so we

can do N=10 and additionally use one of those models to quantify ten

sensitivities. But instead, we have built models that take longer and

longer to compute and are supposedly more realistic. But still we often

do N=1 and no sensitivity analysis. I think this is not the right way

to proceed.

This N=1 practice may be more of a problem in my

own field of movement dynamics than in tissue

mechanics. In movement, there is nonlinear

dynamics with potential for self-regulating

mechanisms. See Wright et al (Clin Biomech,

1998) for a nice example. The question was the

effect of shoe hardness on impact forces in

running. Ten models were used. In some, the

harder shoe increased the impact, in others it

decreased the impact. This was unexpected and

forced us to examine the results carefully. A

self-regulating mechanism was found: a harder

shoe causes the impact force to rise faster,

which causes earlier knee flexion, which then

slows the further rise in impact force. Whether

the peak force ends up being higher or lower than

with the other shoe, depends on subtle

differences between the models. This is an

example where N=1 would have been a lottery,

rather than a scientific study. With ten models,

the statistically correct conclusion was

obtained: no significant effect, but also there

was an understanding of the underlying mechanism.

There is also the problem of quality control. If

there is a bug in your code, multiple models and

sensitivity analyses will all be influenced in

the same way. You may think that results are

consistent, and therefore correct. This is where

commercial software has the advantage of a wide

user base, and a better likelihood that errors

have already been found. For those of us who

want to go beyond commercial tools and do our own

programming, sound software engineering practices

and a healthy dose of self-criticism are

essential.

Finally, models are mainly useful in basic research. We can use a model

to generate and test hypotheses, but before

giving clinical recommendations, I think that a

clinical trial should be done to see if

conclusions are also valid in the real world.

This can be expensive, but it is nevertheless the

standard procedure for drugs, after doing animal

studies. For the same reason it may be a good

idea in biomechanics also, after numerical

studies have been done as a first test of

efficacy and safety.

It seems that in biomechanics we are not always

as rigorous as in other disciplines, perhaps

because biomechanics usually does not deal with

life-threatening problems. By the way, the

problem is not limited to numerical models. In

vitro models are heavily used in orthopaedic

biomechanics, to study a joint or bone in isolation. Results can depend

very much on the mechanical boundary conditions, and this is not always

sufficiently recognized.

Ton van den Bogert

--

--------------------------------------------------

MARCO VICECONTI, PhD (viceconti@tecno.ior.it)

Laboratorio di Tecnologia Medica tel. 39-051-6366865

Istituti Ortopedici Rizzoli fax. 39-051-6366863

via di barbiano 1/10, 40136 - Bologna, Italy

Tiger! Tiger! Burning bright in the forest of the night,

what immortal hand or eye could frame thy fearful symmetry?

--------------------------------------------------

Opinions expressed here do not necessarily reflect those of my employer

-----------------------------------------------------------------

To unsubscribe send SIGNOFF BIOMCH-L to LISTSERV@nic.surfnet.nl

For information and archives: http://isb.ri.ccf.org/biomch-l

Please consider posting your message to the Biomch-L Web-based

Discussion Forum: http://movement-analysis.com/biomch_l

-----------------------------------------------------------------

What follows may be one of the longest

messages ever posted on BIOMCH-L. Thus, before I

challenge your patience, let me summarise the

content of this essay. Some time ago I posted a

question on which criteria we should use when

reviewing papers that draw clinically relevant

conclusions using the results of numerical

models. My original question is reported below

in the last section, where also all the replies I

got are reported.

On the basis of the replies I collected, plus

some further reading, plus some useful

discussions I had during the ESB meeting in Den

Bosch, I wrote a brief essay entitled 'The use of

finite element analysis to produce

recommendations for the clinical practice', that

you find here below. Such essay can be

summarised as follow. I recommend four levels of

validation for a numerical model, and I suggest

for each type of journal which of them should be

considered mandatory or publication:

- Stage #1: verification and proper parameters

identification. This stage only ensures that the

model is numerically correct. This level should

be mandatory for any published paper.

- Stage #2: sensitivity analysis and

inter-subject variability. This level of

validation should be present in papers published

in applied biomechanics journals.

- Stage #3: validation against in vitro

controlled experiments: This level of validation

should be present in papers published in those

journals aiming at the space between biomechanics

research and clinical research.

- Stage #4: risk-benefit analysis and

retrospective clinical studies: This level of

validation should be present in papers published

in clinical research journals and should be

available before clinical trials are started.

This stage requires that the whole research line

is mature enough that we can identify a clear

clinical question to which we can answer using

our numerical model.

- Stage #5: prospective studies: before any

technology can be used routinely in the clinical

practice its validity should be proved

conclusively by a prospective trial.

I hope this draft will create a lively discussion

so that we can translate this draft essay into a

consensus paper on this important topic.

Regards

Marco Viceconti

----

Essay: The use of finite element analysis to

produce recommendations for the clinical practice

What is finite element model?

There was consensus among those who responded to

my original query that a finite element model is

conceptually equivalent to a theory. Thus,

presenting predictions of a non-validated finite

element model is very much like report personal

opinions. Any finite element model should be

verified and validated before we can consider its

predictions of any scientific value.

Verification and validation

We usually indicate the term verification to

indicate the process that ensures that a

numerical model accurately predicts the results

of the theoretical model it is based on. In

other words we verify a model by assessing its

numerical accuracy. On the contrary, the term

validation indicated the process that ensures

that our numerical model is accurately predicting

the physical phenomenon it was designed to

replicate. Thus, we validate a model by assessing

its accuracy tout court, i.e. the accuracy with

which it predicts the reality.

However, as for a theory, also for a numerical

model it is in general impossible to prove its

validity in a complete way. This is why the

formation of a scientific truth, which is a fully

validated theory, is a slow and sedimentary

process. In science we frequently use the

expression 'it is generally believed' to indicate

this consensus process.

It is evident that while the verification is a

process fully internal to the work of the single

scientist, and thus we can consider it mandatory

for publication, the validation process involves

the whole scientific community and in this sense

there is a sort of relativism in it.

Validation of numerical models to be used for clinical purposes

In my opinion the biggest difference between

physicists and engineers is that the physicists

can wait. Newton kept it universal gravitation

theory 30 years in a drawer, until he was able to

clarify all mathematical implications of his

work. Obviously, Newton was not in the need to

use that theory to do something practical, such

as building a house or an engine. Clinical

medicine faces a similar problem. The medical

professional must do something when is faced with

a suffering patient, even if he or she is not

100% sure that what is planning to do will truly

help the patient.

In contemporary medicine one way to formalise

this decisional process is the so-called

risk-benefit analysis. I propose that we should

start to consider this instrument for the

evaluation of the degree of validation of a

numerical model to be used in the clinical

practice. We accept the fact that no numerical

model can be totally validated. Thus, there is

always a risk associated to the use of the

predictions made by this model in the clinical

practice. So the first question we should try to

answer is: what is the level of risk to which we

are exposing the patient when we include the

results of the numerical model into the clinical

process? This is not an easy process. Firstly,

the more the model has been validated by means of

controlled experiments and/or clinical efficacy

studies, the lower is the risk that it provides

results that a re completely wrong. But the

other question we need to ask ourselves is: what

is the risk for the patient if the model is wrong

(within the limits of its validation)?

Once we have an idea of the risk associated with

the uncertainty of our model results, we have to

balance it with the benefits that the use of this

numerical model would bring to the patient, in

order to decide whether this is worth or not.

Obviously the researcher alone cannot conduct the

risk-benefit analysis; the whole

research-clinical team that is considering using

some numerical results into the clinical practice

should be responsible for it.

So, what we do on our journals?

The original motivation of this work was related

to the doubts we have to accept certain papers

entirely or mostly based on numerical models in

journals with a clinical target. On the basis of

the comments I received to my original query, and

because of the line of thoughts I reported above,

which derived form these comments, my conclusions

are:

- zero tolerance for verification: in no case we

should allow the publication of a paper where the

numerical model in use has not been fully

verified. For linear models we need to have

information of the convergence of the mesh

refinement, or even better on any post-hoc

indicator related to mesh refinement. If an

iterative solver is used, this should be made

clear and the convergence tolerance used should

be reported. When the model is non-linear the

verification should be specific for the type of

non-linearity present. If you are including

non-linear frictional contact it is appropriate

to report the peak penetration or equivalent

tolerance, and so on. Also, convergence

tolerance for the Newton-Raphson or other similar

schemes needs to be reported. Please notice that

the proper way to report these convergence

details is to write the convergence variable

(i.e. the force), the norm on which we check the

convergence, and the tolerance itself.

- Proper parameter identification: another

mandatory required for any numerical paper in

order to be published is the proper

identification of its parameters. Parameters

should be associated to physical/physiological

quantities and be independent by the time or the

repetition in the experiment used to identify

them. When one pretends to validate a model by

showing its ability to fit experimental results

over an interval of a given parameter, i.e. time,

we should check that the number of independent

parameters in the model is significantly lower

than the 'order' of the of the experimental

event. In this stage it may be reasonable to

investigate the sensitivity of a few critical

parameters, and eventually perform a 'design of

experiment' study in order to explore the

sensitivity of the solution to the uncertainty of

the parameters.

When a paper presents these features it should be

published in those journals that are interested

to the theoretical speculation, and the deductive

reasoning. In practice with a fully verified

model, we put forward a theory, and we can use

the numerical model to explore all its

implications. At this stage, when deductions are

made with respect to the clinical practice, it

should be made clear that these are theoretical

speculations, in need of further support.

- Sensitivity analysis: the second vital step in

the verification and validation process should be

the sensitivity analysis. Recent statistical

finite element analysis tools made this process

much simpler than in the past. Sensitivity

studies are essential in biomedical research,

where frequently the few available experimental

measurements are affected by large uncertainties.

Rather than 'using' these uncertainty to say that

the predictions of our model fall with the range

of the predicted experiment, it is necessary to

do a full sensitivity study that shows how this

uncertainty impacts on our deductions.

- Inter-subject variability: this is another type

of sensitivity analysis. We know that human been

are very different one to each other, from

anatomical and functional point of views. On the

other hand we usually create our models targeting

a specific subject or an idealistic average

subject. Thus, in principle, we should assess

how much our conclusions are affected by

inter-subject variability. It is very difficult

to provide strict guidelines here. In some cases

the inter-subject variability can be

parameterised, and thus included in the

sensitivity analysis. In these cases I would

consider this mandatory. In other cases you

simply need to build a new model for each new

subject you intend to describe, and this prevents

you from doing a systematic exploration on the

effects of inter-subject variability. In any case

the authors should consider this issue somehow.

One option may be to investigate a few subjects

that are representative of the extremes of

variability of the population of interest. This

is very similar to a design of experiments

approach, and it gives you a gross indication of

the level of variability you may expect in your

results.

At this stage of the validation process, we have

theoretical model that is robustly linked to the

experiments that are used to identify it. I

suggest that this second level of validation

should be mandatory to publish in those journals

aiming to applied biomechanics research.

- Validation against in vitro controlled

experiments: this is the first step in the true

validation process. It is usually very difficult

to perform them, and when you succeed they

usually show you a lot of unexpected weaknesses

in your model. Thus, I consider this a highly

valuable form of scientific result. One word

about how to report the results of this

comparison between numerical and experimental

results. Most authors use a linear regression

between measured and predicted value, and report

the regression parameters and coefficients.

However, I propose that we should always ask also

to report on the residuals of such regression.

One way I like is provide a root mean square

error as an indication of the average residual,

and the peak error, as indication of the maximum

residual.

My suggestion is that we should consider this

third level mandatory for all those journals

aiming at the space between biomechanics research

and clinical research.

- Risk-benefit analysis: as I wrote at the

beginning of this report, I believe that before

we can use results obtained from numerical models

in the clinical practice we need to report the

results of a risk-benefit analysis. To do so we

need a fully verified model on which a complete

sensitivity study and an in vitro validation

study has been conducted. This would provide us a

quantitative basis to estimate the risk

associated with the use of the model. Of course

these studies should be conducted in

collaboration with experts in clinical research,

which should have the necessary knowledge to

estimate the expected benefits.

- Retrospective studies: along the same lines,

but with greater level of confidence, there are

the retrospective studies. If you can use your

model to answer a clinical question over a

population for which the answer to this question

is known, you may get a good insight on the

validity of your model. In many cases the best

way to report these results is in term of

specificity and sensitivity, using the R.O.C.

curves. A key issue here is that the clinical

question must accept a yes-no answer.

When this type of validation studies is available

my suggestion is that we should allow the

publication of numerical studies also on the most

clinical journals. The clinical audience would

have in general clear enough the difference

between a retrospective and a prospective study.

Numerical models at this stage of validation

could also, in my opinion, be used in controlled

clinical trials.

- Prospective studies: as for any other method,

whenever possible the conclusive word on the

clinical use of a numerical model comes from

prospective clinical trials.

Code reliability

Some among those who replied to my original query

pointed out the issue of reliability of

commercial numerical analysis software, which is

a black box that may hide some problems. Some

others reversed the issue warning about the

danger of using software developed in-house.

After some thinking I decided that this has to do

with the general quality control we should apply

to all our laboratory instruments, including

software. Nobody is reporting in a scientific

paper when he calibrated last time the load cell

used in the study. This is left to our

consciences. For sure in our group we shall

start soon to develop a quality control system

for numerical analyses, very much like the one we

have already in place for the experimental

biomechanics unit. We expect to rely a lot on the

NAFEMS independent benchmarks for finite element

analysis codes.

----------------------------------------------------------------------------------------------------

Original posting:

I serve as a reviewer for a few biomechanics or orthopaedics

journals, some of which have a clear clinical perspective. In

particular I frequently revise papers where finite element models are

used as the main research tool.

These models are becoming more and more effective and powerful, and

it is not rare to find papers where the authors, on the basis on the

results obtained with the model, draw conclusions that may have a

clinical relevance, i.e. affect clinical decisions.

The problem I have is methodological: are we allowed to draw

clinically relevant conclusions from the predictions obtained by a

numerical model? Or, more appropriately, what are the conditions a

numerical model must fulfill in order to considered so reliable that

we can reasonably use it to draw clinically relevant conclusions?

In order to foster the discussion let me bring in my two cents.

In my understanding a numerical model is a particular instance of a

theoretical model. Once we are sure that the theoretical model has

been solved with sufficient numerical accuracy (and this is in

general possible with post-hoc indicators) what remains to be proved

is the adequacy of the theoretical model.

A theoretical model (a theory) in principle can be assumed to be true

(in the sense of a scientific truth, i.e. as far as we, as a

community we know, and within the limits of validity of the theory),

if with this model we can predict the outcomes of independent

experiments (independent in the sense that they are not those

measurements that were used to identify the model) and/or if starting

from the model we can deduct derivative conditions that are proved to

be true.

How do we translate these general rules in the specific of our

research domain, biomechanics? Can we say that once a model predicts

with sufficient accuracy the results of an in vitro experiment, we

can consider it valid and use it to draw clinically relevant

conclusions? Or is it sufficient to prove that the model is

numerically accurate and all the model assumptions and parameters are

well supported by experimental observations?

As usual I am ready to post a summery of the comments I shall receive.

----------------------------------------------------------------------------------------------------

Replies:

From: "Daniel P. Nicolella"

Marco,

You have raised an excellent point. Model simulation verification and

validation (V&V - essentially what you have eloquently described) is an

ongoing research topic within several disciplines. My colleague, Ben

Thacker, is actively involved in this area and has published several papers

on the topic of V&V. I have attached one such paper for your information

that should give you some background on the current thinking in this area.

I personally believe this is an important issue that should be addressed

within the biomechanics community and would like to see a consensus

developed on this topic.

Best Regards,

Dan Nicolella

----------------------------------------------------------------------------------------------------

From: "Mahar, Andrew"

Hi Marco,

You raise a very important issue in the world of orthopedic biomechanics.

We are conducting in-vitro experiments and numerical tests in parallel

across a variety of applications (spine/trauma) to optimize the numerical

theories, with the understanding that even in-vitro experiments can't

duplicate the in-vivo situation. At this point, the surgeons I work with

are hesitant to accept surgical interpretation based on the numerical data

and the engineers are hesitant to provide it. At this point, for our

research group, the greatest value numerical tests have are to better

understand implant behavior under a wide variety of testing

scenarios/positions/materials. These data may be used to better understand

the clinical situation, but that is after the fact. Anyway, that is where

our group stands on the issue.

I look forward to your summary.

Andrew

----------------------------------------------------------------------------------------------------

From: "Warren G. Darling"

Dear Dr. Viceconti, I would say that a model must

predict results of in vivo experiments (in a

human or in a close animal model) before it

should be used to draw any clinically relevant

conclusions. In vitro experiments often produce

quite different results from in vivo experiments.

I definitely do not think it is sufficient to

simply show a model is numerically accurate and

that model assumptions and parameters are well

supported by experimental observations. A

numerical model with valid assumptions and

parameters can be used as a guide to design

experiments to test clinically relevant issues,

but should not guide treatment until its

predictive value has been tested.

Sincerely,

Warren Darling

----------------------------------------------------------------------------------------------------

From: Bjorn.Olsen@MEMcenter.unibe.ch

Dear Dr. Viceconti,

Like yourself, I am often faced with the same

questions regarding numerical simulation; from

both surgeons and editors.

Like many on this list I have used various

numerical models in biomechanics, and I find it

useful to answer specific research questions.

However, in my opinion numerical modeling is

*just* a method. There is nothing special about

numerical simulation, and the manner in which

this particular method is employed must be

scrutinized like any other (eg. experimental)

method. Here I think the biomechanics community

has failed to some degree. When compared to most

other engineering disciplines, numerical

simulation is in its infancy when employed in

bio-engineering. The range of published

(modeling) material range from absolutely cutting

edge, to the - to say the least - not so great.

Very often, in the papers I review, I find a lot

of *very* basic mistakes (boundary conditions

applied incorrectly, linear analysis for large

deformation, material property/units wrong,

etc.). Furthermore, often the models do not

correspond to the problem the authors are trying

to solve.

The rise of numerical simulation over the last 10

years has seen a drop in experimental

biomechanics (which I personally find a bit

disturbing). This is most likely also related to

cost. Simulation is, most often, a cheaper and

quicker solution. Whether or not experimental

methods should always be used depends on the

specific problem at hand. As all of you would

know, quite often simulation is the only method

available to study certain problems in our field.

The engineer should always review the existing

methods (experimental/numerical) and chose the

one which is best suited to the answer the

research question.

It is mistaken to think that numerical analysis

(like FEA) is a simple, out-of-the-box method. It

is not. This is an engineering discipline, and

great care should be taken when using it. My

message is: We, as authors and reviewers, must

ensure that these methods are used correctly.

This is the only way it is possible to maintain

and transfer the credibility of numerical

simulation.

With kind regards,

Björn Olsen

----------------------------------------------------------------------------------------------------

From: "Anders Eriksson"

Dear Marco,

I think you have raised a very important issue, which deserves

an extensive discussion in the ongoing quick

development of numerical modelling and

simulations in biomechanics. My viewpoint, being

primarily in computational mechanics but with

several applications to biomechanical problems,

is that there is often/always a too high

credibility given to numerical, e.g. FEM,

calculations -- "the experiments were verified by

FEM calculations" -- without stating anything

about the used elements: nuermical, geometric,

kinematic and kinetic assumptions.

As you say, numerical simulations are always

based on some underlying assumptions and theory:

the 'model'. They can therefore, at best, be as

good as these theories and assumptions. But, and

this is important, they also have their own sets

of assumptions, approximations and numerical

error sources. These must be kept under strict

control, for any conclusions of practical

importance to be drawn. This is as well known by

people within the computational research areas as

is the fact that physical theories are only the

best possible explanation, within a set of

assumptions, is to the physics people. The

general science philosophical conclusion is

probably that we always have a tendency to

consider everyone else's scientific area as

simple and well-defined, whereas all complexities

occur in our own area.

Regarding your underlying question on the

applicability of numerical simulations, I am,

however, not only pessimistic. But, care is

needed. I think that the basic requirement on a

numerical study, aiming to have application

relevance, is a careful documentation (by you or

someone reliable) of the underlying assumption

and theories used in the modelling. And, this is

perhaps even more the case when black box

software is used. In particular, this is true

when the general FEM software is used for

biological problems, which are very often far off

from the parametric ranges where the basic

numerics have been developed and verified. As

strict theories seldom exist for general

problems, testing of limited setups should always

be documented, as a basis for the larger models.

It is also, as you mention, important to state

the problem in fundamental principles, and not

just base parameters and assumptions on

regressions from very similar experiments. This

done, I think that conclusions drawn from

numerical modelling and simulations can be

helpful in the clinical understanding.

In the same context, I would also like to point

to a common type of problem statement in

engineering, and maybe as useful in other

scientific branches, namely the 'inverse problem'

solving, or perhaps a very systematic 'what-if'

research. This can be used for understanding many

types of behaviour qualitatively and (at least

relatively) quantitatively. Assuming that you

have good knowledge of the basic building blocks

inccluded in your simulation package, you can

very easily vary assumptions and parameters in

the simulation model, to see how results are

affected by these assumptions, thereby allowing

some conclusions to be drawn, when these

simulations are compared to experiments (or

general knowledge of behaviour). This is, but

only when you have the underlying knowledge about

the modelling assumptions, something very

different and much more powerful than regression

of parameters in a fixed basic model, as it

allows a much wider space of assumptions to be

tested. Good basic knowledge and a critical view

on the interpretation of the results are the key

ingredients in this method, and the possibility

for independent critical assessment of

assumptions the strength.

As I said, I think that this area needs an

extensive discussion to avoid that many incorrect

clinical conclusions are drawn, based on bad

numerical models, using irrelevant theories and

incompletely known parameters.

Best regards

Anders Eriksson

----------------------------------------------------------------------------------------------------

From: "linping zhao"

Dr. Venceconti:

You brought a very important topic for

discussing. I am expecting such discussion for a

long time. It seems it's time now. I'd like to

see various responses from people with either

engineering or clinical background or from other

point of views.

Here are more concerns regarding to the validation of a FE model:

If a FE model is validated with a mechanical

model or even a cadaver test, is it clinical

relevant? In what sense and what degree?

When we build up a FE model based upon in vivo

CT/MRI data, is it possible to validate a FE

model in vivo? If so, how? If not, how can we say

the FE model is validated?

If the prediction of a FE model is in agreement

with clinical observation, the model can be

considered as validated in some sense. Is this

statement true? If so, in what sense the model is

validated?

Looking forward to reading more.

Linping Zhao, Ph.D.

Plastic and CranioMaxillofacial Surgery

Shriners Hospitals for Children, Chicago

----------------------------------------------------------------------------------------------------

From: Apache

Here is my comment: I think it would be too rigorous to require

every modeller to do the whole job at once, i.e. to find a parameter

fit to an experimental outcome (identifying all unknown parameters

of the numerical model), use the predictive power of the model by parameter

variation and perform an adequate testing experiment.

Now, here, by trying to define what "adequate" means, we find that there

are some prerequisites to be fulfilled in order for a model to increase

the probability of its validity. The probability can never really be 1 (valid)

but 0 (invalid).

1. The parameters of the theoretical model must be independent of time

or experiment (i.e. they may in no way be fitted along time or across

experiments!) AND can be mapped to physical / physiological quantities

which can at least in principle be determined experimentally.

2. The more parameters the model has (in the fitting process) the lower is

its value for identifying fundamental, underlying principles.

In my opinion this statement should be equivalent to saying that for every

structure or phenomenon that can be identified in nature you only need

a finite number of essential parameters describing it. It's just a personal

belief that this number is always small enough to get the chance

of understanding. It is the art of research to identify these essential

parameters.

In FEM models there are heaps of parameters that additionally may be

hidden to the normal user. I.e. even requiring tests of sensitivity

with respect to these parameters (physical or numerical) seems to be

an endless job. I think providers of these software packages should be

urged much more by the scientific community to pass fundamental physical

tests e.g. such as conservation of energy or of moment of momentum in simple

test models that are claimed to be conservative or free of external

forces / torques.

3. More generally: The lower the ratio between the number of fitted parameters

and the number of independent experimental findings that may be used for the

fitting procedure the better the model meaning that the probability

of the model to have predictive power is higher.

The lower the number of model parameters the higher its potential for

gaining insight. Thus, the effort of researches to REDUCE the number of

parameters should be favoured by referees over the apparent success

to provide a perfect multi-parameter fit. Here, it is very useful to

have an estimation of the accuracy of the EXPERIMENTAL findings. Simpler

models (and there potential of gaining deep insight) are discredited

by an inappropriate trust into measured data.

4. Additionally to 2. good referees ask the researches for the sensitivity of

their models with respect to predictions. Here, 'sensitivity' means that

the FITTED parameters should be varied and the deviation of the results

with respect to the a priori experiments should be shown. On the other hand,

a good referee also identifies very few but CRITICAL parameters but does

not make the researcher do a complete new study.

> How do we translate these general rules in the specific of our

> research domain, biomechanics? Can we say that once a model predicts

> with sufficient accuracy the results of an in vitro experiment, we

> can consider it valid and use it to draw clinically relevant

> conclusions?

I tried to speak out explicitly some criteria (1. to 4. above) that I

find essential when dealing with the interplay of theory and experiment.

I personally 'love' extremely reduced models more and more. My very personal

view is that an FEM-model can be considered a little bit more valid if it has

passed one predictive test. Full validity in biomechanics can not be shown

within one study as long as the MINIMAL number of parameters that might be

necessary for predicting underlying processes is not at least estimated.

If one compares to the many years that e.g. the

spring-mass model for locomotion

was and still is tested and developed around the world one can get a feeling

for the long way to high validity of a model. I personally would not draw

clinically relevant conclusions based on an FEM-model before knowing many

things about the set of parameters, sensitivities etc. (see above).

> Or is it sufficient to prove that the model is

> numerically accurate and all the model assumptions and parameters are

> well supported by experimental observations?

As a summary: I think, validity can only be given in terms of probability.

Unfortunately, one can only prove a model to be INVALID if it clearly fails

with a predection. Otherwise, we can only check accuracies, minimize the number

of parameters and do as many tests (some mentioned in 1. to 4.) as possible to

enhance the probability of model validity.

Cheers,

Michael.

--

Michael Gunther

----------------------------------------------------------------------------------------------------

From: "Ben Thacker"

Marco,

The points you raise are exactly those that must

be addressed if numerical simulation is

to play an active role in design, i.e., produce

credible predictions with quantified

accuracy. Requiring this of numerical models

means the stakes have been raised -- over the

next few years it will be interesting (and

exciting) to see if the modeling community can

address these questions to the satisfaction of

the decision makers and customers. It will

not be easy.

I'd like to refer you to our ASME V&V Standards Committee website at

www.usacm.org/vnvcsm

There are several reports there, etc. that may be

of interest to you. You can also

subscribe to our email list from the web site.

I hope this helps.

--Ben

----------------------------------------------------------------------------------------------------

From: Ton van den Bogert

Dear Marco,

This is a great question, and very important.

I had hoped for some public replies, maybe when

you post your summary you can ask if people want

to post additional public responses.

>Or, more appropriately, what are the conditions a

>numerical model must fulfill in order to considered so reliable that

>we can reasonably use it to draw clinically relevant conclusions?

It depends, I suppose. If a model depends only

on known laws of physics, and not on assumptions,

one should have no problem with clinical

conclusions. But usually there are assumptions

and/or simplifications, and this is where your

question applies. Here are my ideas on how to

alleviate such concerns.

First, it is often not known to what extent the

results depend on assumptions. Usually it is

possible to do sensitivity analysis or

perturbation analysis. Simply change the assumption and see how much

difference it makes. If there is any question on the influence of

assumptions, such a sensitivity analysis should always be attempted.

Second, computatonal studies often use just one model, and if you think

of it, this is absurd. You would never use just one human subject, one

animal, or one cadaver specimen. There is biological variation among

humans and when a model represents a sample of N=1, conclusions may be

quite unreliable. Why not make a group of models to reflect variability

within the human population, do the model study

with all of them, and analyze the results

statistically? This also takes care of the

sensitivity analysis to some extent. If a result

is obtained with one model, and not with another,

based on a different human subject, the result is

not reproducible and should not be reported. If

you have N=1, how do you know?

Computers get faster all the time. N=1 with no sensitivity analysis may

have been OK ten years ago. Now, computers are 20 times faster, so we

can do N=10 and additionally use one of those models to quantify ten

sensitivities. But instead, we have built models that take longer and

longer to compute and are supposedly more realistic. But still we often

do N=1 and no sensitivity analysis. I think this is not the right way

to proceed.

This N=1 practice may be more of a problem in my

own field of movement dynamics than in tissue

mechanics. In movement, there is nonlinear

dynamics with potential for self-regulating

mechanisms. See Wright et al (Clin Biomech,

1998) for a nice example. The question was the

effect of shoe hardness on impact forces in

running. Ten models were used. In some, the

harder shoe increased the impact, in others it

decreased the impact. This was unexpected and

forced us to examine the results carefully. A

self-regulating mechanism was found: a harder

shoe causes the impact force to rise faster,

which causes earlier knee flexion, which then

slows the further rise in impact force. Whether

the peak force ends up being higher or lower than

with the other shoe, depends on subtle

differences between the models. This is an

example where N=1 would have been a lottery,

rather than a scientific study. With ten models,

the statistically correct conclusion was

obtained: no significant effect, but also there

was an understanding of the underlying mechanism.

There is also the problem of quality control. If

there is a bug in your code, multiple models and

sensitivity analyses will all be influenced in

the same way. You may think that results are

consistent, and therefore correct. This is where

commercial software has the advantage of a wide

user base, and a better likelihood that errors

have already been found. For those of us who

want to go beyond commercial tools and do our own

programming, sound software engineering practices

and a healthy dose of self-criticism are

essential.

Finally, models are mainly useful in basic research. We can use a model

to generate and test hypotheses, but before

giving clinical recommendations, I think that a

clinical trial should be done to see if

conclusions are also valid in the real world.

This can be expensive, but it is nevertheless the

standard procedure for drugs, after doing animal

studies. For the same reason it may be a good

idea in biomechanics also, after numerical

studies have been done as a first test of

efficacy and safety.

It seems that in biomechanics we are not always

as rigorous as in other disciplines, perhaps

because biomechanics usually does not deal with

life-threatening problems. By the way, the

problem is not limited to numerical models. In

vitro models are heavily used in orthopaedic

biomechanics, to study a joint or bone in isolation. Results can depend

very much on the mechanical boundary conditions, and this is not always

sufficiently recognized.

Ton van den Bogert

--

--------------------------------------------------

MARCO VICECONTI, PhD (viceconti@tecno.ior.it)

Laboratorio di Tecnologia Medica tel. 39-051-6366865

Istituti Ortopedici Rizzoli fax. 39-051-6366863

via di barbiano 1/10, 40136 - Bologna, Italy

Tiger! Tiger! Burning bright in the forest of the night,

what immortal hand or eye could frame thy fearful symmetry?

--------------------------------------------------

Opinions expressed here do not necessarily reflect those of my employer

-----------------------------------------------------------------

To unsubscribe send SIGNOFF BIOMCH-L to LISTSERV@nic.surfnet.nl

For information and archives: http://isb.ri.ccf.org/biomch-l

Please consider posting your message to the Biomch-L Web-based

Discussion Forum: http://movement-analysis.com/biomch_l

-----------------------------------------------------------------