To Explain The Different Types Of Research Methods
The purpose of this technical backgrounder is to explain the different types of
research methods and provide guidance on how to evaluate scientific data
including research studies and other types of data.
Food and dietary supplement product safety and benefit knowledge is acquired
through a variety of sources of scientific data including research studies and
reviews of historical data. Some food and dietary supplement products are
allowed to make claims about product benefits. Product label claims are
regulated by the Food and Drug Administration (FDA), have stringent requirements
set by law, and must be substantiated with scientific data.
The introduction of foods and dietary supplements for uses such as treating
health conditions, improving athletic performance or improving general well
being, presupposes that there is an understanding of the efficacy of these
products. To gain such knowledge is to become educated in the research methods
that are employed to determine efficacy of these products and to understand the
appropriate methods that must be undertaken to insure that such information is
validated and applicable. There are numerous publications in the literature on
research methods employed to determine the efficacy of treatments for different
conditions (1A, 2A, 3A). These same research methods are applicable to food and
dietary supplement research.
When looking at research on food and dietary supplement product benefits, it
is important to keep in mind the difficulty in designing studies to evaluate the
positive benefits of maintaining good health versus treating diseases. This
backgrounder will describe current research methods and provide a few tips on
evaluating the resulting published reports and related media stories.
Research methods have been established in the medical literature to help the
scientific and lay communities discern whether treatments, including
pharmaceutical, dietary supplement, and/or food products, have positive or
negative effects on the wellbeing of the subjects with determined treatable
conditions. Additionally, research can discover if products have negative
effects of which the consumer needs to be aware. These questions can be answered
through research that specifically uses research strategies (methods) to
identify benefits and negative effects.
There are primarily two basic broad categorizations of research that are
performed to determine how treatments affect the consumer/subject. These two
categories are outlined below and are divided into two segments that show, 1)
those that primarily observe, describe, and document the observed
characteristics about groups of subjects or subjects and those that, 2) evaluate
the effect that an intervention (drug, food or dietary supplement) has on study
subjects. These study designs are summarized from Saunders and Trapp (2A) below.
These two groups are called observational/ explanatory and experimental designs.
There are quite a few types of observational/explanatory studies that can be
1. Descriptive or Case Series: which are accounts of some characteristics
observed from groups of subjects.
2. Case-Controlled Studies: which begin with the presence or absence of a
condition or outcome and looks backward to determine the possible causes
and/or risk factors of such a study group.
3. Cross-Sectional Studies: which collect and analyze data collected on a
group of subjects at one time rather than over a particular time period.
4. Cohort Studies: which look at a group of individuals who have a common
characteristic that may be a risk factor for a disease or condition.
5. Retrospective (or Historical) Cohort Studies: which look at previous
records of a cohort study, but going back in time for a particular period
instead of looking at cohorts from one point in time.
The second group of studies, experimental designs, is primarily concerned
with drawing conclusions about a particular treatment or procedure. These
designs generally fall into two categories of studies, 1) those that are
controlled, comparing a drug, food, dietary supplement, or treatment with a
placebo and/or another active treatment, or 2) those studies that have a
treatment prescribed but where there is no comparison to another treatment.
1. Controlled Studies: are studies that treat both groups alike in terms of
the procedures that are used to measure and treat, except for the treatment
itself. Such studies may compare:
a food or dietary supplement to a placebo
a food or dietary supplement to a drug
a drug, food or dietary supplement and placebo, or
food or dietary supplement to another food or dietary supplement or
Both the subjects and doctors (treaters) may be blinded (uninformed at the
beginning of the study as to which group of subjects is receiving the treatment
versus the placebo or alternative treatment). When both the subjects and doctors
are blinded the study is called "double blind" and when only the
subjects are blinded it is called "single blind." Controlled studies
are categorized in the following way:
a. Meta-Analyses: use statistical methods to combine and summarize
the results of several randomized (process by which subjects are randomly
selected, assigned and arranged to groups within the study) controlled trials
and integrates the results offering an objective overall conclusion (4A). A
quantitative result of either a overall positive or negative effect is
determined. The meta-analysis offers the highest level of scientific scrutiny.
The Cochrane Collaboration has conducted many meta-analyses on specific
clinical conditions (5A). The meta-analysis is invaluable for practitioners as
it provides them with a base of evidence from which they can make treatment
b. Systematic Reviews: involve the application of scientific
strategies, in ways that limit bias, to the assembly, critical appraisal, and
synthesis of all the relevant studies that address a specific, often narrow,
clinical question in depth (6A).
c. Randomized Controlled Trials (RCTs): are studies where the
subjects in the treatment and control groups are randomized, meaning they are
randomly selected and assigned to either the control or treatment groups
within the study. Additionally, subjects do not know which treatments they
will receive in the study (e.g., placebo or food/dietary supplement/drug).
This is the most rigorous way of determining a cause-effect relation between a
treatment and outcome and for assessing the cost effectiveness of a treatment.
RCTs may be "crossover controlled" meaning trial subjects are
randomly assigned to a study group and each group receives a sequence of two
or more treatments given consecutively, intervened by a "washout"
period where one "rests" from one treatment for a specific period of
time before commencing the other treatment.
d. Non-Randomized Trials: are those which do not randomly assign
subjects to either the control or treatment groups and therefore are open to
the biases of those selecting and assigning the subjects to the groups.
e. Trials with self-controls: are those where the subjects are
exposed to both the experimental treatment and the control.
f. Crossover Studies: are another form of self-control studies, which
are modified somewhat to provide a blend of self-controls where the subjects
may be switched over to the comparative treatment after a washout period.
2. Uncontrolled studies are studies where there is no comparison of subjects
from different treatment groups. This design assumes that the treatment is the
best possible and there is no real comparison available, or the creation of an
appropriate placebo is presumed to be too difficult.
The above research designs are used to determine new findings and test the
outcomes of treatments (drugs, foods or dietary supplement products) on subject
populations. The importance of the research design lies in the ability of the
research to define when treatments are or are not efficacious. Therefore, the
design of research studies will have an effect on the type of answers that are
achieved in the projects that are undertaken in the food and dietary supplement
HISTORIC RECORD (TRADITIONAL USE)
Useful knowledge regarding foods and dietary supplements, particularly
botanicals, may come from sources other than controlled or uncontrolled studies.
Most of what is known regarding the use of botanicals originates from cultural
historic written and oral records, not scientific studies. While scientific
studies can test the validity of cultural beliefs, a lack of such study does not
suggest that what has been passed down over the generations is not worthwhile.
In fact it is often the accumulated body of human experience that identifies
which botanicals are useful, for what, and how to use them.
Not all historic records carry the same weight. Some information derives from
codified systems of medicine such as Traditional Chinese Medicine (TCM),
Japanese Kampo, Indian Ayurvedic, and many others. It is reasonable to consider
such sources as possibly offering useful information in the absence of
scientific studies. The level of usefulness of such information is often
strengthened when the same material is used by different cultures for treating
the same condition.
The usefulness of historic records and the weight given with regard to its
reliability as factual is dependent upon several factors including how long the
material has been used, how it is prepared, what it is used for, how much is
used, what safety information is available, etc. Obviously, accumulated cultural
experience is likely to be more reliable than a single testimonial or anecdotal
Testimonials and anecdotal reports are based on an individual's claim of some
result(s) from a product. Such reports represent the least rigorous scientific
standard for scientific evidence regarding the validation of the safety and
efficacy of dietary supplement or food products.
EVALUATING RESEARCH ARTICLES AND RELATED MEDIA PUBLICATIONS
In evaluating the scientific literature a published article should have the
following components (7A):
An abbreviated summary of the study
Note: With the advent of the Internet, the ease of access to research
"abstracts," the one paragraph summary of studies, is greater than
ever before. However, reading only the abstract of a study can be
"hazardous" to one's research understanding. Several studies have
shown that abstracts 1) describe data that are not found in the full text of
the article, 2) inaccurately describe data described in the full text of the
article, or 3) exclude sometimes important data found only in the full text.
Whenever possible, it is best to read the full text of the study rather than
just the abstract.
States research purpose and problem
Establishes the importance of the research
Includes sufficient literature review
Should complement the discussion section
3. Materials and Methods (most important section of a research article)
Defines population and sample size
Identifies randomization method
Describes food or dietary supplement used, including physical
description of material and dosage type and amount
Identifies methods used to blind participants
Identifies/defines materials and methods used to obtain physical data
Includes outcome measures
Includes statistical analysis methods
The research design should be clearly identified and include an
explanation of the research design used, i.e.:
Longitudinal (prospective or retrospective)
Controlled or uncontrolled
Observational or interventional
Summarized characteristics of the study groups
Are tables and figures clearly labeled?
Are there sufficient data to interpret the results clearly?
Are the results of appropriate statistical analysis provided?
Are all research outcomes reported?
Summary of outcome measures
Are all subjects accounted for (including study dropouts)?
Is the degree of variability in measurements provided
Are all adverse effects reported?
Are the discussion comments supported by the data?
Are the conclusions consistent with the study objectives?
Are the references appropriate for the study?
Specific to NNFA Backgrounders, a hierarchy has been established for the
references and is the following:
: research article or editorial comment in peer-reviewed
B reference: letters to the editor, professional personal
communications, meeting/conference abstracts (NOT usually peer reviewed), and
pending reviewed publications
C reference: lay/trade press publications, interviews, patents and Web
site material (NOT peer-reviewed)
This hierarchy shows the relative reliability and scrutiny of the science in the
references cited. In general, articles in peer-reviewed journals are subjected
to critical evaluation by disinterested scientists before acceptance for
publication. A backgrounder with few "A" references, or a
preponderance of "B" and "C" references, will usually
indicate a substance with little critically reviewed scientific research to
support its use and/or safety
In 1995, the Food and Nutrition Science Alliance (FANSA) (8A) a partnership
of four professional organizations including the American Dietetic Association,
American Society for Clinical Nutrition, American Society for Nutritional
Sciences and Institute of Food Technologists issued the following list of ten
signs of junk science:
Recommendations that promise a quick fix
Dire warnings of danger from a single product or regimen
Claims that sound too good to be true
Simplistic conclusions drawn from a complex study
Recommendations based on a single study
Statements refuted by reputable scientific organizations
Lists of "good" and "bad" foods
Recommendations made to help sell a particular product
Recommendations based on studies not peer reviewed
Recommendations from studies that ignore differences among individuals or
Another common issue that comes up within the dietary supplement industry is the
use of scientific studies on one product to sell another product with the same
or similar name. It is valid to apply the science from one material to another
when the materials are the same or comparable. However because dietary
supplements, especially botanicals, can vary considerably from one product to
another the science done on one may not be appropriate when applied to another.
Therefore, doing so is not good science.
One example of this would be using the scientific and clinical studies
specifically done with one specialized extract of ginkgo leaf to sell other
seemingly similar ginkgo leaf extract. Companies that do this may claim that
their product has the same safety and efficacy as the tested one though their
product may have never been subjected to any such study. Even if their ginkgo
product has the same percentage of terpenes and flavonoids as the tested one,
that may not make the products equivalent to each other.
In the case of plant extracts that standardize for compounds found in only
small amounts (e.g., hyperforin and hypericin in St. John's wort) the remaining
93-99% may be somewhat or very different in composition. This suggests that
different extracts with the same amounts of these identified compounds may
behave differently in the body. Companies need to compare "apples to
apples" when referencing science from other companies in order not to use
science in a way that is false or misleading. On the other hand it is also
important to realize that if scientific studies are employed to test the
validity of a traditional preparation the results of such research may be
appropriately applied to other similar preparations.
By furnishing this backgrounder, we do not provide any opinion as to:
The safety of any product containing this ingredient;
The efficacy of any product containing this ingredient;
The use of any specific brand of product; or
The level of substantiation for either the safety or efficacy of any such
Neither this backgrounder nor any portion of this backgrounder may be used in
advertising or promotional materials. In addition, this backgrounder does not
constitute, and is not to be used as, "third party literature" as that
term is used in connection with section 5 of the Dietary Supplement Health and
Education Act (DSHEA).
As with any health-related product, consumers should discuss the use of any
products with a health care practitioner.
1A. Creswell, J.W. (1994) Research Design: Qualitative and
Quantitative Approaches, Thousand Oaks, CA, Sage Publications.
2A. Dawson-Saunders, B., Trapp, R.G. (1994) Basic and Clinical Biostatistics,
2nd ed., Norwalk, Connecticut: Appleton & Lange.
3A. Guyatt, G, et al. (1998) A Clinician's Guide for Conducting Randomized
Trials in Individual Patients. Can Med Assn J 139.
4A. Egger, M., Smith, G.D. and Phillips, A.N. (1997) Meta-Analysis: Principles
and Procedures. Brit Med J 315:1533-1537.
5A. The Cochrane Controlled Trials Register. In: Cochrane Library. CD ROM and
6A. Cook, DJ, et al. (1997) Systematic Reviews: Synthesis of Best Evidence for
Clinical Decisions. Ann Intern Med 126:376-380.
7A. Sackett, D.L. (1981) How to Read Clinical Journals: I: Why to Read Them and
How to Start Reading Them Critically. Can Med Assn J 124:555-558.
8A. Food and Nutrition Science Alliance (FANSA). (1995) FANSA's 10 Red Flags of
Junk Science. www.faseb.org/ascn/fansa1.htm
*KEY TO REFERENCE HIERARCHY
A reference: research article or editorial comment in peer-reviewed journal
B reference: letters to the editor, professional personal communications,
meeting/conference abstracts (NOT usually peer reviewed), and pending reviewed
C reference: lay/trade press publications, interviews, patents and Web site
material (NOT peer-reviewed)
This hierarchy shows the relative reliability and
scrutiny of the science in the references cited. In general, articles in
peer-reviewed journals are subjected to critical evaluation by disinterested
scientists before acceptance for publication. A backgrounder with few A
references, or a preponderance of B and C references, will usually indicate a
substance with little critically reviewed scientific research to support its use
The FDA (Food and Drug Administration) has not evaluated these statements. The products, articles and or comments in this site are not intended to diagnose, treat, cure, or prevent any disease. If you have any illness you should always consult with your health practitioner.