...

PURPOSE
The purpose of this technical backgrounder is to explain the different types of research methods and provide guidance on how to evaluate scientific data including research studies and other types of data.

BACKGROUND
Food and dietary supplement product safety and benefit knowledge is acquired through a variety of sources of scientific data including research studies and reviews of historical data. Some food and dietary supplement products are allowed to make claims about product benefits. Product label claims are regulated by the Food and Drug Administration (FDA), have stringent requirements set by law, and must be substantiated with scientific data.

The introduction of foods and dietary supplements for uses such as treating health conditions, improving athletic performance or improving general well being, presupposes that there is an understanding of the efficacy of these products. To gain such knowledge is to become educated in the research methods that are employed to determine efficacy of these products and to understand the appropriate methods that must be undertaken to insure that such information is validated and applicable. There are numerous publications in the literature on research methods employed to determine the efficacy of treatments for different conditions (1A, 2A, 3A). These same research methods are applicable to food and dietary supplement research.

When looking at research on food and dietary supplement product benefits, it is important to keep in mind the difficulty in designing studies to evaluate the positive benefits of maintaining good health versus treating diseases. This backgrounder will describe current research methods and provide a few tips on evaluating the resulting published reports and related media stories.

RESEARCH METHODS
Research methods have been established in the medical literature to help the scientific and lay communities discern whether treatments, including pharmaceutical, dietary supplement, and/or food products, have positive or negative effects on the wellbeing of the subjects with determined treatable conditions. Additionally, research can discover if products have negative effects of which the consumer needs to be aware. These questions can be answered through research that specifically uses research strategies (methods) to identify benefits and negative effects.

There are primarily two basic broad categorizations of research that are performed to determine how treatments affect the consumer/subject. These two categories are outlined below and are divided into two segments that show, 1) those that primarily observe, describe, and document the observed characteristics about groups of subjects or subjects and those that, 2) evaluate the effect that an intervention (drug, food or dietary supplement) has on study subjects. These study designs are summarized from Saunders and Trapp (2A) below. These two groups are called observational/ explanatory and experimental designs.

There are quite a few types of observational/explanatory studies that can be categorized including:

1. Descriptive or Case Series: which are accounts of some characteristics observed from groups of subjects.
2. Case-Controlled Studies: which begin with the presence or absence of a condition or outcome and looks backward to determine the possible causes and/or risk factors of such a study group.
3. Cross-Sectional Studies: which collect and analyze data collected on a group of subjects at one time rather than over a particular time period.
4. Cohort Studies: which look at a group of individuals who have a common characteristic that may be a risk factor for a disease or condition.
5. Retrospective (or Historical) Cohort Studies: which look at previous records of a cohort study, but going back in time for a particular period instead of looking at cohorts from one point in time.

The second group of studies, experimental designs, is primarily concerned with drawing conclusions about a particular treatment or procedure. These designs generally fall into two categories of studies, 1) those that are controlled, comparing a drug, food, dietary supplement, or treatment with a placebo and/or another active treatment, or 2) those studies that have a treatment prescribed but where there is no comparison to another treatment.

1. Controlled Studies: are studies that treat both groups alike in terms of the procedures that are used to measure and treat, except for the treatment itself. Such studies may compare:

• a food or dietary supplement to a placebo

• a food or dietary supplement to a drug

• a drug, food or dietary supplement and placebo, or

• food or dietary supplement to another food or dietary supplement or positive control.

Both the subjects and doctors (treaters) may be blinded (uninformed at the beginning of the study as to which group of subjects is receiving the treatment versus the placebo or alternative treatment). When both the subjects and doctors are blinded the study is called "double blind" and when only the subjects are blinded it is called "single blind." Controlled studies are categorized in the following way:

a. Meta-Analyses: use statistical methods to combine and summarize the results of several randomized (process by which subjects are randomly selected, assigned and arranged to groups within the study) controlled trials and integrates the results offering an objective overall conclusion (4A). A quantitative result of either a overall positive or negative effect is determined. The meta-analysis offers the highest level of scientific scrutiny. The Cochrane Collaboration has conducted many meta-analyses on specific clinical conditions (5A). The meta-analysis is invaluable for practitioners as it provides them with a base of evidence from which they can make treatment decisions.
b. Systematic Reviews: involve the application of scientific strategies, in ways that limit bias, to the assembly, critical appraisal, and synthesis of all the relevant studies that address a specific, often narrow, clinical question in depth (6A).
c. Randomized Controlled Trials (RCTs): are studies where the subjects in the treatment and control groups are randomized, meaning they are randomly selected and assigned to either the control or treatment groups within the study. Additionally, subjects do not know which treatments they will receive in the study (e.g., placebo or food/dietary supplement/drug). This is the most rigorous way of determining a cause-effect relation between a treatment and outcome and for assessing the cost effectiveness of a treatment. RCTs may be "crossover controlled" meaning trial subjects are randomly assigned to a study group and each group receives a sequence of two or more treatments given consecutively, intervened by a "washout" period where one "rests" from one treatment for a specific period of time before commencing the other treatment.
d. Non-Randomized Trials: are those which do not randomly assign subjects to either the control or treatment groups and therefore are open to the biases of those selecting and assigning the subjects to the groups.
e. Trials with self-controls: are those where the subjects are exposed to both the experimental treatment and the control.
f. Crossover Studies: are another form of self-control studies, which are modified somewhat to provide a blend of self-controls where the subjects may be switched over to the comparative treatment after a washout period.

2. Uncontrolled studies are studies where there is no comparison of subjects from different treatment groups. This design assumes that the treatment is the best possible and there is no real comparison available, or the creation of an appropriate placebo is presumed to be too difficult.

The above research designs are used to determine new findings and test the outcomes of treatments (drugs, foods or dietary supplement products) on subject populations. The importance of the research design lies in the ability of the research to define when treatments are or are not efficacious. Therefore, the design of research studies will have an effect on the type of answers that are achieved in the projects that are undertaken in the food and dietary supplement industries.

HISTORIC RECORD (TRADITIONAL USE)
Useful knowledge regarding foods and dietary supplements, particularly botanicals, may come from sources other than controlled or uncontrolled studies. Most of what is known regarding the use of botanicals originates from cultural historic written and oral records, not scientific studies. While scientific studies can test the validity of cultural beliefs, a lack of such study does not suggest that what has been passed down over the generations is not worthwhile. In fact it is often the accumulated body of human experience that identifies which botanicals are useful, for what, and how to use them.

Not all historic records carry the same weight. Some information derives from codified systems of medicine such as Traditional Chinese Medicine (TCM), Japanese Kampo, Indian Ayurvedic, and many others. It is reasonable to consider such sources as possibly offering useful information in the absence of scientific studies. The level of usefulness of such information is often strengthened when the same material is used by different cultures for treating the same condition.

The usefulness of historic records and the weight given with regard to its reliability as factual is dependent upon several factors including how long the material has been used, how it is prepared, what it is used for, how much is used, what safety information is available, etc. Obviously, accumulated cultural experience is likely to be more reliable than a single testimonial or anecdotal report.

TESTIMONALS
Testimonials and anecdotal reports are based on an individual's claim of some result(s) from a product. Such reports represent the least rigorous scientific standard for scientific evidence regarding the validation of the safety and efficacy of dietary supplement or food products.

EVALUATING RESEARCH ARTICLES AND RELATED MEDIA PUBLICATIONS
In evaluating the scientific literature a published article should have the following components (7A):

1. Abstract

• An abbreviated summary of the study

• Includes:
  Purpose
  Research design
  Methodology
  Results
  Conclusion

Note: With the advent of the Internet, the ease of access to research "abstracts," the one paragraph summary of studies, is greater than ever before. However, reading only the abstract of a study can be "hazardous" to one's research understanding. Several studies have shown that abstracts 1) describe data that are not found in the full text of the article, 2) inaccurately describe data described in the full text of the article, or 3) exclude sometimes important data found only in the full text. Whenever possible, it is best to read the full text of the study rather than just the abstract.

2. Introduction

• States research purpose and problem

• Establishes the importance of the research

• Includes sufficient literature review

• Should complement the discussion section

3. Materials and Methods (most important section of a research article)

• Defines population and sample size

• Identifies randomization method

• Describes food or dietary supplement used, including physical description of material and dosage type and amount

• Identifies methods used to blind participants

• Identifies/defines materials and methods used to obtain physical data

• Includes outcome measures

• Includes statistical analysis methods

• The research design should be clearly identified and include an explanation of the research design used, i.e.:

• Cross-sectional

• Longitudinal (prospective or retrospective)

• Controlled or uncontrolled

• Observational or interventional

4. Results

• Summarized characteristics of the study groups

• Are tables and figures clearly labeled?

• Are there sufficient data to interpret the results clearly?

• Are the results of appropriate statistical analysis provided?

• Are all research outcomes reported?

• Summary of outcome measures

• Are all subjects accounted for (including study dropouts)?

• Is the degree of variability in measurements provided

• Are all adverse effects reported?

5. Discussion

• Are the discussion comments supported by the data?

• Are the conclusions consistent with the study objectives?

6. References

• Are the references appropriate for the study?

Specific to NNFA Backgrounders, a hierarchy has been established for the references and is the following:

A reference: research article or editorial comment in peer-reviewed journal

B reference: letters to the editor, professional personal communications, meeting/conference abstracts (NOT usually peer reviewed), and pending reviewed publications

C reference: lay/trade press publications, interviews, patents and Web site material (NOT peer-reviewed)
This hierarchy shows the relative reliability and scrutiny of the science in the references cited. In general, articles in peer-reviewed journals are subjected to critical evaluation by disinterested scientists before acceptance for publication. A backgrounder with few "A" references, or a preponderance of "B" and "C" references, will usually indicate a substance with little critically reviewed scientific research to support its use and/or safety

In 1995, the Food and Nutrition Science Alliance (FANSA) (8A) a partnership of four professional organizations including the American Dietetic Association, American Society for Clinical Nutrition, American Society for Nutritional Sciences and Institute of Food Technologists issued the following list of ten signs of junk science:

• Recommendations that promise a quick fix

• Dire warnings of danger from a single product or regimen

• Claims that sound too good to be true

• Simplistic conclusions drawn from a complex study

• Recommendations based on a single study

• Statements refuted by reputable scientific organizations

• Lists of "good" and "bad" foods

• Recommendations made to help sell a particular product

• Recommendations based on studies not peer reviewed

• Recommendations from studies that ignore differences among individuals or groups

OTHER
Another common issue that comes up within the dietary supplement industry is the use of scientific studies on one product to sell another product with the same or similar name. It is valid to apply the science from one material to another when the materials are the same or comparable. However because dietary supplements, especially botanicals, can vary considerably from one product to another the science done on one may not be appropriate when applied to another. Therefore, doing so is not good science.

One example of this would be using the scientific and clinical studies specifically done with one specialized extract of ginkgo leaf to sell other seemingly similar ginkgo leaf extract. Companies that do this may claim that their product has the same safety and efficacy as the tested one though their product may have never been subjected to any such study. Even if their ginkgo product has the same percentage of terpenes and flavonoids as the tested one, that may not make the products equivalent to each other.

In the case of plant extracts that standardize for compounds found in only small amounts (e.g., hyperforin and hypericin in St. John's wort) the remaining 93-99% may be somewhat or very different in composition. This suggests that different extracts with the same amounts of these identified compounds may behave differently in the body. Companies need to compare "apples to apples" when referencing science from other companies in order not to use science in a way that is false or misleading. On the other hand it is also important to realize that if scientific studies are employed to test the validity of a traditional preparation the results of such research may be appropriately applied to other similar preparations.

NOTICE
By furnishing this backgrounder, we do not provide any opinion as to:

• The safety of any product containing this ingredient;

• The efficacy of any product containing this ingredient;

• The use of any specific brand of product; or

• The level of substantiation for either the safety or efficacy of any such product.

Neither this backgrounder nor any portion of this backgrounder may be used in advertising or promotional materials. In addition, this backgrounder does not constitute, and is not to be used as, "third party literature" as that term is used in connection with section 5 of the Dietary Supplement Health and Education Act (DSHEA).

As with any health-related product, consumers should discuss the use of any products with a health care practitioner.

REFERENCES*
1A. Creswell, J.W. (1994) Research Design: Qualitative and Quantitative Approaches, Thousand Oaks, CA, Sage Publications.
2A. Dawson-Saunders, B., Trapp, R.G. (1994) Basic and Clinical Biostatistics, 2nd ed., Norwalk, Connecticut: Appleton & Lange.
3A. Guyatt, G, et al. (1998) A Clinician's Guide for Conducting Randomized Trials in Individual Patients. Can Med Assn J 139.
4A. Egger, M., Smith, G.D. and Phillips, A.N. (1997) Meta-Analysis: Principles and Procedures. Brit Med J 315:1533-1537.
5A. The Cochrane Controlled Trials Register. In: Cochrane Library. CD ROM and online. (http://hiru.mcmaster.ca/cochrane).
6A. Cook, DJ, et al. (1997) Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions. Ann Intern Med 126:376-380.
7A. Sackett, D.L. (1981) How to Read Clinical Journals: I: Why to Read Them and How to Start Reading Them Critically. Can Med Assn J 124:555-558.
8A. Food and Nutrition Science Alliance (FANSA). (1995) FANSA's 10 Red Flags of Junk Science. www.faseb.org/ascn/fansa1.htm

*KEY TO REFERENCE HIERARCHY
A reference: research article or editorial comment in peer-reviewed journal

B reference: letters to the editor, professional personal communications, meeting/conference abstracts (NOT usually peer reviewed), and pending reviewed publications

C reference: lay/trade press publications, interviews, patents and Web site material (NOT peer-reviewed)

This hierarchy shows the relative reliability and scrutiny of the science in the references cited. In general, articles in peer-reviewed journals are subjected to critical evaluation by disinterested scientists before acceptance for publication. A backgrounder with few A references, or a preponderance of B and C references, will usually indicate a substance with little critically reviewed scientific research to support its use and/or safety.