How to Do a Critique

There are several methods of doing a critique.  Which one you choose to use depends on you background and experience.  The more experienced you become in critiquing and the more you know about research, the more detailed you will become in your analysis.  But when you are starting out, you simply need some basic guidelines to follow.  You will be surprised at how adept you become at critiquing.

Scan the Conclusions and Recommendations

Before you critique an article, even before you read the paper all the way through, scan the conclusions and recommendations.  Using your usability index, decide if you want to read the rest of the paper.  If you cannot use the findings or recommendations, you probably won’t want to use the rest of the article report either. 

If you are looking for a particular research instrument, scan the section on methods or design to see if it is there.  If you can’t use the material, you will not be interested enough in the full report to critically analyze its contents.  So before you commit yourself to an article, scan it for usability and interest.

Look for Key Terms

After selecting an article to critique, scan the entire article from beginning to end.  Look for the key terms: problem, purpose, terms, design, methods, findings, analysis, and conclusions.  Is each area of the report given a subheading that corresponds with each step in the research process?  They usually aren’t, in which case you should reread the report and underline each of the major topic headings.  If you cannot find a title to underline, then jot your own key term in the margin.  This way you give yourself an easy reference guide to all of the steps in the process.

Look for a Logical Order of Headings

Scan the order of the headings and find out if they follow the ordering a research report.  For example, can you find the introductory matter in the headings, or is it somewhere else?  Whether you had to label the content yourself or the author provided the appropriate headings, you are looking for the logical progression of the material in the report.

Look for Gaps

After a basic outline of the report’s content has been established look for gaps.  Has any area been left out?  Make a note of that to yourself. In either case, you will be looking for gaps and misplaced material as you continue your critique.

Now go back to the article and look at the section you have labeled as either “Introduction” or “The Problem”.  Read that section carefully and watch for three things: significance, and documentation of earlier work. 

• If the writing style is ambiguous, unclear (you don’t know the point being made), and inconsistent. 
• The research itself is meaningless, unsolvable, or trivial. 
• Either the documentation is missing entirely or the references are incorrect.

A substandard problem is either incomplete/unclear, of limited interest, or not fully documented. An adequate or standard problem, on the other hand, covers all the major research objectives.  The writing style is clear.  You know and understand what the research is about, and the progression of ideas is logical.  The documentation of the problem seems to be reasonably complete and is used correctly.  Finally, its significance is clear in that the problem needed solving or the results are unquestionable.

In order to make this evaluation of the problem, you must first identify it.  If the author doesn’t clearly label it, you must find it on your own, which means reading the report thoroughly.  Use your outline of the article as your guide.  If you find yourself reading about the sample or data collection, you have read beyond the problem.  Go back to the introduction and look for the problem there.  It might be stated as a question, a statement, or a hypothesis.  You decide how well or how poorly it is stated but first you have to determine what it is.

A defective sample does not represent the population in logical manner.  The sample may have nothing to do with the population.  In other instances, the sample may not be fully described.  You may feel that the author selected a biased sample from the population.  If the sample is, meaningless, inconsistent with the problem, or biased, it is defective.

In a substandard sample, the author is unclear regarding either the population or the sample.  The sample may be meaningful to the problem but not to the population.  Or, you may simply have a hunch that something is wrong within the sample.

An adequate sample is clearly specified, defined, and related to the particular population and problem being studied.  In short, it is representative.

As you recall, data collection methods are based on the problem and the sample.  Again, look for the clarity, significance, and documentation of the methods in relation to the adequacy of the report.

If the data collection methods have no relationship to the problem of the sample, or if they simply are not presented, they are defective.  Phrases like “an interview was conducted,” “a questionnaire was constructed,” and “available data were used” are insufficient in themselves.  Remember, if you cannot use the information if you cannot replicate the research on the basis of the information given, the report is defective.

Substandard methods give only partial information.  You may have a general idea of what the author has done, but not enough detail to use the information.  If the author used a reference for the method, look that reference up.  If the reference is unavailable from the usual sources and you have to write the author, the report is inadequate.  On the other hand, you may have full and complete information on the methods used, but you decide on the basis of your reading that only a partial or tentative solution can be achieved through this method.

An adequate or standard report on methods will tell you what, why and how it was done in sufficient detail that you can make and informed decision about it.

The method must be logically consistent with the problem and sample look at the age reading.  Is it this research?  If not, the relationship between problem, sample, and method is critical.  If the work is new, does the author rely on prior literature to establish the relationship between problem, sample, and methods?  Or does the article deviate completely from established sequences?  When the methods are irrelevant to the problem, the research is illogical and defective.

At this point in your critiquing, your analysis of data is rough.  Unless you understand statistics, you will have some difficulty with this part of the critique.  Nevertheless, you should look at the clarity of the reporting style, the documentation of method, and the relationship between analysis and method.

Most research reports have tables of information, charts, and graphs.  Can you find the sample adequately represented in the table?  What about the methods are they exemplified anywhere?  If you were to develop an answer to the question based on the methods and sample, what would you include?  See if the author has included this critical information.

A defective analysis does not answer the question that was asked.  Such an analysis is unclear ambiguous, unrelated to the data, or inconsistent with the rest of the research.  A substandard analysis shows bias toward one aspect of the data over another, or does not fully present an analytical tool.  The adequate analysis, on the other hand, is comprehensible, responsive to the data, and congruent with all preceding material in the article. 

Finally, you are ready to examine that aspect of the report that discuses the findings, conclusions, and usability of the research. You have read the report quite thoroughly to this point and can form an impression of the findings or conclusions.  Are they clear?   Relevant? Usable?

The findings and conclusions have to be generated from the research.  If the research makes some assumptions or conclusions that have not been adequately substantiated elsewhere in the report, you may suspect bias.  One small research project, as you know, will not solve global problems.  So look for the type of generalizations made by the author.  If they go too far beyond the research, the author probably is too egocentric.  Or, the conclusions might be too narrow or too specific.  You are, after all, looking for some creativity from the researcher.

Defective conclusions are too broad, too specific, or nonexistent.  Substandard conclusions lack completeness.  The adequate conclusion has a sense of finality and closure is derived directly from the problem.

You now know this article backward and forward.  You have an opinion on its value by the end of the report.  Now let’s see how objective that opinion is.

Go back to the beginning of the report.  Get out a pencil and paper.  Start at the top left-hand side of the paper and list each of the major portions of the research report: problem sample, methods analysis, findings/conclusions.  Across the top of the paper, list your headings: defective, substandard, and adequate.  Now check off under which heading each section of the article falls.  Add up the number of checks you have made in the defective column and multiply by 1.  Add the checks in the substandard column and multiply by 2.  Multiply the sum in the adequate column by 3.  Total the scores.  If you gave the report at least 12 points, it is adequate.  If the report scored from 8 to 11, it is substandard.  If the score fell below 8, you have a defective study.

Check your score against the feeling you had after you had thoroughly analyzed the article.  Do they agree, or is the score totally inconsistent with your feeling?  If similar, you have just verified the reliability of your feelings.  But, since feelings and impressions are not always reliable indicators of how good each aspect of the report is, use your objective scoring method until your feelings and the scores agree consistently.