Second, I would argue that the skill of literature searching, in general, may be approaching extinction. One would think that the wonderful improvements in data bases and computer search engines over the past quarter century would have dramatically improved literature searching. Instead, they may have made it worse. My conclusion derives from reading several too many research articles on standardized testing with a wholly erroneous assessment of the research base on the topic.
Too often nowadays, researchers content themselves with a superficial computer search on the most obvious keywords, typically relegating even that mundane task to research assistants or librarians who may have little understanding of the topic. Even worse, other researchers may do nothing more than cite the conclusions of one of these casual, superficial reviews, making no effort whatsoever to familiarize themselves with the research base directly.
A complete reliance on keyword searches is inadequate to the task for several reasons. First is the matter of which keywords to choose. Different folk can attribute different keywords to identify the same concept. Sometimes, the differences in wording are subtle; sometimes they are dramatic. Moreover, different research disciplines can employ entirely different vocabularies to the same topic. A net benefit to an economist, for example, may be called consequential validity by psychometricians, positive effects by program evaluators, or positive washback by education planners.
It is telling, moreover, that research articles based on extraordinarily superficial literature searches seem to have no trouble getting published in the same scholarly journals that minutely scrutinize analytic methodologies. Methodology seems to matter quite a lot; an even minimal effort to understand the literature almost not at all. Some quantitative methodologists believe that they are immune from the research disease common in the softer social sciences and humanities of respecting complexity for complexity’s sake. The reader may have heard the story of the scientist who sent a research article to several humanities journals that was entirely made up and full to bursting with mostly unexplained technical terms, that was accepted at most of those journals, … or, of the quasiexperimental study that compared the journal acceptance rate of articles that were exactly the same in content but different in terms of the technical difficulty and denseness of the prose (the more difficult to read and understand, the better chance an article had of being accepted for publication, the content held equal).
But, some quantitative researchers seem to have their own blind spots, revering complex analytic methodology, and almost completely disregarding the value of quality work in any or all of the many other essential aspects of a research project, such as the literature search. It almost seems that, although good literature searches require organization, persistence, and a wide familiarity with terms, concepts, and classifications because they are unchallenging in terms of analytic methodology, it is simply not considered important to do them well.
From Defending Standardized Tests