DEVELOPMENT CENTER FOR ASIA AFRICA PACIFIC (DCAAP)
HOW CAN WE IMPROVE THE QUALITY OF SOCIAL SCIENCE RESEARCH?*
Cesar M. Mercado, Ph. D.**
INTRODUCTION
The dwindling quality of education remains to be one of the greatest concerns of all sectors in the Philippines. The problem is systemic. It affects all levels of education including the tertiary level.
Of the three major functions of academic institutions at the higher level – instruction, research and extension – research needs most urgent attention. Among the research areas, social science research faces not only the problem of quality, but also the issue of quantity of research outputs.
Between the two issues, quantity of social science research appears to be of wider concern in most of the higher education institutions (HEI) in the country today; quality seems to be the major concern in the bigger HEI that have already a core group of social science researchers or a culture of research.
A number of academic institutions have already some provisions to solve the problem of quantity. These include training of faculty members on social science research, allotment of research budget for faculty members, and additional renumeration for faculty members who serve as thesis adviser.
Solving the quality problem facing social science research is more difficult then solving he quality concern. When it comes to quality, there are lots of truth to the adage: “If you can’t measure it, you can’t improve it†(Denton, 1992).
Realizing the importance of measurement in improving quality, this short paper presents proposed measuring tools for quality. These tools include: 1) quality standard, 2) classification index, and 3) quality index for measuring and improving the quality of social science research.
- - - - - - - - - - - - - - - -
*This paper was written by the author as a modest contribution to the improvement of the quality of social science research in the Philippines and other developing countries. Reactions and suggestions, which are most welcome, should be sent to
[email protected]**Dr. Cesar M. Mercado, Ph.D., is founding president and chief executive officer of the Development Center for Asia Africa Pacific (DCAAP), a Manila-based international training and consulting non-profit organization. He is a former professor and researcher of the College of Mass Communication, University of the Philippines and fellow of the Development Academy of the Philippines and consultant to numerous local and international organizations. He joined the UNDP Asia Pacific Programme for Development Training and Communication where he served as planner/programmer and monitoring/evaluation for ten years. Now he continues to manage DCAAP and act as consultant and trainor to local and international clients.
NEEDS TO IMPROVE QUALITY OF SOCIAL SCIENCE RESEARCH
Feedback from training participants and requests from some universities and colleges suggest that improving the quantity and quality of social science researches is of prime importance.
As mentioned earlier, some schools have put some allocations to encourage faculty members to conduct social science research. However, there are very few or no takers of the funds at the moment. These schools need some interventions that will further encourage faculty members to use the available research budget such as needed training, awards, promotion, etc.
Increasing number of academic institutions now require their undergraduate and graduate students to write their thesis as a requirement for graduation. However, they need more faculty members who can act as thesis advisers.
Some universities and colleges have already faculty members who do research and act as thesis advisers. But they want now to develop a culture of research in their institutions. At the same time, they want to improve the quality of their social science researches and the thesis of their students without much expense.
There are also schools of higher learning that have already core of social science researchers who are now in need of quality standard for social science research and classification standard for student thesis at the undergraduate, masteral and doctoral levels. They also need quality index for rating the quality of social science research report and thesis manuscript. In response to these needs, we have presented in this paper proposed:
• Social science research quality standard (SRQS)
• Social science thesis classification index (STCI)
• Social science research quality index (SRQI)
These newly developed prototype social technologies, derived from classical scientific knowledge agreed upon by scientists, are ready for testing and refinements.
SOCIAL RESEARCH QUALITY STANDARD AND INDECES
A standard is usually defined as a model to be followed or initiated, established by custom and consent (New Lexicon Webster’s Dictionary of the English Language 1990).
An index is usually defined as a pointer on a dial or measuring instrument (New Lexicon Webster’s Dictionary of the English Language1990).
We define Social Science Research Quality Standard (SRQS) as a model in improving the quality of social research studies from planning to reporting based on the best features of social research reports accepted by many authorities in the field of study.
We define Social Science Thesis Classification Index (STCI) as a model in classifying and defining social science theses according to the levels of education such as undergraduate, masteral and doctoral. It is also useful in improving the quality of social science theses at each level.
We define Social Science Research Quality Index (SRQI) as a measuring instrument used for rating the quality of social science theses that are already classified by level-undergraduate, masteral or doctoral level. With some adjustments, this index could also be used in rating the quality of social science research reports of researchers.
USES OF THE SRQS, STCI AND SRQI
The SRQS will be useful to social science research institutions and units in improving the quality of research of their staff. It will also be useful in determining the quality of their completed studies.
To academic institutions, requiring thesis for undergraduate, masteral and doctoral students, the STCI will be useful in establishing standards for each level. For instance, with STCI, these institutions could use qualitative criteria, rather than quantitative criteria, in classifying thesis for different educational levels.
At present, many social science colleges and thesis advisers are asking how they can more clearly delineate the thesis suitable for undergraduate, masteral and doctoral levels. In the absence of quality standards, some of them use quantity standards such as the thickness of the manuscript.
A common practice among many thesis advisers is to expect undergraduates to submit thin thesis manuscripts; masteral students, thick thesis manuscripts; and doctoral students, thicker dissertation manuscripts.
To meet these quantitative standards, many undergraduate theses students cite few review of literature; masteral students, more review of literature; and doctoral students, much more review of literature.
The problem with quantitative standard is that what is thin, thick or thicker thesis manuscript is not operationally defined in terms of number of pages. Thesis advisers know that quantitative standards are not defensible and acceptable to social science researchers. Except for a few social science colleges and units, most tertiary schools do not have yet some qualitative standard to use as guide in improving the quality of their research outputs and to classify the thesis of their students.
The SRQI is useful in rating the internal logic and consistency of the different parts of each individual social research report or thesis manuscript at each level such as undergraduate, masteral and doctoral. It can also be used to improve the draft research manuscript before it is finally submitted for whatever purpose it is intended.
PROPOSED SOCIAL RESEARCH QUALITY STANDARD (SRQS)
This brief paper proposes a sample quality standard that hopefully will be useful to colleges in the social sciences and social science researchers/thesis advisers in improving the quality of their studies and thesis of their students.
A good defensible quality standard (QS) should be based on knowledge-based or consensus-based criteria or features agreed upon by authorities in the field. In science, we should use criteria in developing quality standard (QS) agreed upon by scientists.
Most scientists agree that the quality of research depends on its contribution to the goals or objectives of science (Reynolds, 1971). The five basic goals or objectives of science are to:
1. Organize and categorize things (a typology)
2. Explain past events
3. Predict future events
4. Provide a sense of understanding
5. Potential to control events
The greater the contributions of a study to the goals of scientific knowledge, the higher its quality. A study that could contribute to the five criteria enumerated above could be considered of the highest quality; while a study that could contribute to only the fist goal could be considered of lowest quality. The same criteria could be applied in determining the quality of social science research report and thesis manuscript in the field.
We have realized that the capacity of a social science study to contribute more to the goals of science depends on a package of the best elements in planning, conducting and reporting the study. When the package is used as guides in carrying out a social science study or social science thesis, the output is likely to be of better quality. Here are the best elements expressed in generalizations, that should be observed in planning, conducting, and reporting a study:
1. Types of Study
• Explanatory casual studies are better than explanatory relational studies or exploratory descriptive studies. Explanatory casual studies can contribute to all the five goals of science, while explanatory descriptive study can contribute only to the first goal of science (a typology). The explanatory relational study can contribute to, at most three or four goals of science cited earlier.
2. Types of Study Objectives
• Casual objectives are better than relational or descriptive objectives. The casual objectives have greater capacity to contribute to the five goals of science than either the relational objectives or the descriptive objectives.
3. Research Method
• The experimental method is better than the survey, focus group discussion or case study. The experiment is the only research method that emphasizes on control and demonstrates cause and effect relationship between two or more variables. The other methods could only provide typology or help explain past events or predict future events.
4. Research Design
• The pretest- posttest design usually used in experiment and its variations are better than any of the design used in survey (e.g. after only, before-after or static group comparison). The pretest-posttest design can contribute to all the five goals of science while the other research methods used by survey could not provide the same contributions. The focus group discussion and the case study do not give much emphasis on design.
5. Research Tool
• The questionnaire or interview schedule designed for experiment is generally better than the questionnaire or interview schedule designed for survey. The former is commonly designed to gather nominal, ordinal and interval data while the latter is used for gathering only nominal or ordinal data. We know that the nominal and ordinal data simply classify and rank order variables, but the interval data can provide altogether nominal, ordinal and interval data. The guide questions for case study and focus group discussion provide only descriptive information but not numerical data.
6. Sampling Scheme
• The stratified sampling scheme is considered better than the simple random sampling and multi-stage sampling, systematic sampling, cluster sampling or multi-stage sampling. The stratified sampling gives equal representation to all significant stakeholders in the actual samples, which could not be done by the other sampling scheme.
7. Types of Data
• As mentioned earlier, the interval data are better than the nominal or ordinal data because it could classify, rank order and show the difference while the other two could only classify (nominal) or rank order (ordinal data). Interval data can also become the bases for the ratio, which is usually considered better than interval data.
8. Statistical Analysis
• The regression (r) is better than the analysis of variance (ANOVA), t-test (t), or chi square (x²) because multiple regression could show the significant effects or relationship of 3 or even hundreds of variables on a single dependent variable; while the other statistical analysis can deal with one or two variables only at a time. But this should not tempt us to think that using the r analysis alone will significantly improve the quality of the whole research report.
FEATURES OF HIGHEST QUALITY SOCIAL RESEARCH
Based on the analysis, we can say that the highest quality social science research or thesis manuscript observed the following package of best features that could meet the five goals of science:
1. Explanatory casual study
2. Casual objectives
3. Experimental method
4. Pretest-posttest design or its variations
5. Questionnaire or interview schedule designed to gather nominal, ordinal and interval data
6. Satisfied sampling scheme
7. Interval data
8. Regression analysis
On the other hand, the lowest quality research report or thesis manuscript possesses the following features:
1. Exploratory descriptive study
2. Objectives designed only to classify and describe
3. Case study or focus group discussion
4. After only design
5. Guide questions
6. Accidental or incidental sampling
7. Nominal data
8. Percentage or mean score only
Let us remember that these features to be used in rating the quality of social science research report or thesis manuscript were based on its contribution to the goal of science and not necessarily to its immediate useful contribution to action such as the solution to the broader social, economic, political and environmental problems.
Based on our science-based criteria, there are low quality social science research report that might be more useful to data users in planning, decision-making and/or policy making than high quality social science research report. Many planners, decision makers and policy makers, at a glance, might drop a high quality social science report because the features in the report could not easily be understood and appear threatening. Thus its use may be delayed.
CLASSIFYING THESIS MANUSCRIPT
As mentioned earlier, there are three levels of thesis that are required of thesis students in growing number of universities and colleges in the Philippines and elsewhere, even in more developed countries.
Some university and college social science researchers and thesis advisers are increasingly concerned about the issue on the substantive differences among undergraduate, masteral and doctoral theses. They are now feeling uneasy about the thickness of the manuscript as the main criteria in judging the classification of a thesis manuscript.
The standard we have established in guiding and measuring the quality of social science research reports provide us useful information in classifying systematically the thesis of different levels of thesis students in a more logical way than the varied present system they used in thesis advising.
Based on our analysis of the goals of scientific knowledge and the features of high and low quality studies, we can now prepare a better way of classifying thesis based on quality.
• Bachelors degree thesis should possess the following features:
- Could be classified as exploratory, descriptive study
- Objective is to organize, categorize and explain phenomena or problems
- Use case method and/or descriptive survey
- Use the after only design
- Use simple questionnaire or guide question to generate typologies
- Use non-probability sampling such as accidental, purposive or simple random sampling
- Use nominal and ordinal data
- Use percentage, mean score and/or chi square in analyzing data
• Masteral Degree thesis should possess the following features:
- Could be classified as explanatory, relational study
- Objective is to classify, predict and explain phenomena
- Use survey supplemented by case study and/or focus group discussion
- Use before-after or static group comparison design
- Use more elaborate questionnaire or interview schedule supplement with guide questions
- Use probability sampling scheme such as simple random or systematic sampling
- Use nominal, ordinal and/or interval data
- Use mean score, chi-square and/or t-test
• Doctoral degree dissertation should possess the following features:
- Could be classified as explanatory, relational and/or causal study
- Objective is not only to classify, predict and explain, but also to help understand and/or control phenomena
- Use survey or experimental method, supplemented by case study and/or focus group discussion
- Use before-after or static group comparison if method used is a survey; post-test only control group design if method used is an experiment
- Use more elaborate questionnaire or interview schedule
- Use probability sampling such as stratified sampling or multi-stage sampling
- Use nominal, ordinal, interval and ratio data
- Use mean score, t-test, ANOVA or regression
SOCIAL SCIENCE RESEARCH QUALITY INDEX
We have also developed a proposed Social Science Research Quality Index (SRQI) which is useful in measuring the quality of individual social science research report or social science thesis manuscript.
The index was based on a 50-item/30-item checklist/scorecard developed by this author for rating social science thesis manuscript at all three-levels—undergraduate, masteral and doctoral (Mercado, 2006).
This article went a step further in using the scorecard. The scores are already matched with the grading system used in many universities and colleges in the Philippines. The attached sample next page shows the index’s application in grading undergraduate and graduate thesis manuscript.
SAMPLE CHECKLIST FOR DETERMINING THE QUALITY
OF SOCIAL RESEARCH/ THESIS MANUSCRIPT*
The checklist is designed to measure the quality of a draft or final social research/thesis manuscript using the survey or experimental designs. There are 50 items to be answered by either a YES or NO by a rater. The number of YES constitutes the score of the manuscript. The equivalent grade of the score is shown in the table on survey thesis manuscript.
About 15 questions for the survey might not apply to the case study. Thus, the highest score for case study is 35 instead of 50. The equivalent grade of the score of a case study is shown in the table on case study thesis manuscript
Type of study: ________ Case ________ Survey
Name of rater :
Position :
Date :
• TITLE YES NO
1. Are the main factors/ variables found in the title?
2. Does the title tell readers about the type of study (whether case or survey)?
• INTRODUCTION
3. Does the introduction explain the theoretical importance of the study?
4. Does it explain the practical importance of the study?
5. Does it include the problem(s) the study intends to answer?
• PROBLEMS
6. Is the research problem consistent with the action objectives of the project to be evaluated?
7. Does the research objective contain key factors or variables found in the research objective and action/ project objective?
• HYPOTHESIS*
8. Do the hypotheses answer each research objective?
9. Are there equal numbers of hypothesis to the number of education research objective?
• FRAMEWORK*
10. Is there a theory or principle that served as the foundation for the hypothesis?
11. Are the key factors (KF) or variables identified?
12. Are the key indicators (KI) of each KF identified?
13. Are the key questions (KQ) for each KI identified?
• REVIEW OF LITERATURE
14. Is the review of literature focused on research findings that deal with the key factors/ variables in the current study?
• DEFINITION OF VARIABLES YES NO
15. Are the key variables found in the research objectives operationally defined at the behavioral or measurable level?
• LIMITATION OF THE STUDY
16. Are the limitations of the present study mentioned?
• METHOD
17. Is the main method in gathering data appropriate to the research objective?
• DESIGN*
18. Is the design appropriate to the research objective?
• TOOL
19. Is the questionnaire/ interview schedule/ guide question appropriate to the research objective?
20. Does it contain all the key questions fund in the O-KFIQ matrix?
21. Was the tool pretested before data gathering?
• SAMPLING
22. Is the population of the study cited?
23. Is the chosen sampling scheme appropriate to the research objective?
24. Is the process in choosing the sample size clearly explained?
25. Are the sample type and size mentioned?
• DATA GATHERING
26. Are the position and the number of people who collected the data mentioned?
27. Were they trained before they collected the data?
28. Are the inclusive dates for the data gathering cited?
29. Are the areas covered by the data gathering cited?
• DATA CATEGORIZATION*
30. Are the categories under each independent or dependent variable consistent with the research objectives?
31. Are the categories independent of each other?
• DATA TABULATION*
32. Is there an equivalent table for each objective?
33. Are the main tables consistent with each research objective?
34. Are the categories in the tables on nominal data arranged from those with highest frequency to those with the lowest?
35. Are the single response and the multiple response tables properly presented?
• DATA ANALYSIS*
36. Was the appropriate statistical analysis used to analyze the qualitative data?
37. Was the appropriate logical analysis used to analyze the qualitative data?
• DATA INTERPRETATION YES NO
38. Does the interpretation of table data contain a generalization?
39. Is the generalization supported by the data in the table?
40. Does the data interpretation focus on extreme data (highest and lowest) rather than all the data?
41. Were the percentages cited in the data interpretation rounded off?
42. Is the interrelationship among the different tables explained?
43. Were the findings of prior related studies cited to explain present findings?
• SUMMARY AND CONCLUSION
44. Were all research objectives answered?
45. Were the hypotheses, if there were any, accepted or rejected?
46. Did the manuscript mention whether or not the project was successful?
• RECOMMENDATIONS
47. Are the recommendations in line with the results/ findings?
48. Are there recommendations for action to be implemented by concerned agencies?
49. Are there recommendations for further research?
• ABSTRACT/EXECUTIVE SUMMARY
50. Does the abstract contain the main parts of the manuscript: rational, main objective, key variables, summary and conclusion, and recommendations?
*These parts are not usually relevant to case studies.
----------
Source: Cesar M. Mercado, A New Approach to Thesis Writing. Development Center for Asia Africa Pacific (DCAAP), Quezon City, Philippines, 2006.
DEVELOPMENT CENTER FOR ASIA AFRICA PACIFIC (DCAAP)
SCORING AND GRADING SYSTEMS FOR
MEASURING THE QUALITY OF UNDERGRADUATE
AND GRADUATE SURVEY THESIS MANUSCRIPT*
A. UNDERGRADUATE SCORING AND GRADING SYSTEM
SCORING SYSTEM GRADING SYSTEM
1-5 5.0
6-10 4.0
11-15 3.0
16-20 2.5
21-25 2.25
26-30 2.0
31-35 1.75
36-40 1.5
41-45 1.25
46-50 1.0
B. GRADUATE SCORING AND GRADING SYSTEM
SCORING SYSTEM GRADING SYSTEM
1-10 2.0
11-20 1.75
21-30 1.5
31-40 1.25
41-50 1.0
- - - - - - - - - - - - - - - -
*Based on the 50-item checklist developed by Dr. Cesar M. Mercado, Ph.D., in his book, “A New Approach to Thesis Writingâ€, Development Center for Asia Africa Pacific (DCAAP), Quezon City, Philippines 2006. The scoring system refers to the score earned by a thesis manuscript using the 50-item scorecard shown next page. The grading system refers to the equivalent grade of the score as used in most universities and colleges in the Philippines.
DEVELOPMENT CENTER FOR ASIA AFRICA PACIFIC (DCAAP)
SCORING AND GRADING SYSTEMS FOR
MEASURING THE QUALITY OF UNDEGRADUATE AND
GRADUATE CASE STUDY THESIS MANUSCRIPT
A. UNDERGRADUATE SCORING AND GRADING SYSTEM
SCORING SYSTEM GRADING SYSTEM
1 – 3 5.0
4 – 6 4.0
7 – 9 3.0
10 – 12 2.75
13 – 15 2.5
16 – 18 2.25
19 – 21 2.0
22- 2 4 1.75
25 – 27 1.5
28 – 30 1.25
31 – 35 1.0
B. GRADUATE SCORING AND GRADING SYSTEM
SCORING SYSTEM GRADING SYSTEM
1 – 7 2.0
8 – 14 1.75
15 – 21 1.5
22 – 28 1.25
29 - 35 1.0
-----------
*Based on the 35 – item checklist for case studies developed by the same author.
SUMMARY
This brief article discusses some solutions to the twin problems of quantity and quality facing social science research in higher education institutions (HEIs) in the Philippines.
It focuses on the development of three instruments designed to improve the quality of social science research and thesis writing in the country. The process in developing the three instruments are explained and demonstrated in this brief article.
The newly developed instruments that are expected to contribute to the improvement of the quality of social science research and thesis manuscripts are: 1) Social Science Research Quality Standard (SRQS), 2) Social Science Thesis Classification Index (STCI), and 3) Social Science Research Quality Index (SRQI).
The SRQS is used for raising the quality of social science research or thesis during planning based on certain features derived from the goals of scientific knowledge.
The STCI is used for classifying thesis proposal during planning by level of education: undergraduate, masteral and doctoral.
The SRQI is used for rating the quality of the draft and for final copy of the thesis manuscript or social science research report at each level based on a 50-item/30-item checklist developed earlier by this author.
The instruments may be modified and used for other purposes depending upon the creativity of the user.
The instruments presented in this brief article are circulated at this stage to interested parties for reactions and suggestions before publication. Feedback from recipient of this article will be greatly appreciated.
REFERENCES
1. Denton, D. Keith, How to Give Quality Service to your Customers, UBS, New Delhi-110 002, 1992
2. Mercado, Cesar M. A New Approach to Thesis Writing, Development Center for Asia Africa Pacific, Quezon City, Philippines 2006
3. Reynolds, Paul Davidson, A Primer in Theory Construction, Bobbs-Merril Educational Publishing, Indianapolis, 1971
4. The New Lexicon Webster’s Dictionary of the English Language
POSTED BY:
FIEL ANGELI ESPEJO ARAOARAO – GABIN
DCAAP ALUMNA
MAY 2002, UNIVERSITY OF THE PHILIPPINES
Linkback:
https://tubagbohol.mikeligalig.com/index.php?topic=8275.0
Topic: HOW CAN WE IMPROVE THE QUALITY OF SOCIAL SCIENCE RESEARCH?(Dr. Cesar M. Mercado) (Read 1401 times)