Friday, June 14, 2013

On Methodological approach to Assess Research Output

This a short discussion note, very draft and unedited, on the methodological approach related to the assessment of all research outputs within the centre for public enterprises (CPE), University of the Witwatersrand. I hope this small discussion of mine will add at least modicum of value in an attempt and effort of the centre to improve the quality of research outputs.


Concerning bias intellectually, ideologically or politically


As in the domain of social science, researches on public enterprise are generally qualitative studies, which apply numerous approaches, paradigms, schools and movements encompassed in terms of the ontological, epistemological and methodological assumptions on which they are based. The point is that subjectivity is unavoidable and should not try to avoid it, but needs to be justified in logical and convincing ways.


For instance, there are theoretical and ideological bias relating to the existence or necessity of public enterprises or the desirable size and roles of public sectors in an economy of a nation is natural and unavoidable and should not be afraid of it but just to be aware of it when assessing studies and researches.


Then, a question concerning the role of reviewers arises. 



Who review the reviews of reviewers?


Any academic research article or paper or even book-chapter has to be reviewed by anonymous reviewer or a couple of reviewers before they get published. The due consideration should be given to the matter of choosing reviewers. I would like to suggest that there should be at least two or at best three reviewers, so that their reviews and evaluations should be also assessed and checked against one another. 



Then, what are the characteristics of great journal articles?


To be able write or publish great or at least good articles, the very first step is to know what a great or good article is, what their characteristics and strengths that make them outstanding. According to Litman (2012: 3), “research quality is an epistemological issue (related to the study of knowledge)”. To write or produce good research papers and articles, thorough consideration and conceptualisation, careful preparation and planning, credible and reliable data and information, in-depth and systematic data organisation and analysis are required. However, it also seems to me that there are two important ingredients that make an article or paper make outstanding and significant. These are the question that the research asks and tries to answer or understand, and the impacts that has on the knowledge creation and our understanding about the particular subject. The impact of a research or a article or a paper, in my view, stems from questioning the established conventional beliefs and theories is, in my view, one of the essential characteristics of a great research output.




On questions that are simple but important

In my view, a great research or research journal starts with a simple but important question which questions the conventional beliefs or views or even theories. Most of the new discoveries that revolutionise our thinking and understands are started with a simple but overlooked questions, for example, Galileo’s question that asks “why the Moon orbits around the Earth, while other planets, including the Earth, are orbiting around the Sun” (Millican, 2009). Similarly, Newton asks a simple question: “Why an apple but not the Moon falls down to the ground”.



Logically sound but untrue 


Logical analysis and reasoning are bounded by known facts. In the time when there was no telescope and unable to travel around the world, it looked very logically sound and convincing the structure of Aristotle’s Universe. However, when we have more advanced tools, instruments and technologies, then, in turn, acquired more completed information and data, assumptions and theories that are previously sound and logical become illogical and untenable to answers questions that arised together with new data and information, for example look at two systems of Universes, by Aristotle and by Ptolemy.




Figure 1: Aristotle’s Universe versus Ptolemy’s Universe


  
Aristotle’s Universe
Image from Galileo of Physics department of the University of Virginia,  http://galileo.phys.virginia.edu/classes/109.mf1i.fall03/AristotleToCopernicus_files/image001.gif

Ptolemy’s Universe
Image from From Physics department of the University of Wisconsin (Madison),
http://wisp.physics.wisc.edu/astro104/lecture5/lec5_print.html


Another instance is between Newton and Leibniz on the concept and nature of space, time and motion, and their relationship among each other. While Newton Laws were well-established and accepted, Leibniz argues, against Newton, that space, time and motion are relative, not absolute. In one of his letters to Clarke, Leibniz wrote "As for my own opinion, I have said more than once, that I hold space to be something merely relative, as time is, that I hold it to be an order of coexistences, as time is an order of successions" (Leibniz, 1956: 25-26). Einstein (1905) later proved that Leibniz is damn right in his historic paper that does not have any references to any other publications, although it uses many of the ideas that had already been published by others, however, its substantial contribution is an introducing of a theory of time, distance, mass, and energy.



Portrait of Gottfried Leibniz (1646-1716), German philosopher (oil, circa 
College of Liberal Arts & Social Sciences,  University of Houston



Portrait of Isaac Newton (1642-1727) (Oil, 1689)
Image from Isaac Newton Institute for Mathematical Sciences,



Albert Einstein
From National Public Raido (NPR), 
http://www.npr.org/blogs/13.7/2011/09/28/140839445/is-einstein-wrong


Another memorable example includes a case between “Phillips curve”, and then the concept of NAIRU (Natural Rate of Unemployment). While people were so convinced by and comfortable with the theory of “Phillips curve”, there were two persons Friedman (1968) and Phelps (1967) who dared to question the validity of it, and are also capable to test it against the data from different contexts, and finally repudiate it once and for all. The point of Phillips curve is simple: there is a negative relationship between the unemployment rate and the price level. When the unemployment rate declines, the overall price level (i.e., inflation rate) rises (Phillips, 1958).
Important works are simple but very original and different from the majority of the articles and papers.

Some articles and papers could be very convincing logically and analytically but it does not mean that their data or evidence or results or conclusion is right. There are famous fake scientific and research articles that were first accepted as great and later found out their fakeries (Litman, 2012: 12; Scheer, 1988; Nash, 1996). Litman (2012: 2) calls them “manipulating researches”, which are constructed with, in the terms of Frankfort (2005), bullshit or manipulative misrepresentations). Frankfort argues that bullshit (manipulative misrepresentations) is worse than an actual lie because it denies the value of truth. “A bullshitter’s fakery consists not in misrepresenting a state of affairs but in concealing his own indifference to the truth of what he says. The liar, by contrast, is concerned with the truth, in a perverse sort of fashion: he wants to lead us away from it.”


On the other hand, an article or a paper, which seems to be "dull . . . really dull" and language composition could be lower quality and standard, could also be a seminal work based on its ingenious methodological approach, or its important question and findings, or its impacts on future studies and knowledge creation, for example Keynes’s General Theory, that, according to Paul Samuelson (cited in Moggridge, 1992, cited again in Strathern, 2002: 281) is "badly written, poorly organised .... it is arrogant, bad-tempered, polemical and not overly generous in its acknowledgements. It abounds in mare's nests and confusions". However, Samuelson continues, "in short, it is a work of genius" and Strathern (2002: 281) regards it as the best single-volume biography of Keynes’ life, times and ideas.


It is, according to Nash (1996: 67), because of a fact that research journal or paper writing involves two skills: research and writing. The point is that there are two parts in a research journal or paper their qualities need to be assessed. In my view, theoretical and methodological is more important than the language composition because it can easily improved by language editors. In short, my point is that the prime focus should be on the assessment of research problem formulation and theoretical and methodological choices, rather than of language composition or presentation.



Then, what should we do? 


The first step is to understand what outstanding research papers or articles are and then share these knowledge among us. In order to do them, we have to, need to, prepared and are ready to learn from the masters and stand on the shoulders of giants, as Newton did, and then to imitate them and follow their foot-steps. There is no new thing under the Sun. Best practices and lessons of experiences are readily available if we know how to find them. 

Matters concerning data collection

One of the life-blood of a research and a research journal or paper is data collection. Its quality hinges on the quality of data it used and on the soundness of data-analysis. There are a few matters that I am aware about data-collection. 



Validity of data versus confidentiality and anonymity of respondents


A conflict can be arisen from these standard practices of researching. When respondents demand not to be named and also not to be tape-recorded, the researcher, then, has to rely only on his or her note. Then how can reviewers or examiners validate the authenticity, defensibility, rigour and credibility of the data provided by the researcher? 



Limited usefulness of company’s reports


It is a concern about the reliability of data used in the study. Companies’ reports are necessary to be studied but not reliable; that should be reviewed them cautiously and critically. It is because “companies report what they did, not what they did not do. So finding errors of omission is inherently harder than finding errors of commission” (Carroll & Mui, 2008: 87). Triangulation in terms of data collection is, therefore, indispensible.



Necessity to be aware of differences among cases that are compared


When providing lessons of experiences or best practise, there are some points that need to be aware of and considered carefully. Constraints and challenges of enterprise could be quite similar but they can be different to one another in terms of, for example, types as well as sizes of industries, cultural, social and even religious backgrounds, political backdrops, and institutional structures. 


The same mechanism, tools and processes, as well as capacity, would not always produce the same results in different organisations that have different organisational cultures, and different attitudes of people in them. Simon (1991) also argues that a successful organization is one that musters organizational loyalty among its members, as otherwise the organization would have to spend enormous amounts of time and resources in bargaining and monitoring their performances. There is one interesting and instructive illustration that supports that argument. It is a remark from a top executive of the famous Japanese company Kobe Steel, which is retold by Chang and Singh (1997: 868): The Japanese executive claimed that because of the diversity and complexity of the company's operations, there was no way board members and management could make an informed decision about most of the projects that their staff present to them. However, it is necessary for them to worry and they are not worried because they believe that their company members were mostly doing their best to advance the interest of the company.  


Another consideration needs to be make is about the nature and practices of management of an enterprise. There are quite well-known theories concerning management nature, called Theory X and Theory Y assumptions of manager or management. Performance of an enterprise or an organisation is largely dictated by the nature, practices and quality its management. Managers have their own assumptions about their staff.



Creating an encouraging environment


Another thing what we should try and succeed is to create an environment that are conducive and supportive to researchers and students and, in turn, empower, enable and assist them to be able to produce, write and publish great research papers and articles.


In history, we can see that great thinkers, writers and persons who creates outstanding works tend to come from a team that is comprise with like-minded, so brilliant, people who motivate, inspire, challenge and push to higher level of standard each other. There are plenty of examples of it, such as the Bloomsbury Group grouped some of the finest and brightest authors, philosophers and economists such as John Maynard Keynes, G. E. Moore, Bertrand Russell. E. M. Forster, Lytton Strachey, Virginia Woolf; the Austrian School of economics that is composed by great Austrian economists such as Ludwig von Mises, Friedrich von Hayek; the Chicago school of economics that are a home of Frank Knight, Aaron Director, George Stigler, Milton Friedman, just to name a few of them, where they met together in frequent intense discussions that helped set a group outlook on economic issues, based on price theory, and that has fielded more Nobel Prize laureates and John Bates Clark medalists in economics than any other university or even a country.  



The house of the Bells, Charleston Farmhouse, 
in Bloomsbury where the group used to get together.
Image from Periwork (English online learning resources & News database portal,
http://www.periwork.com/peri_db/wr_db/2006_February_18_10_21_43/part1.html



The Bloomsburg Group
Image from James Madison University,
http://www.jmu.edu/honorsprog/abroad.shtml



It is important to understand that these groups are first thought on, conceptualised, mobilised, organised, founded, nurtured and sustained by a person or a couple of persons who are visionary and forwarded minded. In the case of Bloomsbury Group, it was G. E. Moore; Paul Samuelson and Robert Solow had transformed the MIT (Massachusetts Institute of Technology) to become a great economic department. Similarly, those were Carl Menger Ludwig von Mises, and Friedrich von Wieser who founded and developed the Austrian School of economists.


Conclusion
To recap, to improve research output as well to groom and mentor researchers and students to become outstanding researchers, so that they can produce remarkable and significant research papers and articles, it is important and indispensible to understand what great research papers and articles are and then to build an environment that are conducive, supportive and encouraging, stimulating to have a vibrant and powerful research society or team.


While writing this note, I tried to remember to ask regularly myself a question “who do you think you are to talk and discuss these issues?” This is just sharing what I think important and just means to provoke a brain-storming or more debates. And I believe that at least one point of my discussion in this note will be of some help for at least one person.







References 


Einstein, A. (1905) On the Electrodynamics of Moving Bodies. In Annalen der Physik, 17 (1905), pp. 891-921. (English translations by W. Perrett and G.B. Jeffery from the German Das Relativatsprinzip, 4th ed., published by in 1922 by Tuebner). Retrieved on 6 June 2013 from http://www.pitt.edu/~jdnorton/teaching/HPS_0410/ assignments/04_origins/On-the_electrodynamics/index.html
Frankfort, H. G. (2005) On Bullshit. Princeton: Princeton Press.
Friedman, M. (1968) The Role of Monetary Policy. In American Economic Review, 58 (May 1968), pp. 1-17. Retrieved on 6 June 2013 from www.aeaweb.org/aer/top20/58.1.1-17
Litman, T. (2012) Evaluating Research Quality: Guidelines for Scholarship. Retrieved on 5 June 2013 from www.vtpi.org/resqual.pdf
Moggridge, D. (1992) Maynard Keynes: An Economist’s Biography. London: Routledge.
Nash, T. (1996) The Fake Research Paper: A Creative Approach to Modern Language Association Style. In English Education, Vol. 28, No. 1 (Feb., 1996), pp. 67-71. Retrieved on 6 June 2013 from http://www.jstor.org/stable/40172889
Millican, P. (2009) General Philosophy. A series of lectures delivered by Peter Millican to first-year philosophy students at the University of Oxford. The lectures comprise the 8-week General Philosophy course and were delivered in late 2009, which are watchable and downloable at http://www.philosophy.ox.ac.uk/podcasts/general_philosophy
Phelps, E. (1967) Money Wage Dynamics and Labour Market Equilibrium. In Journal of Political Economy, 75 (July-August 1967), pp. 678-711. Retrieved on 6 June 2013 from http://stevereads.com/papers_to_read/money-wage_dynamics_and_labor_market_equilibrium.pdf
Phillips A. W. (1958) The Relation between Unemployment and the Rate of Change of Money Wage Rates in the United Kingdom, 1861-1957. In Economica, New Series, Vol. 25, No. 100 (Nov., 1958), pp. 283-299. Retrieved on March 22, 2013 from  http://www.jstor.org/stable/2550759
Scheer, S. C. (1988). The fictitious term paper. In Journal of Teaching and Writing, 12, 223- 229. Retrieved on 6 June 2013 from http://journals.iupui.edu/index.php/ teachingwriting/article/view/940/907
Strathern, P. (2002) Dr Strangelove’s Game: A Brief History of Economic Genius. London: Penguin.