Bianca's advisor on draft version 0: "It's not clear. Make it more clear."
On v1: "It's quite vague..."
On v2: "Things are ambiguous..."
On v3: "It's not quite clear..."
On v4: "Things are not well-defined..."
...
On v23: "Things are still fuzzy..."
Bianca: "Did you ever consider getting your eyes checked?"
Friday, March 09, 2012
Thursday, February 23, 2012
Normal Science
Earlier, I had written on the how I thought it wrong to force academic researchers in Computer Science to promise technology transfer and collaborations with the industry. The crux of my argument was that industry and academia have different goals. I want to elaborate on that some more, especially from an academic perspective.
I think academic researchers are too hard on themselves when they judge the merit of their contributions in terms of adoption by industrial practitioners or whoever the perceived end-users are thought to be or by the amount of money they have saved someone. Those are arguably the standards that an industrial software firm would apply to judge the success of its product. Should academic researchers apply the same standard?
I think not.
As Kuhn famously pointed out, the researcher works in a community. The community judges the merit of a researcher's claims based on the paradigm the community follows. The paradigm is everything. The open problems comes from the paradigm. The solutions ought to make sense in the paradigm. A solution that makes sense may open up new problems, which then becomes part of the paradigm. That is what Kuhn calls normal science (as opposed to a paradigm shift, which is not the point of this post). A contribution then amounts to solving a problem within the paradigm.
Clearly, the problems of a paradigm cannot be generic things such as "to make contribution to social welfare" or "to have impact on industry practice" or "to foster end-user adoption". They have nothing to do with a particular paradigm. The problems of a paradigm are always technical in nature. Then why are researchers tempted to judge their contributions by standards that don't apply to them?
You will be surprised how many times a presenter is asked by another researcher from the audience, "...but how much money did you save anyone by applying your techniques?" (I had this asked of me once many year ago and even though I was relatively green, the question felt wrong to me even then), and how many times a researcher touts what he saved the world. Isn't this anti-intellectualism?
I suspect one reason is the excessive focus on churning out publications. Many researchers have become adept at churning them out. I think in many areas of Computer Science, this has the adverse effect of blurring the paradigm. Few people really know what others are doing, except superficially. (I am pretty sure there are some who don't even know what is in the paper that bears their name.) So in this sense, some researchers have unknowingly stopped judging contributions by the criteria of the relevant paradigm. And when that happens, they start resorting to the other non-paradigm criteria, by someone's else criteria.
I do not begrudge a researcher his success in commercializing his research or taking it to the masses. In fact, it is to be applauded, for it points to an additional skill-set that he has. But let us not make that the criteria of judging someone's research contributions.
I also have nothing against those who collaborate actively with industrial partners. To each his own. But that is no reason to apply non-paradigm criteria to those who don't.
At the beginning of his book "The Tacit Dimension" Polanyi mentions how he was struck by how communism required that scientists turn to solving the problem of the current five-year plan. It makes me wonder if he would have found questionable the constant harping about researchers delivering what the industry wants.
I think academic researchers are too hard on themselves when they judge the merit of their contributions in terms of adoption by industrial practitioners or whoever the perceived end-users are thought to be or by the amount of money they have saved someone. Those are arguably the standards that an industrial software firm would apply to judge the success of its product. Should academic researchers apply the same standard?
I think not.
As Kuhn famously pointed out, the researcher works in a community. The community judges the merit of a researcher's claims based on the paradigm the community follows. The paradigm is everything. The open problems comes from the paradigm. The solutions ought to make sense in the paradigm. A solution that makes sense may open up new problems, which then becomes part of the paradigm. That is what Kuhn calls normal science (as opposed to a paradigm shift, which is not the point of this post). A contribution then amounts to solving a problem within the paradigm.
Clearly, the problems of a paradigm cannot be generic things such as "to make contribution to social welfare" or "to have impact on industry practice" or "to foster end-user adoption". They have nothing to do with a particular paradigm. The problems of a paradigm are always technical in nature. Then why are researchers tempted to judge their contributions by standards that don't apply to them?
You will be surprised how many times a presenter is asked by another researcher from the audience, "...but how much money did you save anyone by applying your techniques?" (I had this asked of me once many year ago and even though I was relatively green, the question felt wrong to me even then), and how many times a researcher touts what he saved the world. Isn't this anti-intellectualism?
I suspect one reason is the excessive focus on churning out publications. Many researchers have become adept at churning them out. I think in many areas of Computer Science, this has the adverse effect of blurring the paradigm. Few people really know what others are doing, except superficially. (I am pretty sure there are some who don't even know what is in the paper that bears their name.) So in this sense, some researchers have unknowingly stopped judging contributions by the criteria of the relevant paradigm. And when that happens, they start resorting to the other non-paradigm criteria, by someone's else criteria.
I do not begrudge a researcher his success in commercializing his research or taking it to the masses. In fact, it is to be applauded, for it points to an additional skill-set that he has. But let us not make that the criteria of judging someone's research contributions.
I also have nothing against those who collaborate actively with industrial partners. To each his own. But that is no reason to apply non-paradigm criteria to those who don't.
At the beginning of his book "The Tacit Dimension" Polanyi mentions how he was struck by how communism required that scientists turn to solving the problem of the current five-year plan. It makes me wonder if he would have found questionable the constant harping about researchers delivering what the industry wants.
Tuesday, February 21, 2012
Argumentation in Research
There are many ways of supporting a thesis. Formal proofs and empirical evidence gathered from systematic experimentation are two of the most common in computer science. A third one that I find equally attractive is by presenting sound arguments for the thesis. However, in computer science, especially software engineering, it is a technique that is seldom-used and little-respected.
Advancing a thesis by argumentation is an analytical technique. Doing it well is not easy.
One must setup criteria, that is, the values, by which to judge the thesis. Further, the criteria better have broad appeal or else you will have to argue forcefully that they ought to be broadly accepted. If you fail to make a case for the criteria, your evaluation of the thesis against them would have little appeal to others.
One must have a clear grasp of the area that the thesis concerns, which includes being aware of the minute details of other people's approaches in the area: knowing exactly what they mean by the terms they use, knowing the details of their approach, and how what they are doing relates to your thesis. If you want to show the novelty of your thesis, you must show that others fail some of the criteria. Further, no one must be able to reasonably argue that they meet your criteria. Anticipating and addressing the arguments that others will put forward to reject your thesis is where a significant portion of your energy will be spent. You have to close potential all loopholes, at least in your mind if not in the paper itself.
It is a big mistake to put a well-argued paper in the same category as vision papers, or position papers, or "nice to read, but lacks evaluation". A vision paper would chart out the world an author would like to see, perhaps listing specific challenges and promising approaches. A position paper just says what the author thinks. Sometimes the author may take a contrarian stance in a position paper; however, no deep analysis of the strength of his position and the weakness of others need be involved. And it is patently wrong to say that paper that relies solely on arguments to make its points lacks evaluation: doesn't it evaluate the thesis against the criteria (and the criteria themselves).
A vision paper: Berners-Lee et al.: Scientific publishing on the semantic web
A position paper: Chopra et al.: Research Directions in Multiagent Systems
A well-argued technical paper: Singh.: Agent Communication Languages: Rethinking the Principles
Sometimes a good argument is all it needs to show limitations in a whole class of approaches that have been adequately formalized and "proved", implemented, and tested empirically. If instances of the latter were found worthy of publication, isn't the argument that shows their limitations at least as worthy?
Argumentation is a first-class scientific technique of validating claims. Let us not judge a good argument by the criteria of others techniques. Let us not outright consider a well-argued paper as unscientific or unsubstantiated. Let us give them a fair evaluation.
There is indescribable beauty in a well-argued paper. Dijkstra said that a formula is worth a thousand pictures. An argument is worth a thousand formulas.
Advancing a thesis by argumentation is an analytical technique. Doing it well is not easy.
One must setup criteria, that is, the values, by which to judge the thesis. Further, the criteria better have broad appeal or else you will have to argue forcefully that they ought to be broadly accepted. If you fail to make a case for the criteria, your evaluation of the thesis against them would have little appeal to others.
One must have a clear grasp of the area that the thesis concerns, which includes being aware of the minute details of other people's approaches in the area: knowing exactly what they mean by the terms they use, knowing the details of their approach, and how what they are doing relates to your thesis. If you want to show the novelty of your thesis, you must show that others fail some of the criteria. Further, no one must be able to reasonably argue that they meet your criteria. Anticipating and addressing the arguments that others will put forward to reject your thesis is where a significant portion of your energy will be spent. You have to close potential all loopholes, at least in your mind if not in the paper itself.
It is a big mistake to put a well-argued paper in the same category as vision papers, or position papers, or "nice to read, but lacks evaluation". A vision paper would chart out the world an author would like to see, perhaps listing specific challenges and promising approaches. A position paper just says what the author thinks. Sometimes the author may take a contrarian stance in a position paper; however, no deep analysis of the strength of his position and the weakness of others need be involved. And it is patently wrong to say that paper that relies solely on arguments to make its points lacks evaluation: doesn't it evaluate the thesis against the criteria (and the criteria themselves).
A vision paper: Berners-Lee et al.: Scientific publishing on the semantic web
A position paper: Chopra et al.: Research Directions in Multiagent Systems
A well-argued technical paper: Singh.: Agent Communication Languages: Rethinking the Principles
Sometimes a good argument is all it needs to show limitations in a whole class of approaches that have been adequately formalized and "proved", implemented, and tested empirically. If instances of the latter were found worthy of publication, isn't the argument that shows their limitations at least as worthy?
Argumentation is a first-class scientific technique of validating claims. Let us not judge a good argument by the criteria of others techniques. Let us not outright consider a well-argued paper as unscientific or unsubstantiated. Let us give them a fair evaluation.
There is indescribable beauty in a well-argued paper. Dijkstra said that a formula is worth a thousand pictures. An argument is worth a thousand formulas.
Subscribe to:
Posts (Atom)