Dijkstra, Lamport, & Misra

We know them as brilliant computer scientists. But their "non-paper" stuff is worth reading too.

Dijkstra: http://www.cs.utexas.edu/users/EWD/
Lamport: http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html
Misra:http://www.cs.utexas.edu/users/misra/speeches.html

Masters in Social Computing Syllabus

No fluff. To get the degree students must

• pass the Computing, Engineering, and Analytics courses (18 credits), 
• pass two of the optional courses (6 credits)
• complete a project or thesis  (6 credits)
  

Computing

• Interaction models (a distributed computing course starting with models of distributed computation and finishing with interaction protocols for social actors)
• Information & programming models (a databases course, but taking into account developments in the Semantic Web, NoSQL database, and things such as map-reduce)

Engineering

•  Service-oriented computing
•  Software engineering

Analytics

• Statistics
• Data mining

Optional

• Organizational Computing (norms, argumentation, and trust in organizations)
• Security
• Mechanism Design (incentives, auctions, and so on)
• Health Information Systems
• Social networks
• Social simulations
• Machine learning

On Clairty in Papers

Bianca's advisor on draft version 0: "It's not clear. Make it more clear."
On v1: "It's quite vague..."
On v2: "Things are ambiguous..."
On v3: "It's not quite clear..."
On v4: "Things are not well-defined..."
...
On v23: "Things are still fuzzy..."

Bianca: "Did you ever consider getting your eyes checked?"

Normal Science

Earlier, I had written on the how I thought it wrong to force academic researchers in Computer Science to promise technology transfer and collaborations with the industry. The crux of my argument was that industry and academia have different goals. I want to elaborate on that some more, especially from an academic perspective.

I think academic researchers are too hard on themselves when they judge the merit of their contributions in terms of adoption by industrial practitioners or whoever the perceived end-users are thought to be or by the amount of money they have saved someone. Those are arguably the standards that an industrial software firm would apply to judge the success of its product. Should academic researchers apply the same standard?

I think not.

As Kuhn famously pointed out, the researcher works in a community. The community judges the merit of a researcher's claims based on the paradigm the community follows. The paradigm is everything. The open problems comes from the paradigm. The solutions ought to make sense in the paradigm. A solution that makes sense may open up new problems, which then becomes part of the paradigm. That is what Kuhn calls normal science (as opposed to a paradigm shift, which is not the point of this post). A contribution then amounts to solving a problem within the paradigm.

Clearly, the problems of a paradigm cannot be generic things such as "to make contribution to social welfare" or "to have impact on industry practice" or "to foster end-user adoption". They have nothing to do with a particular paradigm. The problems of a paradigm are always technical in nature. Then why are researchers tempted to judge their contributions by standards that don't apply to them?

You will be surprised how many times a presenter is asked by another researcher from the audience, "...but how much money did you save anyone by applying your techniques?" (I had this asked of me once many year ago and even though I was relatively green, the question felt wrong to me even then), and how many times a researcher touts what he saved the world. Isn't this anti-intellectualism?

I suspect one reason is the excessive focus on churning out publications. Many researchers have become adept at churning them out. I think in many areas of Computer Science, this has the adverse effect of blurring the paradigm. Few people really know what others are doing, except superficially. (I am pretty sure there are some who don't even know what is in the paper that bears their name.) So in this sense, some researchers have unknowingly stopped judging contributions by the criteria of the relevant paradigm. And when that happens, they start resorting to the other non-paradigm criteria, by someone's else criteria.

I do not begrudge a researcher his success in commercializing his research or taking it to the masses. In fact, it is to be applauded, for it points to an additional skill-set that he has. But let us not make that the criteria of judging someone's research contributions.

I also have nothing against those who collaborate actively with industrial partners. To each his own. But that is no reason to apply non-paradigm criteria to those who don't.

At the beginning of his book "The Tacit Dimension" Polanyi mentions how he was struck by how communism required that scientists turn to solving the problem of the current five-year plan. It makes me wonder if he would have found questionable the constant harping about researchers delivering what the industry wants.

Argumentation in Research

There are many ways of supporting a thesis. Formal proofs and empirical evidence gathered from systematic experimentation are two of the most common in computer science. A third one that I find equally attractive is by presenting sound arguments for the thesis. However, in computer science, especially software engineering, it is a technique that is seldom-used and little-respected.

Advancing a thesis by argumentation is an analytical technique. Doing it well is not easy.

One must setup criteria, that is, the values, by which to judge the thesis. Further, the criteria better have broad appeal or else you will have to argue forcefully that they ought to be broadly accepted. If you fail to make a case for the criteria, your evaluation of the thesis against them would have little appeal to others.

One must have a clear grasp of the area that the thesis concerns, which includes being aware of the minute details of other people's approaches in the area: knowing exactly what they mean by the terms they use, knowing the details of their approach, and how what they are doing relates to your thesis. If you want to show the novelty of your thesis, you must show that others fail some of the criteria. Further, no one must be able to reasonably argue that they meet your criteria. Anticipating and addressing the arguments that others will put forward to reject your thesis is where a significant portion of your energy will be spent. You have to close potential all loopholes, at least in your mind if not in the paper itself.

It is a big mistake to put a well-argued paper in the same category as vision papers, or position papers, or "nice to read, but lacks evaluation". A vision paper would chart out the world an author would like to see, perhaps listing specific challenges and promising approaches. A position paper just says what the author thinks. Sometimes the author may take a contrarian stance in a position paper; however, no deep analysis of the strength of his position and the weakness of others need be involved. And it is patently wrong to say that paper that relies solely on arguments to make its points lacks evaluation: doesn't it evaluate the thesis against the criteria (and the criteria themselves).

A vision paper: Berners-Lee et al.: Scientific publishing on the semantic web

A position paper: Chopra et al.: Research Directions in Multiagent Systems

A well-argued technical paper: Singh.: Agent Communication Languages: Rethinking the Principles

Sometimes a good argument is all it needs to show limitations in a whole class of approaches that have been adequately formalized and "proved", implemented, and tested empirically. If instances of the latter were found worthy of publication, isn't the argument that shows their limitations at least as worthy?

Argumentation is a first-class scientific technique of validating claims. Let us not judge a good argument by the criteria of others techniques. Let us not outright consider a well-argued paper as unscientific or unsubstantiated. Let us give them a fair evaluation.

There is indescribable beauty in a well-argued paper. Dijkstra said that a formula is worth a thousand pictures. An argument is worth a thousand formulas.

Professor Jay Misra

Read his speeches.

I had the opportunity to meet Professor Misra when he came to give a talk at NC State in 2008. What I took from his talk was the beauty of minimality: how a computer scientist must strive to find the minimal set of operators or specification constructs and then build everything systematically from those.

Earlier that day, Munindar (my PhD advisor) had taken me to Professor Misra's hotel so we could talk over breakfast. Three generations ate together -- Munindar was once Professor Misra's student at UT Austin. And that day, I observed that Munindar was still his student.

Read Professor Misra's speeches, and you will know why he deserves all our admiration.

Unfortunately for me, I didn't impress him much.

Jeff Naughton on some challenges for the community

The presentation may be found here:
http://pages.cs.wisc.edu/~naughton/naughtonicde.pptx

Some of his key points

- Low acceptance rates is a major cause for the current situation of publish or perish. The solution is to accept more papers, perhaps all papers.

Although I never felt comfortable boasting about the acceptance rate of the venues where my papers have appeared, I never gave it much thought either. Conferences should stop boasting about their selectiveness.

- Three random reviewers simply does not cut it. Reviews should be more discussion-oriented.

- To mitigate the problem of bad reviews, publish the reviews. Let the authors be anonymous, but reveal the reviewers' identity. Let's put our money where our mouths are.

I have seen many many bad reviews. We should not let reviewers get away with bad reviews.

- Let's take advantage of technology to come up with new publishing and dissemination models! Check out the LiquidPub project (http://liquidpub.org/)

We need to think hard about these things. We need courage too.

RIP Future Work

Future work from many years ago remains unscathed. I prefer to not write these sections anymore -- they make me uneasy, especially the work part of it. However, I cannot jettison the section altogether. A situation where the a reviewer says 'It's a solid, conceptually well-founded paper that reconciles some long-standing issues in our field. I feel however that lacking a future work section the next steps are unclear, which will likely plunge our field into chaos. Hence the reject.' is not entirely unimaginable. So I started using Future directions instead. It is not only noncommittal, it is also in the spirit of spreading enlightenment -- I may follow them, but if you were any good, you'd follow them too.

The possibilities in a finality

In one of the papers I coauthored: "Finally, Section 6 concludes this paper". May be the writer thought about the possibility that finally one may not conclude a paper. Or perhaps, finally, one could conclude some other paper. Or perhaps, there is a Section 7, but Section 6 concludes this paper. Or some combination of the above -- Section 6 concludes this paper, but Section 7 some other. Or perhaps the author knows how very tedious the paper is, so he wrote the sentence to reassure the reader that there is an end to the tyranny.