ECO-7026A Programming and Analytics for Behavioural Economists Summative Coursework 2 Brief

Instructions

The TOTAL number of marks available for the paper is 100 MARKS. There are two questions, with marks as indicated.

Assessment Task: Scientific replicability and the efficiency of prediction markets

An important part of scientific progress is the ability of researchers to replicate the results of experiments reported by others. Experiments may produce “false positives,” either simply because of random chance, or because of uncontrolled variables, poor experimental technique or accidents, or even, in some cases, outright fraud.

This problem has received special attention in recent years in the social sciences – in particular psychology and behavioural economics. Because of this, it is rational that, when a researcher reads a paper presenting a new finding, they in general will not have 100% confidence that the result is actually true and not just a false positive. It would be very interesting to aggregate this information to come up with an overall assessment of which results are likely to stand up to further scrutiny.

 Enter markets. The efficient market hypothesis suggests that markets can be quite good at aggregating private information. There is a long history of the use of “prediction markets” for doing just this. For example, many times the long-running Iowa Electronic Markets, as well as other commercial exchanges, do a better job of predicting election outcomes than polls.

Prediction markets were used to predict the successful (or unsuccessful) replication of psychology experiments in a paper, “Using prediction markets to estimate the reproducibility of scientific research,” by Dreber et al in Proceedings of the National Academy of Science in 2015. The article is available at https://www.pnas.org/doi/epdf/10.1073/pnas.1516179112, and as part of the background for this task you should read it carefully. It is not especially technical. 

Further, pay close attention to the “Materials and Methods” section as it explains important details about the data you will be working with.

You will be working with the data from this paper in this assessment. We have provided the data as a separate archive file which you will find alongside this assessment brief. The file contains the data as published by the authors of this paper, including their README file which explains the data. We will supplement this documentation with some additional notes and hints in the description of the two tasks.

Task One

In Task One, you will reproduce some of the data and findings of the paper. The file data4Rmerged_final_20151027.csv contains some summarised statistics for the hypotheses tested in the replications.

1. Reproduce the aggregate values for Market price, Survey result, Number Of Trades, and Vol. You will do this by using the other four files provided, which contain, respectively, the results of the survey of the participants, and the trades made in each of the markets. Note that you do not have to reproduce the “weighted” survey result.
2.  Produce a “nice” figure which is similar to the one in the paper as Figure 1. Try to replicate as many of the attributes of the figure as you can in terms of its content – but you do not need to match exactly its layout, colours, and so on. To accomplish these tasks, you will do the same types of steps as we have done in the workshop units and you already have done in Assessment 1. The important difference between this and Assessment 1 is that in Assessment 1 you were guided step-by-step what to do – now you will have to figure out what steps to do in what order!

Some notes and helpful hints:

• Market price is not given in the raw market trades data. However, the price is just the ratio of the total cost of the trade and the number of units purchased. When computing market price, note that trades can be made on both the “Yes” and “No” sides; see the Materials and Methods section.
• You can expect to use most of the techniques we have learned on working with DataFrames, including data cleaning, reshaping, and combining (both using concatenation and merges).
• The aggregated file is in fact correct, so if you write your code correctly you should obtain results that match those provided by the authors.
• If you find yourself struggling to reproduce the calculations fully, remember (a) try to do as much as you can as well as you can, and (b) showing awareness of the shortcomings of your code is better than ignoring them!
• Observe that in the event you encounter difficulties in completing the first question above, you can use the final merged data file and complete this part independently. So do not let getting stuck on parts of the first question keep you from missing out on opportunities to earn marks from the second.

Task Two

The paper focuses on reporting the aggregate results of the predictions. For this task, we will look at the structure of the trading data itself at a more micro level.

You should pick one (and only one) of these two options: 

1. The markets were open for two weeks, but trading did not take place uniformly across that time. Conduct an analysis of the pattern of the timing of trades. To what extent are the price trajectories monotonic, or are there those that go back-and-forth? Is there any correlation between the pattern of price formation and the degree to which there is consensus in the probability of replication as derived from the survey data?
2. Participants could trade in as many of the markets as they wanted to. Conduct an analysis of the patterns of individual traders. Did traders tend to trade in many markets or few? Are there any patterns in the size of the positions they took? Were some markets characterised by a few participants taking large positions, or many taking smaller ones? [Unfortunately, while we have survey data and trading data, the authors anonymised the survey data so we cannot link survey responses and trading, which would have been very interesting to do.]

For these options, the questions set are indicative. You do not need to answer them all, and/or you may answer related questions you might think of along the lines suggested.

For this task, you will provide a Jupyter notebook containing the codes conducting your analysis. In addition, you will write a report of no more than 1500 words which introduces the question(s) you asked and summarises the result with appropriate and nicely-formatted tables and visualisations.

For this report, you should look to the example of the original PNAS paper and attempt to imitate it in terms of style and content. In particular I would like to call your attention to the following:

1.  It is succinct and focused, and does not introduce irrelevant commentary or random citations. 
2. It does not provide a “play-by-play” description of how they analysed the data or what their analysis codes did; they focus on results. 5 ECO-7026A Version 1 Structure of your submission Submissions must take the form of