Download png, svg

Problem variable:	Which variable has a problem? The number of movies Christopher Lee appeared in Master's degrees awarded in Education Neither / both
Issue:	What is the issue? The data is incorrect The chart isn't displaying properly There is a typo The source link is incorrect A calculation is wrong Something else
Additional details: Optional
Confirm you are a human:

Data details

Why this works

1. Data dredging: I have 25,237 variables in my database. I compare all these variables against each other to find ones that randomly match up. That's 636,906,169 correlation calculations! This is called “data dredging.” Instead of starting with a hypothesis and testing it, I instead abused the data to see what correlations shake out. It’s a dangerous way to go about analysis, because any sufficiently large dataset will yield strong correlations completely at random.
2. Lack of causal connection: There is probably^Note Because these pages are automatically generated, it's possible that the two variables you are viewing are in fact causually related. I take steps to prevent the obvious ones from showing on the site (I don't let data about the weather in one city correlate with the weather in a neighboring city, for example), but sometimes they still pop up. If they are related, cool! You found a loophole.
   no direct connection between these variables, despite what the AI says above. This is exacerbated by the fact that I used "Years" as the base variable. Lots of things happen in a year that are not related to each other! Most studies would use something like "one person" in stead of "one year" to be the "thing" studied.
3. Observations not independent: For many variables, sequential years are not independent of each other. If a population of people is continuously doing something every day, there is no reason to think they would suddenly change how they are doing that thing on January 1. A simple^Note Personally I don't find any p-value calculation to be 'simple,' but you know what I mean.
   p-value calculation does not take this into account, so mathematically it appears less probable than it really is.
4. Y-axis doesn't start at zero: I truncated the Y-axes of the graph above. I also used a line graph, which makes the visual connection stand out more than it deserves. ^Note Nothing against line graphs. They are great at telling a story when you have linear data! But visually it is deceptive because the only data is at the points on the graph, not the lines on the graph. In between each point, the data could have been doing anything. Like going for a random walk by itself!
   Mathematically what I showed is true, but it is intentionally misleading. Below is the same chart but with both Y-axes starting at zero.
5. Outlandish outliers: There are "outliers" in this data.^Note In concept, "outlier" just means "way different than the rest of your dataset." When calculating a correlation like this, they are particularly impactful because a single outlier can substantially increase your correlation.
   
   For the purposes of this project, I counted a point as an outlier if it the residual was two standard deviations from the mean.
   
   (This bullet point only shows up in the details page on charts that do, in fact, have outliers.)
   They stand out on the scatterplot above: notice the dots that are far away from any other dots. I intentionally mishandeled outliers, which makes the correlation look extra strong.

Try it yourself

# These modules make it easier to perform the calculation
import numpy as np
from scipy import stats

# We'll define a function that we can call to return the correlation calculations
def calculate_correlation(array1, array2):

    # Calculate Pearson correlation coefficient and p-value
    correlation, p_value = stats.pearsonr(array1, array2)

    # Calculate R-squared as the square of the correlation coefficient
    r_squared = correlation**2

    return correlation, r_squared, p_value

# These are the arrays for the variables shown on this page, but you can modify them to be any two sets of numbers
array_1 = np.array([12,6,2,2,0,2,0,0,1,0,])
array_2 = np.array([179047,164652,154655,146581,145792,145624,146368,146429,146989,153756,])
array_1_name = "The number of movies Christopher Lee appeared in"
array_2_name = "Master's degrees awarded in Education"

# Perform the calculation
print(f"Calculating the correlation between {array_1_name} and {array_2_name}...")
correlation, r_squared, p_value = calculate_correlation(array_1, array_2)

# Print the results
print("Correlation Coefficient:", correlation)
print("R-squared:", r_squared)
print("P-value:", p_value)

Reuseable content

• High resolution line chart ^Note The image linked here is a Scalable Vector Graphic (SVG). It is the highest resolution that is possible to achieve. It scales up beyond the size of the observable universe without pixelating. You do not need to email me asking if I have a higher resolution image. I do not. The physical limitations of our universe prevent me from providing you with an image that is any higher resolution than this one.
  
  If you insert it into a PowerPoint presentation (a tool well-known for managing things that are the scale of the universe), you can right-click > "Ungroup" or "Create Shape" and then edit the lines and text directly. You can also change the colors this way.
  
  Alternatively you can use a tool like Inkscape.
  
• High resolution line chart, optimized for mobile
• Alternative high resolution line chart
• Scatterplot
• Portable line chart (png)
• Portable line chart (png), optimized for mobile
• Line chart for only The number of movies Christopher Lee appeared in
• Line chart for only Master's degrees awarded in Education