Download png, svg

Problem variable:	Which variable has a problem? Arson in Massachusetts Gasoline pumped in North Korea Neither / both
Issue:	What is the issue? The data is incorrect The chart isn't displaying properly There is a typo The source link is incorrect A calculation is wrong Something else
Additional details: Optional
Confirm you are a human:

Data details

Why this works

1. Data dredging: I have 25,237 variables in my database. I compare all these variables against each other to find ones that randomly match up. That's 636,906,169 correlation calculations! This is called “data dredging.” Instead of starting with a hypothesis and testing it, I instead abused the data to see what correlations shake out. It’s a dangerous way to go about analysis, because any sufficiently large dataset will yield strong correlations completely at random.
2. Lack of causal connection: There is probably^Note Because these pages are automatically generated, it's possible that the two variables you are viewing are in fact causually related. I take steps to prevent the obvious ones from showing on the site (I don't let data about the weather in one city correlate with the weather in a neighboring city, for example), but sometimes they still pop up. If they are related, cool! You found a loophole.
   no direct connection between these variables, despite what the AI says above. This is exacerbated by the fact that I used "Years" as the base variable. Lots of things happen in a year that are not related to each other! Most studies would use something like "one person" in stead of "one year" to be the "thing" studied.
3. Observations not independent: For many variables, sequential years are not independent of each other. If a population of people is continuously doing something every day, there is no reason to think they would suddenly change how they are doing that thing on January 1. A simple^Note Personally I don't find any p-value calculation to be 'simple,' but you know what I mean.
   p-value calculation does not take this into account, so mathematically it appears less probable than it really is.
4. Outlandish outliers: There are "outliers" in this data.^Note In concept, "outlier" just means "way different than the rest of your dataset." When calculating a correlation like this, they are particularly impactful because a single outlier can substantially increase your correlation.
   
   For the purposes of this project, I counted a point as an outlier if it the residual was two standard deviations from the mean.
   
   (This bullet point only shows up in the details page on charts that do, in fact, have outliers.)
   They stand out on the scatterplot above: notice the dots that are far away from any other dots. I intentionally mishandeled outliers, which makes the correlation look extra strong.

Try it yourself

# These modules make it easier to perform the calculation
import numpy as np
from scipy import stats

# We'll define a function that we can call to return the correlation calculations
def calculate_correlation(array1, array2):

    # Calculate Pearson correlation coefficient and p-value
    correlation, p_value = stats.pearsonr(array1, array2)

    # Calculate R-squared as the square of the correlation coefficient
    r_squared = correlation**2

    return correlation, r_squared, p_value

# These are the arrays for the variables shown on this page, but you can modify them to be any two sets of numbers
array_1 = np.array([43.1,38.2,34.7,34.5,22.3,16.6,19.2,16.9,18.5,72.9,73.5,47.5,50.3,7.8,10.5,0.4,10.8,10.4,10.5,9.3,9.7,10.3,10.6,10.6,10.7,10,8.5,9.3,7.6,7.1,5.7,6.5,5.6,4.3,4.5,6.3,4.5,])
array_2 = np.array([21.4646,23,24,21,21,25,25,25,25,29,26,21,15,9.6,36,4.49806,4.97778,5.00115,4.8843,4.91757,4.15984,3.1783,3.80929,3.8921,4.34679,4.39353,4.39353,4.35822,4.79082,4.4169,4.51038,4.54467,4.48701,4.48701,4.55712,4.31885,4.45975,])
array_1_name = "Arson in Massachusetts"
array_2_name = "Gasoline pumped in North Korea"

# Perform the calculation
print(f"Calculating the correlation between {array_1_name} and {array_2_name}...")
correlation, r_squared, p_value = calculate_correlation(array_1, array_2)

# Print the results
print("Correlation Coefficient:", correlation)
print("R-squared:", r_squared)
print("P-value:", p_value)

Reuseable content

• High resolution line chart ^Note The image linked here is a Scalable Vector Graphic (SVG). It is the highest resolution that is possible to achieve. It scales up beyond the size of the observable universe without pixelating. You do not need to email me asking if I have a higher resolution image. I do not. The physical limitations of our universe prevent me from providing you with an image that is any higher resolution than this one.
  
  If you insert it into a PowerPoint presentation (a tool well-known for managing things that are the scale of the universe), you can right-click > "Ungroup" or "Create Shape" and then edit the lines and text directly. You can also change the colors this way.
  
  Alternatively you can use a tool like Inkscape.
  
• High resolution line chart, optimized for mobile
• Alternative high resolution line chart
• Scatterplot
• Portable line chart (png)
• Portable line chart (png), optimized for mobile
• Line chart for only Arson in Massachusetts
• Line chart for only Gasoline pumped in North Korea