Normalizing a histogram with matplotlib

Question

I want to plot a histogram with Matplotlib, but I'd like the bins' values to represent the percentage of the total observations. A MWE would be like this:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import matplotlib.pyplot as plt
import matplotlib.ticker as tck
import seaborn as sns
import numpy

sns.set(style='dark')

imagen2 = plt.figure(1, figsize=(5, 2))
imagen2.suptitle('StackOverflow Matplotlib histogram demo')

luminance = numpy.random.randn(1000, 1000)
# "Luminance" should range from 0.0...1.0 so we normalize it
luminance = (luminance - luminance.min())/(luminance.max() - luminance.min())

top_left = plt.subplot(121)
top_left.imshow(luminance)
bottom_left = plt.subplot(122)
sns.distplot(luminance.flatten(), kde_kws={"cumulative": True})

# plt.savefig("stackoverflow.pdf", dpi=300)
plt.tight_layout(rect=(0, 0, 1, 0.95))
plt.show()

The CDF here is OK (range: [0, 1]), but the resulting histogram doesn't match my expectations:

Why are the histogram's results in the range [0, 4]? Is there any way to fix this?

Answer 1

What you think you want

Here's how to plot the histogram such that the bins sum to 1:

import matplotlib.pyplot as plt
import matplotlib.ticker as tck
import seaborn as sns
import numpy as np

sns.set(style='dark')

imagen2 = plt.figure(1, figsize=(5, 2))
imagen2.suptitle('StackOverflow Matplotlib histogram demo')

luminance = numpy.random.randn(1000, 1000)
# "Luminance" should range from 0.0...1.0 so we normalize it
luminance = (luminance - luminance.min())/(luminance.max() - luminance.min())

# get the histogram values
heights,edges = np.histogram(luminance.flat, bins=30)
binCenters = (edges[:-1] + edges[1:])/2

# norm the heights
heights = heights/heights.sum()

# get the cdf
cdf = heights.cumsum()

left = plt.subplot(121)
left.imshow(luminance)
right = plt.subplot(122)
right.plot(binCenters, cdf, binCenters, heights)

# plt.savefig("stackoverflow.pdf", dpi=300)
plt.tight_layout(rect=(0, 0, 1, 0.95))
plt.show()

# confirm that the hist vals sum to 1
print('heights sum: %.2f' % heights.sum())

output:

heights sum: 1.00

The actual answer

This one is actually super easy. Just do

sns.distplot(luminance.flatten(), kde_kws={"cumulative": True}, norm_hist=True)

Here's what I get when I run your script with the above modification:

Surprise twist!

So it turns out that your histogram was normalized all along, as per the formal identity:

In plain(er) English, the general practice is to norm continuously valued histograms (ie their observations can be expressed as floating point number) in terms of their density. So in this case the sum of the bin widths times the bin heights will 1.0, as you can see by running this simplified version of your script:

import matplotlib.pyplot as plt
import matplotlib.ticker as tck
import numpy as np

imagen2 = plt.figure(1, figsize=(4,3))
imagen2.suptitle('StackOverflow Matplotlib histogram demo')

luminance = numpy.random.randn(1000, 1000)
luminance = (luminance - luminance.min())/(luminance.max() - luminance.min())

heights,edges,patches = plt.hist(luminance.ravel(), density=True, bins=30)
widths = edges[1:] - edges[:-1]

totalWeight = (heights*widths).sum()

# plt.savefig("stackoverflow.pdf", dpi=300)
plt.tight_layout(rect=(0, 0, 1, 0.95))
plt.show()
print(totalWeight)

And the totalWeight will indeed be exactly equal to 1.0, give or take a smidge of rounding error.

Answer 2

tel's answer is great! I just want to provide an alternative to give you the histogram you want with less lines. The key idea is to use weights arguments in the matplotlib hist function to normalize counts. You can replace your sns.distplot(luminance.flatten(), kde_kws={"cumulative": True}) with the following three lines of code:

lf = luminance.flatten()
sns.kdeplot(lf, cumulative=True)
sns.distplot(lf, kde=False,
             hist_kws={'weights': numpy.full(len(lf), 1/len(lf))})

If you want to see the histogram on a second y-axis (better visual), add ax=bottom_left.twinx() to sns.distplot: