Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add percentages instead of counts to countplot #1027

Closed
hnykda opened this issue Sep 29, 2016 · 30 comments
Closed

Add percentages instead of counts to countplot #1027

hnykda opened this issue Sep 29, 2016 · 30 comments

Comments

@hnykda
Copy link

hnykda commented Sep 29, 2016

Hello,

I would like to make a proposal - could we add an option to a countplot which would allow to instead displaying counts display percentages/frequencies? Thanks

@mwaskom
Copy link
Owner

mwaskom commented Sep 29, 2016

As of v0.13, normalization is built directly into countplot:

sns.countplot(diamonds, x="cut", stat="percent")  # or "proportion"

The recommendation is otherwise to use histplot, which has a flexible interface for normalizing the counts (see the stat parameter, along with common_norm), although its defaults are not identical to countplot so you'll need to be mindful of that. Here's an example:

sns.histplot(tips, x="day", hue="sex", stat="percent", multiple="dodge", shrink=.8)


Original answer (context for the rest of the thread):

This is already pretty easy to do with barplot, e.g.

import numpy as np
import pandas as pd
import seaborn as sns

df = pd.DataFrame(dict(x=np.random.poisson(4, 500)))
ax = sns.barplot(x="x", y="x", data=df, estimator=lambda x: len(x) / len(df) * 100)
ax.set(ylabel="Percent")

@hnykda
Copy link
Author

hnykda commented Sep 29, 2016

Oh! y="x" was that missing piece! :D

IMHO it would worth to add this example somewhere in the docs.

Thanks

@hnykda hnykda closed this as completed Sep 29, 2016
@napsternxg
Copy link

Unfortunately, this doesn't work if both x and y are non_numeric. I get the following error:

sns.__version__
'0.7.0'
ax = sns.barplot(x="name", y="name",
                 estimator=lambda x: len(x),
                 data=df, color="grey")
/PATHTO/anaconda2/lib/python2.7/site-packages/seaborn/categorical.pyc in infer_orient(self, x, y, orient)
    343         elif is_not_numeric(y):
    344             if is_not_numeric(x):
--> 345                 raise ValueError(no_numeric)
    346             else:
    347                 return "h"

ValueError: Neither the `x` nor `y` variable appears to be numeric.

@mwaskom
Copy link
Owner

mwaskom commented Sep 30, 2016

Pass orient="v" to avoid the attempt to avoid inferring the orientation, or pass any numerical column to y (it doesn't have to be the same as the x variable).

@hnykda hnykda reopened this Oct 4, 2016
@hnykda

This comment has been minimized.

@mwaskom

This comment has been minimized.

@hnykda

This comment has been minimized.

@hnykda hnykda closed this as completed Oct 5, 2016
@beniz

This comment has been minimized.

@gandhis1

This comment has been minimized.

@beniz

This comment has been minimized.

@mwaskom

This comment has been minimized.

@rselover

This comment has been minimized.

@mwaskom

This comment has been minimized.

@rselover

This comment has been minimized.

@mwaskom

This comment has been minimized.

@rselover
Copy link

Can you explain from your original example -

ax = sns.barplot(x="x", y="x", data=df, estimator=lambda x: len(x) / len(df) * 100)

  1. Why is y="x"?

  2. estimator=lambda x: len(x) / len(df) * 100 - OK x has been used a few times here, in your example it makes sense, but are we talking about the same x as x="x" and y="x"?

If I understood the answers to these two questions, I could move forward solving this on my own.

Thanks so much for your time and attention.

@mwaskom
Copy link
Owner

mwaskom commented Feb 22, 2017

y can be anything since you're not using the values, you're just counting how many there are. As stated above, the actual code for countplot is short and instructive as to what's going on. For your second question, no, the name used for the function parameter is arbitrary (as is always the case).

@joakimleijon

This comment has been minimized.

@neutralrobot
Copy link

neutralrobot commented Jan 26, 2018

Honestly, I think some way to handle percentages well would be an excellent quality of life addition. The proposed trivial solution, when "hue" is added, does not perform as I would naturally hope:
image
turns into:
image
I compare this to ggplot in R:

p5 <- ggplot(all[!is.na(all$Survived),], aes(x = Pclass, fill = Survived)) +
geom_bar(stat='count', position='stack') +
labs(x = 'Training data only', y= "Count") + facet_grid(.~Sex) +
theme(legend.position="none")
p6 <- ggplot(all[!is.na(all$Survived),], aes(x = Pclass, fill = Survived)) +
geom_bar(stat='count', position='fill') +
labs(x = 'Training data only', y= "Percent") + facet_grid(.~Sex) +
theme(legend.position="none")
plot_grid(p5, p6, ncol=2)

In its context this yields:
image

The stacked bars might be overkill, but the general point remains that seeing these makes it easier to evaluate percentages between categories at a glance. The first set of images was from my efforts to divide the ages up into discrete categories based on their different survival rates in Kaggle's Titanic dataset. I based this off of observations with distplot, but there was a little bit of guesswork in the exact cutoff lines and when I looked at various graphs using countplot, it would have been really convenient to be able to stretch them into normalized values as the R output does above, without having to figure out the best way to do it myself from the bottom up.

I'd like to propose the possibility that the most headache-free way to do this might be:
0. Pass a value into countplot, something like, 'percent=True'

  1. If hue is not specified, then the y axis is labeled as percent (as if sns.barplot(x="x", y="x", data=df, estimator=lambda x: len(x) / len(df) * 100) had been called)
  2. If hue is specified, then all of the hue values are scaled according to percentages of the x-axis category they belong to, as in the graph on the right from R, above.

Does this make sense?

@mwaskom
Copy link
Owner

mwaskom commented Jan 26, 2018

That's certainly one way to do it. But it is by no means the only way to do it. What if someone wants to have both x and hue but normalize so all bars add up to 1? Or what if they want to use facets? The challenge, which might not always appreciated by a userswho is focused on their particular use-case, is coming up with a suitably general API.

That said, I think people are somewhat forgetting that, while it can be convenient to be able to pass a full dataset to a plotting function and get a figure in one step, pandas is quite useful. It's really not very difficult to generate the plot you want, exactly the way you want it, with just one more step external to seaborn:

df = sns.load_dataset("tips")
x, y, hue = "day", "prop", "sex"
hue_order = ["Male", "Female"]

f, axes = plt.subplots(1, 2)
sns.countplot(x=x, hue=hue, data=df, ax=axes[0])

prop_df = (df[x]
           .groupby(df[hue])
           .value_counts(normalize=True)
           .rename(y)
           .reset_index())

sns.barplot(x=x, y=y, hue=hue, data=prop_df, ax=axes[1])

image

You can even do this in one method chain, saving a temporary variable name, if that's your preferred style:

(df[x]
 .groupby(df[hue])
 .value_counts(normalize=True)
 .rename(y)
 .reset_index()
 .pipe((sns.barplot, "data"), x=x, y=y, hue=hue))

@neutralrobot
Copy link

neutralrobot commented Jan 27, 2018

I appreciate the response. And naturally it's not the only way to do it. And I can also appreciate the difficulty in finding where to draw the line for a suitably general API. Had I not seen the R snippet above and also stumbled across this discussion thread, I would probably not have bothered to say anything. But It looks to me like having some kind of normalized rendition could be a pretty generalized need. (I notice that ggplot outputs these values with

y="Percent"

but still gives normalized values on the graph. I doubt it throws anyone for too big of a loop. Or am I misunderstanding how you propose that normalized values are obtained?)

I may be completely wrong in my idea that this is a reasonably generalized desire, and I'm not sure if there's a good way to find out, though this thread and stackexchange are suggestive at least. I posted because the ggplot inclusion of this functionality was also suggestive to me that it is of general use. My inexperience with ggplot may mean that there's something important I'm missing.

I can also appreciate the argument that this can be done in basically a one-liner in pandas. But I find this line of reasoning a little strange, because of the inclusion of countplot in the first place. I've only had a glance at the code for countplot and haven't fully wrapped my head around it, but am I right in my understanding that countplot is basically a special case function implementing the same underlying plotting functionality as barplot? This is what confuses me: surely it would be even more trivial to pass counts into barplot than it is to pass percentages or normalized values. So why include countplot? This is part of what I really like about seaborn.

Anyway, It's possible that this "quality of life" handling of percentages out of the box is not worth the effort. Honestly, I don't know. Would it be worth including the code snippet above as an example in countplot? I guess I might just write some wrapper function that performs as desired, but I have to think that something like this would interest more people than just me.

Edit: Another idea might be to include something like 'scaling' as a passed parameter in countplot and factorplot. It would take a function, similar to the 'estimator' parameter in barplot, and scale the counts according to that function. Maybe this would be generalized enough while also being convenient enough. I guess things like gaussian distributions would be trivial to do then also, for example?

Repository owner deleted a comment from tdpetrou Aug 14, 2018
@emigre459
Copy link

I was able to get the early barplot code from @mwaskom to work for visualizing the distribution of a categorical variable with a small DataFrame, but when working with a DataFrame that has millions of rows my kernel seems to freeze up.

What's odd is that countplot has no issue and runs in under 2 seconds for the same dataset. Any ideas why that might be the case?

@mwaskom
Copy link
Owner

mwaskom commented Sep 12, 2018

You probably want ci=None.

@emigre459

This comment has been minimized.

@ishant21

This comment has been minimized.

@danielkurniadi

This comment has been minimized.

@amueller

This comment has been minimized.

@gupta-rajat7

This comment has been minimized.

@Divjyot

This comment has been minimized.

@mwaskom
Copy link
Owner

mwaskom commented Jun 17, 2020

With #2125:

tips = sns.load_dataset("tips")
sns.histplot(tips, x="day", stat="probability")

image

tips = sns.load_dataset("tips")
sns.histplot(tips, x="day", hue="sex", stat="probability", multiple="dodge")

image

tips = sns.load_dataset("tips")
sns.histplot(tips, x="day", hue="sex", stat="probability", multiple="fill", shrink=.8)

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests