Eliminating all data over a given percentile
Use the Series.quantile() method:
In [48]: cols = list('abc')In [49]: df = DataFrame(randn(10, len(cols)), columns=cols)In [50]: df.a.quantile(0.95)Out[50]: 1.5776961953820687To filter out rows of df where df.a is greater than or equal to the 95th percentile do:
In [72]: df[df.a < df.a.quantile(.95)]Out[72]: a b c0 -1.044 -0.247 -1.1492 0.395 0.591 0.7643 -0.564 -2.059 0.2324 -0.707 -0.736 -1.3455 0.978 -0.099 0.5216 -0.974 0.272 -0.6497 1.228 0.619 -0.8498 -0.170 0.458 -0.5159 1.465 1.019 0.966
numpy is much faster than Pandas for this kind of things :
numpy.percentile(df.a,95) # attention : the percentile is given in percent (5 = 5%)is equivalent but 3 times faster than :
df.a.quantile(.95) # as you already noticed here it is ".95" not "95"so for your code, it gives :
df[df.a < np.percentile(df.a,95)]