Python pandas: how to remove nan and -inf values
Use pd.DataFrame.isin and check for rows that have any with pd.DataFrame.any. Finally, use the boolean array to slice the dataframe.
df[~df.isin([np.nan, np.inf, -np.inf]).any(1)] time X Y X_t0 X_tp0 X_t1 X_tp1 X_t2 X_tp24 0.037389 3 10 3 0.333333 2.0 0.500000 1.0 1.0000005 0.037393 4 10 4 0.250000 3.0 0.333333 2.0 0.5000001030308 9.962213 256 268 256 0.000000 256.0 0.003906 255.0 0.003922
You can replace inf and -inf with NaN, and then select non-null rows.
df[df.replace([np.inf, -np.inf], np.nan).notnull().all(axis=1)] # .astype(np.float64) ?or
df.replace([np.inf, -np.inf], np.nan).dropna(axis=1)Check the type of your columns returns to make sure they are all as expected (e.g. np.float32/64) via df.info().
df.replace([np.inf, -np.inf], np.nan)df.dropna(inplace=True)