"Piping" output from one function to another using Python infix syntax

It is hard to implement this using the bitwise or operator because pandas.DataFrame implements it. If you don't mind replacing | with >>, you can try this:

import pandas as pddef select(df, *args):    cols = [x for x in args]    return df[cols]def rename(df, **kwargs):    for name, value in kwargs.items():        df = df.rename(columns={'%s' % name: '%s' % value})    return dfclass SinkInto(object):    def __init__(self, function, *args, **kwargs):        self.args = args        self.kwargs = kwargs        self.function = function    def __rrshift__(self, other):        return self.function(other, *self.args, **self.kwargs)    def __repr__(self):        return "<SinkInto {} args={} kwargs={}>".format(            self.function,             self.args,             self.kwargs        )df = pd.DataFrame({'one' : [1., 2., 3., 4., 4.],                   'two' : [4., 3., 2., 1., 3.]})

Then you can do:

>>> df   one  two0    1    41    2    32    3    23    4    14    4    3>>> df = df >> SinkInto(select, 'one') \            >> SinkInto(rename, one='new_one')>>> df   new_one0        11        22        33        44        4

In Python 3 you can abuse unicode:

>>> print('\u01c1')ǁ>>> ǁ = SinkInto>>> df >> ǁ(select, 'one') >> ǁ(rename, one='new_one')   new_one0        11        22        33        44        4

[update]

Thanks for your response. Would it be possible to make a separate class (like SinkInto) for each function to avoid having to pass the functions as an argument?

How about a decorator?

def pipe(original):    class PipeInto(object):        data = {'function': original}        def __init__(self, *args, **kwargs):            self.data['args'] = args            self.data['kwargs'] = kwargs        def __rrshift__(self, other):            return self.data['function'](                other,                 *self.data['args'],                 **self.data['kwargs']            )    return PipeInto@pipedef select(df, *args):    cols = [x for x in args]    return df[cols]@pipedef rename(df, **kwargs):    for name, value in kwargs.items():        df = df.rename(columns={'%s' % name: '%s' % value})    return df

Now you can decorate any function that takes a DataFrame as the first argument:

>>> df >> select('one') >> rename(one='first')   first0      11      22      33      44      4

Python is awesome!

I know that languages like Ruby are "so expressive" that it encourages people to write every program as new DSL, but this is kind of frowned upon in Python. Many Pythonists consider operator overloading for a different purpose as a sinful blasphemy.

[update]

User OHLÁLÁ is not impressed:

The problem with this solution is when you are trying to call the function instead of piping. – OHLÁLÁ

You can implement the dunder-call method:

def __call__(self, df):    return df >> self

And then:

>>> select('one')(df)   one0  1.01  2.02  3.03  4.04  4.0

Looks like it is not easy to please OHLÁLÁ:

In that case you need to call the object explicitly:
select('one')(df) Is there a way to avoid that? – OHLÁLÁ

Well, I can think of a solution but there is a caveat: your original function must not take a second positional argument that is a pandas dataframe (keyword arguments are ok). Lets add a __new__ method to our PipeInto class inside the docorator that tests if the first argument is a dataframe, and if it is then we just call the original function with the arguments:

def __new__(cls, *args, **kwargs):    if args and isinstance(args[0], pd.DataFrame):        return cls.data['function'](*args, **kwargs)    return super().__new__(cls)

It seems to work but probably there is some downside I was unable to spot.

>>> select(df, 'one')   one0  1.01  2.02  3.03  4.04  4.0>>> df >> select('one')   one0  1.01  2.02  3.03  4.04  4.0

While I can't help mentioning that using dplyr in Python might the closest thing to having in dplyr in Python (it has the rshift operator, but as a gimmick), I'd like to also point out that the pipe operator might only be necessary in R because of its use of generic functions rather than methods as object attributes. Method chaining gives you essentially the same without having to override operators:

dataf = (DataFrame(mtcars).         filter('gear>=3').         mutate(powertoweight='hp*36/wt').         group_by('gear').         summarize(mean_ptw='mean(powertoweight)'))

Note wrapping the chain between a pair of parenthesis lets you break it into multiple lines without the need for a trailing \ on each line.

You can use sspipe library, and use the following syntax:

from sspipe import pdf = df | p(select, 'one') \        | p(rename, one = 'new_one')