Get the average year (mean of days over multiple years) in Pandas
Group by the month and day, rather than the year and day:
test = ts.groupby([ts.index.month, ts.index.day]).mean()
yields
1 1 0.499264 2 0.449357 3 0.498883...12 17 0.408180 18 0.317682 19 0.467238... 29 0.413721 30 0.399180 31 0.828423Length: 366, dtype: float64