IIP-71-Final : Launching the Robot Index (iRobot)

Thanks very much for the Discord chat yesterday. I appreciate your time. The chat helped me understand what you’re trying to achieve. I’m mindful I’m coming into this discussion late so excuse me if I’m raising questions that have already been answered.

Let’s focus on the first question regarding momentum. iRobot’s strategy is to buy past winners in order to “buy high and sell higher”. However, winners tend to underperform in the future due to “reversion to the mean” also called the “winner-loser” phenomenon. This is consistent with investment advice not to “chase winners” or “buy into a pump”.

iRobot’s strategy therefore depends on there being positive, persistent momentum in the crypto market, i.e., winners continue to win. The literature on momentum in the crypto market is contradictory but the most recent and comprehensive studies show that momentum effects are small and short-lived (i.e., minutes to weeks) followed by large reversals. See Dobrynskaya (2021)

Is there any evidence of positive momentum in your data set and, if so, that the momentum persists on long-enough time horizons to be relevant to our investors? Alternatively, it would be helpful to show how the “winners” chosen for the iRobot portfolio performed (out-of-sample) against the “losers” and “average” tokens not chosen for the portfolio.

My other question was whether the Sortino ratio can actually predict future performance for equity or crypto portfolios but let’s come to that after we’ve resolved the momentum question.

2 Likes

Hey @JosephKnecht, thanks a lot for the stimulating conversation over the WE :wink:

I agree the immunity to mean reversion (winner-loser effect) is an important point that needs to be clarified based on empirical, out-of-sample data which I have endeavoured to do since Sunday.

Unfortunately, while I was extracting the “average” and “losers” portfolio performance from the backtest covering May to July '21, I discovered an issue with my price feed API whereby random bits of price history would be missing across the 800-ish tokens currently monitored.

The methodology behind iRobot relies on characterizing historical returns against downside volatility to capture early upside potential or exit of accumulation phases : complete datasets are required to be relevant / unbiased.

My priority before going into DG2 is therefore to understand the effect that this issue has got on iRobot’s composition as well as on backtesting performance.

Concretely, this means comparing my actual datasets with the results coming from another API, which is a fair bit of work : I will continue updating you and the community on my findings.

As shared in our chat, progressing iRobot without ensuring 100% transparency and integrity is the last thing I want to do.

On the bright side, testing another API is something I intended to do before launch anyway since my current one is also starting to be a limiting factor in terms of available price feeds.

I’m happy to have caught this issue now, and I’m sure we will look at this step back positively once the 1st decentralized Robo-Advisor is finally out on the market :mechanical_arm:

2 Likes

Likewise it’s always a pleasure speaking with a fellow math wonk. I’m glad you caught the missing values issue sooner rather than later. In TradingView I sometimes find missing values for thinly traded pairs. As the Coop matures it will probably be useful to have shared data management workflows to ensure best practices and minimize duplication.

Good luck fixing the issue and I’ll look forwad to the updated results.

4 Likes

Dear owls,

After 10 months growing this idea into a viable project, the last few weeks saw us overcoming the biggest challenge for the Robot Index to date. The time has come for the Index community to decide on iRobot’s future, and this time there will be no turn around !

Assuming positive feedback is received to the updates listed below, I am targeting the DG2 vote to start next week, Monday 8th November (cc @sixtykeys / @mel.eth for the pre-warning and the IIP number attribution :wink:). Note that a POAP has been prepared to reward all participants on Snapshot !

I’m not able to edit the original post above, but here are the most important updates to implement in Snapshot / Gitbook before the vote :

→ Example composition : Illustrated below is the draft composition of iRobot as of 13th October 2021, following the complete methodology described below and taking into account the latest data available.

Note that this composition is subject to change until launch to reflect the market dynamics and liquidity situation.

→ Backtest data :

A backtest covering the period from 1st May to 31st July 2021, and based on a sub-optimal montly rebalancing cadence, showed (link updated) that iRobot would have been the only asset on par with ETH, featuring a positive Sortino ratio and outperforming BTC, BED as well as DPI with more than 70% cumulative gains in the last 2 weeks of July.

→ Fee structure and initial liquidity strategy :

  • A simple streaming fee of 1.5% is proposed to take into account the complexity of the methodology and the amount of market research / data analysis required for its maintenance. This ideally positions iRobot in an intermediate range between DPI / MVI and the FLI suite.
  • It is proposed that Index Coop retains 100% of the streaming fee as well as the methodologist bounty for this product.
  • It is proposed that the community methodologist(s) contributing to iRobot’s maintenance claim rewards from the Product Working Group / Simple Indices Pod (cc @Cavalier_Eth / @DocHabanero).
  • Index Coop will provide the initial liquidity for iRobot . A launch on Sushi Onsen is envisaged to further incentivize liquidity provision and bootstrap initial AUM.

→ Author Background & Commitment :

  • Ex-F1 engineer, data analysis specialist.
  • Spent the last 2 years developing personal investment strategies based on this same framework used for the Robot Index.
  • Joined Index Coop in December 2020 and started contributing on DPI’s performance evaluation against its competitors.
  • Have continuously assessed and refined this methodology, its relevance as a product as well as the tools necessary for data processing and backtesting over the last 10 months.
  • Now able to work full time on iRobot’s success and growth. My future endeavours also include the development of bloom.fi as a Coop sub-DAO, specialized in global crypto market research & designed to support projects like iRobot.

DG2 vote : targeted for CW45

4 Likes

Thank you for the post, @Monportefeuille.

Before we schedule DG2 snapshot we would like to get feedback from the community about the perceived market opportunity for this product. Product-market fit is critical to the success of any product and getting this right ensures the success of our products and IC as a whole.

Keeping in mind that it’s not just time that’s dedicated but significant capital to:

  1. Provide seed liquidity
  2. Pay out reward incentives (if applicable)
  3. Pay gas fees for initial composition and rebalance

As PWG reviews various methodologies we feel that this product, though well thought out, will likely not drive adoption because it does not solve a need in the market.

Before we assign more time and resources, we would like to get feedback from the community. Mainly answering the question, “Would you buy this?”

We would like to keep this post open for one more week and decide next steps based on feedback.

2 Likes

Hey @DocHabanero,

As shared recently with @Cavalier_Eth, I’m more than happy to collect community feedback and ways to improve the narrative / positioning around iRobot.

I don’t pretend to own the key for the point you’re raising, I don’t think anyone does as is often the case for innovative products :

I would like to flag here that doing this kind of survey for iRobot at scale, via the Coop’s socials, is something I already suggested several months ago but nothing happened.

Nevertheless, as far as I’m aware, the Work Team Analysis and the Product Prioritization Score - that have also been issued for iRobot 1,5 months ago - are the main 2 instruments supposed to help PWG and the broader community building an unbiased opinion on a product. Otherwise, what would have been the point of the Product Onboarding Process overhaul ?

I hope that I’m not misinterpretating your intention here, because it’s a written discussion. But, at the moment, I don’t really understand how PWG unilaterally stopping the governance process, and posting this single question on the governance forum, is going to provide an accurate indication of what’s supposed to happen next ?

I fear this may create a dangerous precedent for internal methodologists, particularly as it comes several weeks after the project completed all required stages of the DG1 to DG2 process, and in spite of a continuous effort from the methodologist to work at the highest possible standard of transparency and quality over the past year.

9 Likes

You really don’t think the market needs a diversified portfolio, with long-lasting performance, and minimal downside, in a single token? I can’t think of a single person who doesn’t need this. With the right marketing, this could be a core holding for every native and noob alike. And quite possibly the coop’s biggest success, over the long run, as we watch sectors and flavors come and go. If we look at TradFi, the funds most suggested by advisors are not sector specific, or blue chip, like most IC products, but those with the broadest most diversified market exposure. That’s the value prop offered by iRobot, and it’s what the general market is buying. So far iRobot is the only IC product attempting to fill this HUGE need.

And I agree with @Monportefeuille points above regarding the process. Specifically, the onboarding process would require the review team to present “significant blockers that can not be avoided” in order to issue a “No” vote for DG2 at this point. Surely unsustainable product-market fit would qualify as such, but as @Monportefeuille pointed out, how do we reconcile a Market Opportunity score of 3 on the product scoring chart, with “does not solve a need in the market”?

6 Likes

Gotta say, I’m left pretty confused about this reply as I see no precedent for it. I have so many questions about the approach, I can’t imagine how Julien must feel to see a response like this from the Coop.

Fundamentally (and I may have missed something having pulled back from the forum recently) I don’t understand why the community is deciding whether an index has PMF prior to DG2, but after work team analysis is already complete. Is that part of the onboarding process now? Aren’t the experts in the product team supposed to assess this as part of the analysis?

Speaking of which, iROBOT was rated higher than BED (a product that is already live!) in the product scoring table, and YHI which didn’t have to jump through the same hoop prior to DG2. What is the difference here? If the response is that the numbers in that sheet are illustrative then why have it at all.

If the community feedback is lukewarm or negative, will iROBOT be shelved or scrapped entirely? How do you even judge it, who judges, and what is the pass criteria to move forward?

I think Coop members should put themselves in Julien’s shoes, as he’s worked tirelessly for months on this product, and put his faith in the product onboarding process during that time. If part of that process suddenly changes, then it is no longer credibly neutral. A big reason Coop products do so well is the trust in methodology and things not changing on a whim, we should strive to repeat that in all our interactions. I see no reason why iROBOT shouldn’t proceed to DG2 where the community gets to vote on it anyway.

Since the question is ‘what does the community think?’ I have to say I’ve been looking forward to this product launching. This and $GMI to me both fill the niche of projects I’m interested in allocating to but haven’t had the time or conviction. Index products using a methodology to allocate in this middle ground are something that’s missing in our lineup, and as stated in the GMI proposal:

‘$DEGEN by Indexed Finance – perhaps the closest alternative to GMI – is the most successful non-Coop index in the market.’

So there is room for IC offerings in this space, a sentiment backed up in the original work team analysis.

11 Likes

Hi @Monportefeuille , @DocHabanero – wanted to respond from the Work Team perspective! Chatted with @jdcook this afternoon on this and this comment reflects our thinking on the analysis specifically (not intending to comment on the product)

A few pieces of context:

  • We made changes to the onboarding process & WTA while iROBOT was in flight. Moving forward, WTA includes the supporting pod so that PWG feedback & crucial perspective is included. We really appreciate Julien’s patience with the process changes!
  • Separately, we are open to deprecating the Work Team Analysis or removing the current process if it no longer serves the community.

On this score:

  • Market Opp of 3 feels right for this product: it’s differentiated and a nice-to-have as other posters have pointed out and Julien presented research for. The rubric for nice-to-have specifically reflects thinking that we should launch as many products as possible to be successful – though this point is being re-evaluated throughout the Coop.
  • 1.74 is an ok score: it ranks below all of our live products with the exception of BED, but above products that also received serious consideration like SMI
  • If PWG or others have has supporting data that merits changing the score or redoing the rubric, we’d be happy to do so, especially given the 2 months that have elapsed since publishing it!
4 Likes

Would re-iterate everything mentioned above. Delaying or suspending the standard governance process at this stage seems unwarranted. Unless there are factors, technical or otherwise, that make this product untenable in which case those should be highlighted publicly.

While this could be true, I don’t think it warrants deviation from the standard process. PWG can certainly present the case for why this product wouldn’t be a good fit for the Coop. As @catjam suggested, that can be reflected in the updated score, if those concerns are backed by research/data. At which point the Coop should still follow the process and move to DG2.

Wouldn’t DG2 vote qualify as “feedback from the community”?

10 Likes

I read two interpretations of DocH’s post. If the proposal is that this is an ad hoc decision-gate, then I strongly agree with earlier comments that that would violate the process and be wholly unfair to the project team. If the request is for more community feedback ahead of the scheduled DG2 snapshot vote, then that would seem to be consistent with the process and fair, and something Julien has been asking for consistently albeit much earlier in the process. Let’s take the latter interpretation.

My main concern with iRobot is that the evidence that the strategy would outperform an ‘average’ portfolio is very weak. It claims to have discovered a new source of alpha which would require much more evidence than was presented here. The back-test showed that an ‘average’ portfolio would beat iRobot’s ‘winner’ portfolio ~80 days out of 90. The evidence barrier is high because there’s a lot of independent literature that any momentum in crypto is small and fleeting. Also, there’s a lot of evidence in the equity literature that the Sortino ratio (or Sharpe ratio) does not predict future performance.

On Sortino-optimization, it’s important to note that its inventor has completely disavowed his namesake methodology: “There was a time that I believed the Sortino Ratio was the best way to measure performance. When the evidence began to accumulate in the 1990s that I was wrong, I wrote a paper pointing out the flaw and posted it on my website.”

My second concern, which I don’t think has been addressed in the liquidity analysis or WTA, is the the product will have high NAV decay and rebalancing costs due to the very high turnover, mid/small cap composition, and $10M proposed AUM. As part of the DG2 post, it would be helpful to have a rough estimate of the value decay and rebalancing costs, particularly since the Coop will need to bear the latter costs.

Index Coop’s market survey indicated interest in a "Top 30 Market Cap Index”, “S&P500 equivalent”, “Value Index”, and “Diverse Blockchain segments” but there was no request for a momentum index or algorithmically-traded index. One could argue that the respondents were unaware such a product was even possible and so couldn’t ask for it but that’s impossible to disprove and the other requests showed a high level of sophistication.

As always, I’m glad we can have these discussions in the sprit of shared respect, process-focus and fact-based reasoning. Cheers.

3 Likes

Hi @JosephKnecht !
Thanks for your feedback.

On this aspect :

I think it’s important to consider the latest backtest data, which were compiled to answer one of your previous questions, and illustrates how iRobot would perform in an “up” market :

Regarding the value decay and rebalancing costs, is this something that your profitability tool would enable to recompile from past data ? In which case, I’m more than happy to have a quick introduction and try to produce this additional info.

Generally speaking, I totally agree with the principles of a fact-based, process-focused discussion and think that any reasoning around the methodology’s performance should revolve around this kind of data analysis.

1 Like

Wanted to post some comments on this initiative that I previously shared with PWG (separate from work team analysis.)

I’m super supportive of raising the bar on the products we launch and doing more comprehensive analysis on rebalancing costs, target AUM, etc (@JosephKnecht also makes good points here.) I personally think we will launch better products and drive higher success for the Coop when we empower PWG to have authority and ownership over the product roadmap.

That said, I think we can also elevate the approach PWG has taken to this product. While our current token-based voting system is not a good way to determine potential PMF, neither is personal opinion or using internal feedback as a proxy for market research. We have a great team on board and will make sure to bring data and customer & market research to future analysis.

Finally – I’d be remiss not to note that I’ve personally enjoyed working with @Monportefeuille and appreciate his dedication to this product. You’re a valued member of our community regardless of the outcome for iROBOT!

7 Likes

Hi Julien,

These back-test results are definitely more encouraging, particularly since they cover a down and up cycle. How difficult would it be to generate that over a 1-year period? Statistically, if you wanted to prove the effect was real you could also calculate the SD of returns across the average portfolios.

The ‘product profitability model’ we’re developing with @prairiefi , @Cavalier_Eth , and others (more on that Monday) takes the NAV decay and rebalancing costs as inputs and not outputs. @jackiepoo and @overanalyser are the liquidity experts but here’s a crude estimate.

By eyeball, it looks like iRobot’s turnover is ~20%/month (=$2M/month). From the October composition, the trading depth for a typical component is ~$10k @ 50 bps. The NAV decay would therefore be 0.2%/month (=20% x 0.5% x 2) and the rebalancing fees $40k/month (=$2M/$10k x $200/tx). Compare that to the monthly streaming fee income of $12.5k/month (=$10M x 1.5% / 12). One can do the same calculation for 100 bps. This may underestimate the costs because the more volatile components are likely less liquid. Also, this assumes there’s enough volume to arb the price back to baseline. To reduce the value decay and/or fees I can suggest reducing the AUM, increasing the liquidity weighting, or reducing the turnover with the understanding that that might reduce the return. Unfortunately, IC’s rebalancing algorithm is a black box so it’s hard to cost optimize. It’s worth noting that most momentum equity funds under-return precisely because of the high turnover costs. I hope that’s useful. Cheers.

1 Like

While I can see value in the wider conversations on methodology robustness, index running cost, etc, as others have pointed out, a sudden stop now against our old (and maybe new) consultation and launch processes feels like a problematic signal to send to methodologists.

If this product is not technically hard to launch and support, I feel we should go to DG2 and potentially launch it and deliver against quite a lot of historical signal to an external partner. It’s hard to be exposed to our changes of process too - we’re a young DAO, we’ll change processes again in the future.

Regardless of what the technical, nuanced answer is re momentum in crypto, market participants might not fully appreciate that (momentum lives as a real meme in TradFi (sometimes it - as a factor - works, sometimes it doesn’t. Just like ‘value’), and it’s very hard to know which products work until we launch them. I and many others - including the methodologists - thought MVI was a fun, small allocation, ‘satellite holding’ back in January - now MVI is far more than that. Twitter polls have our followers thinking MVI is the thematic index with the most potential this next year - I now think there are whole new larger (than defi) business models and platforms in tokens in MVI too. At recent rates of growth it’ll match DPI one day not too far out.

Product traction is an imprecise science. MVI benefitted from Axie, Illuvium and then… Lord Zuk - none of this was known in Q1. Similarly here, AI meeting crypto is a powerful meeting of trends, a momentum meme (correctly or incorrectly) could develop powerfully in crypto (the pumps are so hard to catch repeatably!) and an index like iRobot might be really well placed to capture attention and TVL.

6 Likes

DG2 for Launching the Robot Index (iRobot) has been queued for Monday, 8/11/2021.

https://snapshot.org/#/index-coop.eth/proposal/0x6e77a0e6d1fd3f2b73df0e4dad1df2daf17279370397f4f61af420076e2a69bc

7 Likes

Thanks a lot for unlocking the vote @Mringz ! Quickly flagging here that some parts of the text on Snapshot still mention the Yield Hunter Index.

Also, I have contacted the POAP team on their Discord to link the event I created with the proposal, hope they can process the request ASAP :mechanical_arm:

6 Likes

The intent was never to change the current process, but to provide PWG feedback and allow the community time to reflect. Any changes to the process will always be communicated and approved by the community.

Quantitative data should have been provided in lieu of ‘community feedback’, but shortly after DG2 is approved the process to start developing the product begins. At this point time and capital begin to incur.

As we continue to enhance the onboarding process, co-organized market research will be a critical input for determining product success. To @Monportefeuille’s point, he suggested this and we never followed up.

We will keep the community up to date on any concerns earlier in the process and provide objective feedback. This ensures everyone is comfortable and well informed for all community decisions.

11 Likes

Hey @catjam ! Sorry for the late reply, and thank you so much for the kind words ! Likewise, it’s been a pleasure contributing to the revamp of the onboarding process by sharing this first hand feedback :nerd_face:. Beyond this, as shared already with other members (and regardless of the outcome of the vote), feeling this energy and growing alongside this talented, focused community really is an incredible experience.

1 Like