Unleashing the Robot 🤖

[Monportefeuille]

Dear owls,

10 days ago, the DG2 process for iRobot was paused in order to understand a technical issue that was discovered in the toolchain used to build this complex product.

A short update was provided last Monday during our weekly stand-up, as well as during the Product WG call.

The week-end approaching I thought it would be important to take some time again to go through the details, and above all the progress being made to finally unleash the Robot

1/ Flashback : what happened ?

While extracting the “average” and “losers” portfolio performance from the latest backtest data, some incoherences prevented clarifying this important point raised by @JosephKnecht.

Indeed, after reprocessing an old dataset from May, multiple different results came out across the 800-ish assets of which average return and downside volatility are tracked on a daily basis.

The initial investigation revealed that the length of the datasets provided by a long time reference API had changed randomly for several assets. After digging a bit more, it appeared that the beginning of the available price history would also be delayed by a couple of days to several weeks in comparison with the data available from CoinGecko. The variation in dataset length would simply be additional bits of history added on a random basis.

2/ What consequences did this have and what has been done so far to rectifiy the situation ?

While this issue has got a minor impact on the ranking of the oldest assets, the missing bits of history have introduced a bias in the calculation of the Sortino ratio for the “youngest” tokens. This has prevented their integration in the list of potential candidates for iRobot, such that the draft compositions used for backtesting or liquidity analysis would not match with the original methodology anymore.

At this point, it was decided to pause the DG2 process in order to re-build confidence in the whole toolchain, and deliver transparent / unbiased data to the community before the vote.

To achieve this, the whole data processing suite has been rebuilt and automated nearly from scratch. Most importantly, it is now based on data directly imported from CoinGecko (for the analytic geeks out there who would like to have a look at the tool, please DM me in Discord).

The old dataset from May, which led to this discovery, has been re-processed again. For the same asset tracked over the same period of time, it was checked that the new suite is delivering identical results to the old one.

This part of the investigation was concluded by drafting a new composition as of 1st May, and checking the differences with the original backtest. It was found that, among 15 underlyings, the new composition would retain 10 of the old one and introduce 5 new tokens with subsequent variations in weights.

It is worth mentioning that this new result reinforces my confidence in the methodology’s ability to identify strong candidates, but I will do my best to support this with more data in a future post.

3/ What’s next ?

While the backtest could simply be run again based on this latest composition, it’s also important to mention that automating access to the CoinGecko price history for any token significantly extends the universe of “processable” candidates.

This is exactly the reason why it was planned to give this a try before launch… Even if the experience has been more painful than expected, now that the most difficult has been done I think it makes sense to focus on the integration of this extended universe to :

• Draft the final iRobot composition
• Pick the most relevant selection to backtest again from May onwards.

We can reasonably expect that this will require at least one more week of work, but of course further updates will follow along the way. As usual, any question or comment, don’t hesitate to get in touch here or in Discord !

[catjam]

Thanks for the updates @Monportefeuille ! Impressed by your thoroughness

[Monportefeuille]

Dear all,

As mentioned during today’s PWG call, the toolchain that has been rebuilt in the last couple of weeks has started delivering results, and it’s a great pleasure to finally unveil a new draft composition for iRobot that uses its full potential :

The list of candidate tokens has been extended to 1429 (and counting …), which has opened the doors to more recent DeFi projects like Tokemak, dYdX or Convex Finance as well as metaverse / NFT highlights Yield Guild Games or Superrare. Besides these new entrants, 8 former components are still part of the selection, even though their respective weights have been re-evaluated.

Several additional comments :

• A lot of potential candidates have been excluded thanks to the pool liquidity threshold of $4m that has always been considered as minimum limit. The backtest data show that this is a crucial help to onboard solid components and avoid quickly fading hypes.
• Some components have seen their weight capped to 5%, as per the additional criteria on pools concentrating “only” $4 to $7m liquidity.
• Thanks to this extended database, it should be possible to limit the overlaps between iRobot and other Index Coop products to 3 or 4 identical tokens. Besides diversification, this will also avoid vampirizing liquidity from the smallest pools.
• 7 projects have been dismissed due to the absence or poor results of security audits, or due to code reviews still ongoing. 4 projects have been discarded due to their price and liquidity history not exceeding a month. For these reasons and due to the ongoing market dynamics, this composition is subject to change again before launch.

What’s next :

• The price history of these 1429 tokens has been re-processed all the way back to 1st May, this week’s remaining challenge is to put backtest results together again (including the loser’s portfolio which “caused” all this trouble @JosephKnecht ).
• We’re still on target to proceed with DG2 before Halloween !

[Monportefeuille]

Dear owls,

It is a great pleasure and relief to confirm that iRobot is back !

After long hours of re-building and re-processing, you will find below the updated results of the Robot Index backtest covering the initial period from 1st May to 31st July 2021. I am still working on extending this backtest without delaying the DG2 vote much further.

• Backtest Composition :
  Illustrated below is the evolution of iRobot “v2” ’s composition from 1st May (inception) until 1st July (2nd and last rebalancing of the considered period):

As mentioned in the last updates, the new tool suite used to build the index not only opens the doors to a wider asset universe, but also enables a much more accurate calculation of the rankings based on bits of price history that were missing before.

Particularly worth mentioning are the appearance of our own $INDEX in iRobot’s composition from 1st June 2021 onwards, as well as the early inceptions of $ILV, $CVX and $PERP.

As before, the biggest additions / deletions to the portfolio would have been in a +/- 5 to 7% order of magnitude.

Last but not least : a 5% cap has been applied at inception on the tokens sourced from pools with less than 7 m$ liquidity. Preventively, the same cap has been applied on the weights of tokens whose liquidity was falling under 4 m$ during the course of the backtest.

• Performance Metrics :

Illustrated below are iRobot “v2” ’s average performance metrics from :

• 1st to 31st May
• 1st to 30th June
• 1st to 31st July

Similar to the 1st time, the focus has been put explicitly on this period to simulate the worst possible market conditions for the product launch :

The results of the 1st backtest haven’t been added here to avoid information overload, nevertheless I’m pleased to confirm that v2’s performance would have been very similar despite the extra constraints on less liquid tokens.

• Value of 1000$ initial investment :

The direct comparison between iRobot “v1” (first backtest), “v2”, the market benchmarks as well as the “average” and “losers” portfolios can be done here, confirming that the methodology helped to identify momentum over this 3 months period.

Note that the only constraint applied for the construction of the “average” and “losers” portfolios was the minimum of 1 month price history (without extra considerations on liquidity), therefore a greater advantage can be expected if these limitations were lifted up.

• Final ranking :

This graph confirms that the Robot Index and ETH were the only assets to post a positive Sortino ratio over the considered time range. As mentioned before, this performance could have been further improved by switching from a monthly to a bi-weekly rebalancing cadence.

Any comment or question, please do not hesitate to post it here or in Discord (#irobot-discussion). I am now going to amend the DG2-IIP with the link to these results.

[JosephKnecht]

It’s unclear if the back-testing was out-of-sample. What was the training period?

The winner and average portfolios may be statistically indistinguishable. Roughly, what was the sd on the average portfolio return?

What’s the relative performance in an up market?

There’s a lot of rebalancing. What’s the approximate NAV decay and fees?

The persistence of losers is an interesting result. I guess it’s not true when they say just because you lost doesn’t mean you’re a loser

[Monportefeuille]

Hey Joseph,

@JosephKnecht

It’s unclear if the back-testing was out-of-sample. What was the training period?

The overall price history of the 1429 tokens composing the actual database has been processed 3 times successively up until 1st May, 1st June and then 1st July to compose each time the 3 portfolios (iRobot v2, “average” and “losers”).

The bargraphs above (“Performance Metrics”) indicate the respective performance of the Robot Index v2 during the month following each rebalance. The main graph (“Value of 1000$ Initial Investment”) aggregates these 3 sequences in a single curve.

@JosephKnecht

The winner and average portfolios may be statistically indistinguishable. Roughly, what was the sd on the average portfolio return?

The standard deviation on the “average” portfolio return (both positive and negative) was around 7.6%.

@JosephKnecht

What’s the relative performance in an up market?

The additional month of backtesting (August) should give even more information on this but the market reversal following the May turmoil already happened on 20th July.

@JosephKnecht

There’s a lot of rebalancing. What’s the approximate NAV decay and fees?

I haven’t calculated the fees for each rebalance, but the 5% cap on the less liquid token is the best way we found with @overanalyser in order to limit the maximum number of trades while sticking to 0.5% price impact for each trade … and sticking as much as possible to the nature of the index !

[JosephKnecht]

So was the model trained on May to July data and then evaluated on the same period?

[Monportefeuille]

No, trained on whatever price history is available for each token until the last day of month X-1, and then evaluated from the 1st to the last day of month X.

[JosephKnecht]

I see. That’s interesting. It would be worth calculating the NAV decay and fees from rebalancing. I was surprised when I found out how expensive rebalancing is. What’s the planned AUM at launch?

I’d echo an earlier comment that the term ‘robo-advisor’ is too broad since it’s used to describe passive investing generally. This is essentially a momentum strategy so maybe MNTM, MOJO, or PACER instead.

High Momentum ETF List

[Monportefeuille]

Dear owls,

Let’s leave the data speak for themselves with the latest and final backtest update, now covering 1 additional month i.e. the overall period from 1st May until 31st August '21 :

• Composition :
  Illustrated below is the evolution of iRobot “v2” ’s composition from 1st May (inception) until 1st August (3rd and last rebalancing of the considered period):

• Performance Metrics :
  Illustrated below are iRobot “v2” ’s performance metrics against the benchmarks (now including MVI) :

• Value of 1000$ initial investment :

• Final ranking :