Request for Feedback: Index Coop KPIs v2.0

Product Analytics
1

v1.0 of the Index Coop North Star & Core KPIs were set in early 2021. At the time, DPI was our only launched product. These KPIs have served us well, and I believe we got a lot of things right when determining what was most important to measure as we grew this business. A few things (from my perspective and that I came up with while writing this) that we have learned about our KPIs:

  • Simple Index Products & Leverage Products may have some overlapping KPIs but should also have some different KPIs
  • Mint volume (alone) is a poor KPI
  • New product launches (or the lack thereof) are difficult to surface in our current KPIs
  • 7-day moving averages may not be the best time-frame for KPIs which require a moving average
  • Net Capital Inflows should be a core KPI
  • Protocol revenue can be tricky due to volatile market conditions - it may be best to try and break out revenue growth do to increased units and mint/redeem vs just larger AUM
  • Users are only represented in one North Star / Core KPI (# unique holders)
  • Milestone KPIs (ex: % of users with 2nd product buy) should be explored as potential Core KPIs
  • Cohort KPIs (ex: retention % of new users in the last 60 days) should be explore as potential Core KPIs

So, as with all things, our KPIs must evolve. The Analytics Working Group will be looking to recommend some updates to our KPIs and how they are reported. I wanted to put this out to the community because it would be extremely helpful to get any and all feedback from Coop leaders and community members around the effectiveness, or lack of effectiveness, of our current KPIs and ideas for Index Coop KPIs v2.0.

I want to also make it clear that this is not intended to be a giant shake-up. The majority of our KPIs serve us well - I think this is more about making a few adjustments and leveling up when it comes to how we think about the health of our business and organization.

Appreciate yā€™all!

Tagging some individuals who have provided feedback or shown specific interest in KPIs in the past:
@LemonadeAlpha @MrMadila @Thomas_Hepner @anon10525910 @dylan @fallow8 @Matthew_Graham @jackiepoo @sidhemraj @anthonyb.eth

17 Likes
2

Great write up and I am in complete agreement. A couple of thoughts:

  • Leverage Products need better KPIs. There was a query somewhere that shows the gas fees spent on the BTC2x product and showing that this product was running at a loss for a while. This is something that seems pretty important, but I donā€™t know a lot about it.
  • Agreed, mint volume doesnā€™t tell us a whole lot

New product launches (or the lack thereof) are difficult to surface in our current KPIs

What did you mean by this @jdcook?

  • I would prefer 28d Moving Averages. It eliminates noise due to weekly cycles and it is usually a long enough period to smooth out volatility.
  • Net Capital Flows (i.e. N$F) is critical because it measures growth in AUM that is independent of the underlying asset prices.
  • We definitely need to focus more on the users holding our products. We should be focused on getting them to allocate more of their wallet to Index Coop products.

Great work JD.

5 Likes
3

100%

7 days MA can serve a solid purpose for flagging potential issues and identifying immediate/short term impact of say events but due to volatility agree 28 days as a minimum for performance measurement [KPI] also keen on ~ 60/90 days metrics for reporting

Helpful to have milestone KPIā€™s

i.e [WIP] APAC are stoked to have @anthonyb.eth developing a growth marketing KPI which applies aggregate trade volume as a proxy for ā€˜raised awarenessā€™ however, I am unsure whether our Q4 milestone should target 10% 40% or 120% growth.

Milestones KPIā€™s would help us reverse engineer (estimate) resource requirments and subsequently budget.

1 Like
4

Thanks for the continued drive to iterate on and improve how Index Coop operates @jdcook.

Iā€™m by no means a statistician, but in the spirit of this request for feedback, I will share some thoughts.

Well developing organizational KPIs, I often find it useful to separate out KPIs into ā€œleadingā€ and ā€œlaggingā€

  • Leading indicators are sometimes described as inputs. They define what actions are necessary to achieve your goals with measurable outcomes. They ā€œleadā€ to successfully meeting overall organizational objectives, which is why they are called ā€œleadingā€.
  • If a leading indicator informs an organization of how to produce desired results, a lagging indicator measures current performance. They are opposites, and as such a lagging indicator is sometimes compared to an output metric.

Under this framework of leading and lagging KPIs, I would recommend IC break out KPIs as follows:

Core KPIs - lagging

  • TVL (unincentivsed by Index Coop)
  • No Products with over 10 million TVL
  • Net Capital Flows
  • Gross profit (replacing revenue to better account for operating costs of FLI products)
  • Unit Supply*

On unit supplyā€¦

  • Unit supply is an important metric, but in its current form doesnā€™t work.
  • All IC products are launched with a price of $100. However, over time performance means prices between products can diverge significantly, for example MVIs($100) and DPI ($300). This means someone buying $900 of DPI will have 1/3 of the impact on Unit Supply (+3) than if they bought MVI (+9).
  • Our current method for measuring unit supply is heavily impacted by the relative price of different products - an effect that will only increase as our product lineup grows.

=======

I will leave it to smarter minds than mine to solve this, but some solutions might include:

Option 1

  • by ā€œbasingā€ unit supply to $100 (i.e. the start price point of products). This means if $900 were spent on DPI (price $300) the ā€œbasedā€ unit supply increase would be 9 (900/100). If $900 were spent on mvi, the based unit supply would also be 9 ($900/100). I would have to run the maths by you @jdcook, but would a formula like this work:

AUM / (100/Current Price)

Worked exampleā€¦
A. $1,000 / (100/$200) = 500 unit supply
*$1,000 AUM, price of $200 per unit so based unit supply of 500. *

*B. $1,000 / (100/$100) = 1000 unit supply *
*Price of unit drops to $100, but AUM has remained constant. For this to happen, unit supply must double (formula keeps this consistent) *

*C. *
$500 / (100/$50) = 1000 unit supply (AUM h
AUM drop by $500 (half) as price for the product half. No change in unit supply. Again this is captured well by the model.

Option 2
Alternatively, you could use metrics that compare unit supply growth across different products (i.e. 28 day moving average). These could be viewed separately, or weighted based on the AUM of the different products (so a unit supply increase for DPI would have a greater impact on the combined metric than unit supply increases with smaller products).

=======

Core KPIs - leading

These metrics are critical to monitor to drive the long term success of the Coop. In particular I would be keen for more emphasis to be placed on Growth and Product related metrics.

  • Twitter impressions / Following (or any other suitable metrics for ā€œGrowthā€ @LemonadeAlpha)
  • Products launched in the last 3 months
  • Unique holders
  • Total Volume
  • Milestone KPIs (ex: % of users with 2nd product buy)
  • Cohort KPIs (ex: retention % of new users in the last 60 days)

I would see Milestone and Cohort KPIs are broadly ā€œbehavioural indicatorsā€ - and I would love to start getting get deeper analytics insights into customer behaviour.

@anthonyb.eth I like this idea:

General reflections on KPIs

One of my personal bugbears is having different Working Groups using the ā€œCore KPIsā€ as their primary success metrics. Having multiple different WGs talking to their success as evidence by high-level KPIs such as TVL is not helpful.

I would prefer WGs to increasingly focus on developing more granular leading KPIs we ultimately would drive overall success as defined by our core KPIs.

Hopefully, those reflections are useful @jdcook - the AWG has my total confidence.

7 Likes
5

I agree here completely. WGs should be focused on KPIs which are related to the work they are doing. This is the more challenging path, but itā€™s much more fruitful. Some of this requires a lot of creativity and experimentation to get right. For example, Iā€™m working with @lee0007 to create proxy measures for regional growth in APAC. Because of the limitations of data we are using Volume during specific Time Windows as a proxy for growth in Asia. This might not end up being the best metric, but itā€™s certainly better than crutching on TVL for example. Creating these WG KPIs that are ultimately the most useful, requires collaboration between AWG and the other WGs. My DMs are always open if anyone wants to work together to find unique, effective KPIs for their WG.

3 Likes
6

100% on this one for me! Obvs a bit biased as Iā€™ve been banging on about it lately but from a GWG perspective I personally have found this to be really useful. Imho N$F is the new TVL! :slight_smile: (Note F is for flows as sometimes they are not always in :cry:)

I also really value having both absolute and relative metrics side by side. For me, this helps to understand and measure the rate of change in activity XYZ etc.

Working with @anthonyb.eth recently I have also found moving away from sometimes defaulting to the usual denominators such as TVL etc also useful. (ie, weekly N$F / Cummulative N$F to date instead of TVL etc)

@Pepperoni_Joe, defo interested in your ā€œbasingā€ ideaā€¦ Iā€™d be interested to know if you think N$F can achieve the desired outcome? Just thinking in terms of simplicity it kinda ā€œrebasesā€ things back to something familiar ie $'s but maybe Iā€™m misinterpreting what you are driving at?

*EDIT: adding screenshots. As per below (MVI) N$F can be illustrated in weekly/monthly breakdowns as well as an aggregate/cumulative overtimeā€¦ Not only does it provide us with a simple, consistent and reliable demand-side indicator, it also gives us an indicator over retention. Another added bonus is when TVL is overlayed we also get a handy in/out the money indicatorā€¦

image
image1420Ɨ830 73.3 KB

image
image1420Ɨ830 76.7 KB

image
image912Ɨ700 66.2 KB

For the thematic products. DPI, MVI and DATA I would also really like to see ā€œMarket captureā€ as a much more prominent metric also. For instance, if LINK, GRT, FIL etc total $10B market cap and DATA = $100m ā†’ m.capture = 1% (I used to track this in a spready but recently have not had time to maintain it. I can provide workings etc if needed).

@jdcook maybe we can make room for this on a future org call to agree more on?

7 Likes
7

Yes, I was thinking the same thing here. N$F is the same as the ā€œbasingā€ idea where the base is $1 instead of $100, but the $1 is far more intuitive IMO. Unless, of course, I misunderstood as well.

2 Likes
8

Iā€™ve been thinking more about this and I would like to contribute some specific ideas for KPIs instead of just general thoughts. My broad idea is that our core KPIs should reflect our mission as an organization to be the BlackRock of Crypto. That means:

  1. Lead the crypto index industry in market share
  2. Grow the crypto index industry

A while ago I read ā€œPlay Biggerā€ which is about Category Design. The premise is that businesses in new markets need to define the market, become synonymous with it, and capture the market. Think Kindle (e-readers), Uber (ridesharing), iPhone (smartphones), Netflix (streaming services), and Index Coop (crypto indices). Leaders need to not only market and grow their own business, but they need to also grow the category.

Itā€™s undeniable that we are the leader in the crypto index market, but, we need to focus on growing the market as a whole.

Right now, TVL in DeFi is $175.4B.

Combined TVL in decentralized crypto indices is $403.2M which is 0.23%.

Iā€™m not sure that this is a perfect representation of the crypto index adoption, but it shows me that we have a lot of room to run. To this point, we have done a great job of capturing market share (81%). But, we need to improve in growing the market. Back to the point, our KPIs should reflect our performance in growing the market.

Category KPIs

  • Crypto Index Market Size > TVL in Crypto Indices
  • Crypto Index Market Share > TVL in Crypto Indices / TVL Locked in DeFi
  • Crypto Index Market Share 30 Day Growth > Todayā€™s Crypto Index Market Share / 30 days ago Crypto Index Market Share
  • Index Coop Market Share > TVL in Index Coop / TVL in Crypto Indicies
  • Index Coop Address Exposure Market Share > Address Exposure / Active DeFi Addresses

It may be worth looking at these numbers more granularly as well. For example, Metaverse Index / TVL in Metaverse. This would be more challenging, but important.

Customer Behavior KPIs

  • Percentage of Wallet in Index Coop > For wallets holding that have held and Index Coop Product, (Wallet Value of Index Coop Products / Total Wallet Value).

We should want to get our customers to dedicate their entire wallet to us. Letā€™s say you have a passive investment retirement account. Itā€™s almost always with a single investment manager. We should be implicitly or explicitly encouraging people to set up a passive investment wallet and dedicate the entire thing to Index Coop. Honestly, I might do this myself and set up a retirement wallet. Side note: Recurring buys would be extraordinarily helpful. This is how Vanguard continues to snatch so much of my money, haha.

  • Hold Time Distribution > Distribution of Days Held by Product.

Again, our non-leveraged products are meant to be buy-and-hold passive investments. People should be holding onto these for periods of years, not days. We should know how our customers are using our products AND we should be educating them/encouraging them to HODL. Note that this metric is related but different than user retention.

Financial KPIs

At the end of the day, our KPIs should also be about making money. Revenue, Net Revenue, Gross Profit are all important. I will need to spend some more time thinking about the financials.

8 Likes
9

Thanks for this prompt JD!

We definitely need to focus more on the users holding our products. We should be focused on getting them to allocate more of their wallet to Index Coop products.

Agree here, some things weā€™ve been tracking are the number/% of addresses holding multiple products and the wallet composition of our larger holders.

Anyone whoā€™s listened to at least three of my calls has heard of the AARRR (Acquisition, Activation, Retention, Revenue, Referral) funnel, I think it makes sense to have core KPIs within each of those buckets.

Another thing Iā€™ve mentioned in GWG calls, and which @anthonyb.eth quickly built out a dashboard for, is growth rate (N$F or otherwise) post-launch - where I posit we might benchmark the growth of products new and old alike against some rate (e.g. 0-6 months 10% w/w, 7-12 months 5% w/w, etc) - https://dune.xyz/queries/181927.

7 Likes
10

Love this metric! we could use your spreadsheet and develop a job board task for copper owls to update and maintain on say an ongoing monthly basis? Or as often as you need, every week for a month?

Id love to see ALL of this! I removed the wallet exposure one because I donā€™t really understand that me w. still 5 different wallets

100% on these too

2 Likes
11

Thanks @lee0007 , time for that stuff to graduate to dune though. @anthonyb.eth, hit me up if you need anything :slight_smile:

2 Likes
12

Is there a KPI objective for number of product launches per year? Eg, 5 simple + 2 medium + 2 complex. I think that would help scope how much EWG and PWG resource is required.

1 Like
13

This is a great overview & discussion.
iā€™m relatively new here to Coop, but two questions/thoughts to add:

a) Is Market share a metric worth tracking?
In essence %TVL/Total TVL in Indexed products
b) (and related) Is growth of the total market (total Pie) and the relevant alternative products worth tracking?
This could help surface Fast movers - products that grow faster than total market - which in turn could yield new product/tech ideas.

Alex

2 Likes