APWG Growth & Performance Measures: Q4 Updates

Growth APAC
1

Overview

This post is intended to

  • clarify APWG current objective and key performance indicators
  • provide an up to date reference for monthly reporting during Q4 2021 @pujimak_in
  • ensure transparency and opportunity for community feedback
    CC: @lanks @Vanita @Tudou
KPI BASELINE TARGET OCTOBER OBJECTIVE SOURCE FREQUENCY
Strategic
APAC Median Buy Size - ? WIP Strategic Dune 30 / 90 Days
APAC Net Buys - ? WIP Strategic Dune 30 / 90 Days
APAC % Trade Volume - ? WIP Strategic Dune 30 / 90 Days
APAC % N$F - ? WIP Strategic F. Nest 30 / 90 Days
Tactical
IB/BD Relationship 23 26 31 IB/BD Notion Month (Mth)
Bi Weekly BD IB Community Call 10 10 - IB/BD Count Month (Mth)
Community Autonomy BD 2.5 3.0 - IB/BD Survey End Q4
Community Autonomy IB 1.7 2.2 - IB/BD Survey End Q4
Community Autonomy GM 2.8 3.3 - GM Survey End Q4
Website Users Top 10 22% 22% 9.84% GM GA4 Mth / Qtr
Website user Top 50 25% 25% 18.69% GM GA4 Mth / Qtr
Website users Top 100 24% 24% 16.72% GM GA4 Mth / Qtr
Gold/Silver/Bronze Owls 45% 45% 51% TOC Discord Mth / Qtr
Women/Non-Binary Contributors 15% 15% 13.5% TOC Discord Mth / Qtr
Lastest Update: 4 Nov 2021

Discussion

Strategic Objective: Deliver quantifiable value ≥ 5 x APWG Requested Budget = ~$770k quantified in terms of net inflow, trade volume, raised awareness and minimised costs

  • A [Strategic KPI] APAC % Net $ Inflow
  • B [Strategic KPI] APAC % Trade Volume
  • C [Strategic KPI] APAC Median Buy Size
  • D [Strategic KPI] APAC Net Buys
    Target: baseline unknown
    Status: Experimental WIP requiring Analytics & Treasury Support

Although the strategic objectives remain unchanged our approach to quantifying value continues to evolve. We recognise that measuring quantifiable value is a challenge for which we continue to seek answers. As such, there has been a number of changes to the tactical objectives and key performance indicators identified in the Q4 APWG proposal as discussed below.

Increase Net $ Inflow through strategic Business Development & Institutional Business partnerships
[KPI] Growth % APAC Net Inflow
Target: baseline unknown
Status: Experimental
Changed: from a tactical to a strategic objective as defined by requiring on-chain data

We believe it is safe to say that a % of global trade volumes, net flows and assets under management, are attributable to people in APAC countries. However, we have not yet established a way to attribute Net $ Inflow to APAC trading or activity, so at this point, we can neither baseline nor quantify the value of APAC work in N$F terms.

The primary challenge is the disconnect between on-chain ($) and off-chain (marketing) data sources, and so we are in the process of experimenting with data, in search of trends and testing hypotheses’. For this purpose, a second google analytics account was established to install the former Universal Analytics (UA). UA began collecting data on 17 October to provide functionality that GA4 no longer allows.

  • Account views: allows us to create and maintain an APAC filtered (by country) view of website analytics
  • Users by the time of day: provides 24/7 website activity reporting to the hour level
  • Run API scripts for the filtered APAC account view

We are looking to surface answers to the following questions:-

Q: Are there any trends in the data to indicate typical times and days of high activity in APAC?
Q: Can off-chain data (website user activity) be mapped to on-chain transactions?

We will monitor and look to export (API scripts) day and time day from UA to see if there is any correlation between website users and signed or rejected transactions. If we can correlate APAC ~25% website user activity to x% trade volume (TV), the hypothesis is that following an event OR during spikes in APAC web user-activity, we could expect to see an increase from baseline % TV. There may be no correlation. And at this point, we have very limited data to work with (est. Oct 21), but if the scripts can automate hourly data feeds our talented data analysts will soon identify or rule out this type of approach @anthonyb.eth @prairiefi

Q: Can we attribute purchase intent signals on the website - events such as unlock wallet or click buy buttons - to signed and or rejected transactions?

At the moment the only product being “tracked” is DPI. We would like to see this expanded to include all products. Work to explore the purchase funnels and signals that correlate to transactions is currently on hold as we await the delivery of a marketing facing website. Working with @0x_Dev once the website is live we can progress the tagging of events on the website in order to see how APAC website users interact with our products online. APAC specific funnel information could enable future KPI’s to better reflect the GMWG AAARRR marketing funnel - awareness, acquisition, activation, retention, revenue and referrals @MrMadila

Tactical Objective 1 Drive [Strategic Success Measures] through Business Development & Institutional Business partnerships
1.1
[KPI] # Strategic Relationship established/progressed by IB BD team
Baseline: 23
Target: 26

Tactical Objective 2:: Support new contributors to develop IB BD context, depth of market and product knowledge and empower high-impact contributors through increased autonomy
2.1
[KPI] # Attendance IB BD Bi-Weekly Community Call
Baseline: TBC
Target: 10
2.2
[KPI] Self Reported Autonomy of community members in relation to Business Development, Institutional Business and Growth Marketing
Baseline: TBC
Target: 20% growth of self-reported 3 or 4 level contributors

Due to the difficulty of quantifying N$F we have created tactical objectives 1 & 2 in reference to strategic KPI’s. The actual tactical KPI’s are focused on establishing relationships, processes and removing barriers to participation.

Establishing a Bi-Weekly IB BD Community call was identified as a key process to overcome time zone barriers to participation with existing IB BD Working Group Calls. It is hoped that these calls will help the APWG community to develop context together as a cohort and further decentralise knowledge in order to build autonomy.

We will baseline self-reported autonomy at the first meeting via a simple Typeform survey.

On a scale of 0 - 4: at this stage of your journey, how confident do you feel contributing to the operations of each of the Index Coop’s existing working groups? (BD, IB, GM)

0 = Not Applicable
1 = learning & seeking support
2 = capable & seeking guidance
3 = experienced & seeking collaborators
4 = professional & seeking full autonomy

At the beginning of Q1 2022, we will again survey the community to identify if these meetings have helped community members develop autonomy and confidence in the areas of institutional business, business development and growth marketing. The first meeting is scheduled for Tuesday 2 November at which point baselines will be added to the summary table.

Tactical Objective 3: Increase $ Aggregate Trade Volume by raising awareness of Index Coop Products through regional content development and events with CTA that drive visitors to the IC website.
3.1
[KPI] # Website Users Top 10
Baseline Target: 22%
3.2
[KPI] # Website Users Top 50
Baseline Target: 25%
3.3
[KPI] # Website Users Top 100
Baseline Target: 24%

These new website users KPI’s were added as we gained access to Google Analytics data since the development of the proposal. Global GMWG has significantly more budget and resources for content development and as we are not competing with them APWG goal will look to monitor and maintain this baseline presence. It is expected that these KPI’s could change significantly once the new website is live with translated copy that will better serve APAC markets or audiences from countries such as China, Japan and South Korea.

Tactical Objective 4: Identify, develop and retain contributors owls through the use of APAC onboarding community & talent scouting and mentoring initiatives.
4.1
[KPI] # Gold, Silver, Bronze level owls
Baseline: 48%
Target: 45%
4.2
[KPI] Representation of women/non-binary contributors (3% above industry standard 1)
Baseline: 14%
Target: 15%

Removed

The following KPIs listed in the Q4 APWG proposal have been removed

  • [KPI] # New IB BD opportunities identified: removed as duplicate refer strategic relationships established/progressed
  • [KPI] # Event participants: removed because we will instead measure engagement following events based on website user activity
    [KPI] # Impressions while we intend to develop regional content it is unlikely we can effectively measure impression on channels that we do not own, if the content is appropriate for sharing on Twitter we may be able to measure impressions but we do not intend Twitter impressions to be an APWG KPI.
  • Community: outreach focused on crypto native audiences such as young professional and university students. We note here that our focus on onboarding APAC university students signalled support for the development of an internship program, however, this project is undertaken by and sits best with TOC Talent @bradwmorris @fishbiscuit

Q4 Ongoing Edits

4 November: Update Oct actual performance & changes to autonomy KPIs
5 Nov: Reporting Community KPI’s

6 Likes
2

<3 Awesome to see this challenge being tackled further in depth and with a SME such as yourself!

Growth could certainly piggyback on some of this work for broad application.

3 Likes
3

APWG keeps killing it - Great to see this information laid out. We are only as strong as our ability to work together across our entire organization. This document really highlights how the work of each different WG feeds into the others and supports the entire organization.

3 Likes
4

EDITS: 4 Nov 2021 (GMT+13)

The following table was removed from the forum post above with updated performance metrics and October: Actual results

KPI TARGET OBJECTIVE DATA SOURCE TIME
APAC Median Buy Size Baseline Req. Strategic Dune Analytics 30 / 90 Days
APAC Net Buys Baseline Req. Strategic Dune Analytics 30 / 90 Days
APAC % Trade Volume Baseline Req. Strategic Dune Analytics 30 / 90 Days
APAC % N$F Baseline Req. Strategic Finance Nest 30 / 90 Days
IB/BD Relationship 23 IB/BD Notion Month (Mth)
Bi Weekly BD IB Community Call 10 IB/BD Head Count Month (Mth)
Self-Identified Autonomy = 3 or 4 20% IB/BD Survey Quarter (Qtr)
Website Users Top 10 22% GM Google Analytics Mth / Qtr
Website user Top 50 25% GM Google Analytics Mth / Qtr
Website users Top 100 24% GM Google Analytics Mth / Qtr
Gold/Silver/Bronze Owls 45% TOC Discord Dash Mth / Qtr
Women/Non-Binary Contributors 15% TOC Discord Dash Mth / Qtr
Last Update: 31 October 2021

The primary change to the forum post was to self-identified autonomy levels.

  • Information on each of the levels was added to the forum post.
  • KPI used to measure self reported autonomy have changed from a % to the group average score for each of BD IB & GM.

The new KPI’s better reflect APWG opportunities to build context, learn and develop shared understanding. At this capture only 10 people have reported their level of autonomy and it is key to note that our key BD people @Lanks @pujimak_in are not include in this reporting.

2 Likes
5

This looks very comprehensive. @lee0007!

I am new to forum, is there anywhere to catch-up on a bit more background on these metrics? (N$F etc)

Would love to read more on this. Thanks!

2 Likes
6

I’d recommend talking to the experts by joining a weekly community call for the Analytics Working Group or bi weekly Growth Marketing working group. This conversation is a good read Request for Feedback: Index Coop KPIs v2.0

2 Likes
7

November Results Updates

KPI BASELINE TARGET OCT NOV OBJECTIVE DATA SOURCE FREQUENCY
APAC Median Buy Size Required ? WIP Review Data UA Strategic Dune Analytics 30 / 90 Days
APAC Net Buys Required ? WIP Review Data UA Strategic Dune Analytics 30 / 90 Days
APAC % Trade Volume Required ? WIP Review Data UA Strategic Dune Analytics 30 / 90 Days
APAC % N$F Required ? WIP Review Data UA Strategic Finance Nest 30 / 90 Days
IB/BD Relationship 23 26 31 35 IB/BD Notion Month (Mth)
Bi Weekly BD IB Community Call 10 10 - 10 IB/BD Head Count Month (Mth)
Community Average Autonomy BD 2.5 3.0 - - IB/BD Survey End Q4
Community Average Autonomy IB 1.7 2.2 - - IB/BD Survey End Q4
Community Average Autonomy GM 2.8 3.3 - - IB/BD Survey End Q4
Website Users Top 10 22% 22% 9.84% 13.64% GM Google Analytics Mth / Qtr
Website user Top 50 25% 25% 18.69% 20.41% GM Google Analytics Mth / Qtr
Website users Top 100 24% 24% 16.72% 21.56% GM Google Analytics Mth / Qtr
Gold/Silver/Bronze Owls 45% 45% 51% 38% TOC Discord Dash Mth / Qtr
Women/Non-Binary ‘Active’ Contributors 15% 15% 13.5 21% TOC Discord Dash Mth / Qtr
Last Update: 3 Dec 2021

Comments

  1. Moving towards trying to align off-chain and on-chain data we now have a full month of APAC by countries (16) website user data from Google Universal Analytics. This visualisation by @anthonyb.eth shows the average session by day of the week and time of the day. Where 1-01 is Monday 01:00 - 01:59 and 5-07 is Friday 07:00-07:59. There are evident trends in users activity but is it possible to correlate this user activity to actual buying? Understanding this out is a work in progress the questions we have asked are:-
    Q: Is it enough though to draw a correlation between website users and on-chain purchases
    Q: Would it help to have the same global data to compare (would that need to exclude APAC?)
    APAC Sessions_vs._Day_Hour
  2. APWG baselines and targets for Website Users were based on Q3 averages. They are currently reported monthly which does not allow for a direct comparison but still sets a realistic Quarterly target.
  3. Percentage of Gold Silver Bronze Owls is down slightly due to a number of new contributors joining the community.
  4. APWG measure of diversity has changed and is reflected in the jump from 13.5% of All APAC Community (Oct) to 21% of APWG Active Contributors(Nov).

Of note, both Japan and Thailand have shown considerable growth in the past month. Japan has overtaken Australia as the leading country for website activity. Thailand has also driven significant increase in traffic to the website

APAC Users
APAC Users1035×676 90.1 KB

3 Likes
8

December & Q4 Performance Reporting

KPI BASELINE TARGET OCT NOV DEC Q4 OBJECTIVE DATA SOURCE
APAC Median Buy Size Required ? WIP Review UA Defer Wise Owl Not Measured Strategic Dune Analytics
APAC Net Buys Required ? WIP Review UA Defer Wise Owl Not Measured Strategic Dune Analytics
APAC % Trade Volume Required ? WIP Review UA Defer Wise Owl Not Measured Strategic Dune Analytics
APAC % N$F Required ? WIP Review UA Defer Wise Owl Not Measured Strategic Finance Nest
IB/BD Relationship 23 26 31 35 35 - IB/BD Notion
Bi Weekly BD IB Community Call 10 10 - 10 10 - IB/BD Head Count
Community Average Autonomy BD 2.5 3.0 - - - 2.33 IB/BD Survey
Community Average Autonomy IB 1.7 2.2 - - - 2.50 IB/BD Survey
Community Average Autonomy GM 2.8 3.3 - - - 2.00 IB/BD Survey
Website Users 16988 20% - - - 28637 GM Google Analytics
APAC Community Members 29 35 29 38 51 51 TOC Discord Dash
Gold/Silver/Bronze Owls 45% 45% 51% 38% 36% 36% TOC Discord Dash
Women/Non-Binary Contributors 15% 15% 13.50% 21% 22% 22% TOC Discord Dash

Strategic Objectives

The Universal Analytics (UA) was created to enable view, applied filters of global data. It was hoped that we might be able to identify patterns in use/time on website for the APAC region to align with on chain data. While patterns are evident, there is no easy /effective means to link on chain and off chain data relative to users. Instead, we need to better track the purchase funnels and buy signals such as “connect wallet” and “buy” actions on the website and app. The major challenge is that more than 90% of purchases occur offsite and so activation of customers on the website is a key long-term priority to better attribute website activity to conversion (on-chain purchases).

AAARRR

Our experience using GA to measure growth performance, highlighted several gaps. At this stage recommendations In regard to the AAARRR funnel for primary website performance metrics includes

  • Increasing number of Users as a measure of increased AWARENESS
  • Decreasing Bounce Rate as a measure of audience ALIGNMENT
  • Increasing Average Session Duration as a measure of ACQUISITION
  • Focus on Purchase Signals "connect wallet and “buy” as a measure of ACTIVATION
  • Increasing New vs Return Visitors as a measure of RETENTION
  • Connecting buy signals with on chain data to attribute REVENUE to web activity
  • Increasing the number of “Shares” or Reviews on content as a measure of REFERALS

CC @LemonadeAlpha @Metfanmike @dev

Website Users

Where formerly we identified Top 10 Top 50 Top 100 due to

  1. a lack of complete data for Oct in the universal analytics reporting (started Oct 17) and
  2. the changing list of countries included in APAC now appearing in analytics

APWG website performance measure was changed for Q4 reporting to "All Website Users " as identified by GA4 version of Google Analytics. This shows that website users increased ~60% from the Q4 baseline showing increased awareness of IndexCoop.com. Moving forward other funnel metric will be applied and monitored

New views have now been created and filtered by country for India, Japan and Australia/New Zealand. Creating a filtered view of the data needs to happen as early as possible in the process of establishing regional teams as individual tracking of GEO data being more granular will provide better insight into the website performance driven by regions.

In terms of the current views for APAC regional teams, data for January will be incomplete. But as of end of February will be able to report monthly KPI’s for each of these regions. Q1 Performance baselines and targets will need to be set using GA4.

Community Growth & Learning

Our APAC community grew well beyond our estimated target. Where we expected 35 community member end of Q4 we had 51. While we continue to maintain an excellent level of diversity within the contributor team (22% on target of 15%) more newcomers means that Bronze owl and above numbers fell as a percentage of community from 45% to 36%. This correlates to the drop seen in our teams Growth Marketing and BD confidence. To be noted here is that only 12/51 people reported. Confidence around Institutional business increased!

We recognised a need for improved process and communication with budding regional teams and outlined a framework for accelerating region growth, that focuses on communication from core builders and direct responsible individuals (DRI) to support regional teams to develop research, set strategy, objectives and KPI’s and identify their target market in order to begin BD outreach. At this juncture we await news of the new Nest structure and hope that our Q4 experimentation within the APAC region can inform global expansion moving forward

CC @pujimak_in @Lanks @Vanita @Louisaraj @kohei @incyann

2 Likes