Engineering Constraints Follow Up & Next Steps

1

Where are we today?

Since roughly the beginning of Q2, the EWG has set out to reduce technical operations overhead on the Set team. This includes independently deploying monthly contributor rewards, funding liquidity mining updates, calculating & paying out methodologist rewards, and conducting product rebalances. Reducing this load on the Set team enabled core smart contract devs to focus on technical improvements to existing products (e.g. FLI) and new product development.

As measured against this original goal, progress has been fairly acceptable. Many of the highest cost, lowest context technical operations (e.g. monthly contributor rewards) have been off boarded to the Index Coop entirely. Onboarding to more sensitive/high context technical operations (e.g. product rebalances) are underway and making steady progress.

However, the original goals were set with limited information, and those goals are now clearly inadequate. The Set engineering team have spent the past several weeks examining the current system & processes and produced this rough list of technical capabilities the Index Coop must achieve in order to succeed in the short to medium term.

Key Results - How You’ll Know if the EWG succeeds

Indices

  • Rebalances: Infrastructure to handle 10 rebalances a week safely w/o MEV, slippage issues
  • New Products: Preparedness to launch 2 index products a month
  • Maintenance: Ability to handle 100 metagovernance transactions a week

Leverage Indices

  • Supply Scalability: Infrastructure to scale ETH 2x FLI to 2,000,000 supply (400k today) for 10 products
  • Maintenance Scalability: Teams and infrastructure to support 10 products, their parameter updates, notifications, etc.
  • New Products: Preparedness to launch and support 10 products

Previously, the EWG set out to reduce technical operations overhead weathered by the Set Team.

The new goal is to scale technical operations of Index Coop products. Scaling these operations will unlock the next stage of growth for the Coop & enable more products to be launched more quickly.

What do we do next?

Engineering has been the least understood and least effective working group of the Index Coop because we have relied on Set Labs to service almost all technical needs. It is now clear that in order to satisfy just the current number of technical operations, we need to begin aggressively investing in the engineering processes to enable many more technical contributors. The first glimpses of what that future org & process might look like is in the FLI Experts Group, which is gaining a level of domain expertise that will soon surpass Set Labs’.

Therefore, the Engineering Working Group is rallying around two initiatives:

  1. Aggressively scaling technical operations: the near term goal that will unblock product launches at the Index Coop and put us back on track to advance key metrics (AUM, revenue).
  2. Organizing effective engineering working group processes: a requirement to sustainably scale the required technical operations.

Aggressively Scaling Technical Operations

In order to scale technical operations we need:

Current Process Documentation

EWG will need help from many more contributors. Contributors need high context on the current technical processes in order to make decisions about how the system can be scaled. So far context has been most commonly shared via detailed forum posts & documentation, but other context sharing strategies like Q&A’s and process walkthroughs have been very effective and will be promoted.

Technical Process Design & Automation

Once enough context has been shared, expert groups can begin establishing processes to efficiently & safely own aspects of technical operations, with the goal of maximizing automation & safety.

Organizing Effective EWG Processes

Organizing effective EWG processes means answering the question: how is engineering work prioritized and executed in the Index Coop?

The FLI Experts Group is showing great promise and has been set up in a relatively short amount of time. Piloting other expert working groups tasked with establishing safe processes for technical operations is another near term goal of EWG. This means mapping out the expert group’s internal processes & defining its interface with the rest of the Coop.

Organizing effective EWG processes also means agreeing on engineering best practices, how code reviews are conducted, how code is deployed, how repositories are maintained and more.

Roadmap, Vision & Engineering Culture

In the medium to longer term engineering has important questions to answer regarding:

  • Roadmap and Vision - Where do we want the EWG to be by the end of 2021?
  • Engineering Culture - How do we attract new high quality contributors and level-up EWG members?

These are extremely important questions that we will work to answer soon. But right now the top priority is addressing current technical & operational needs and unblocking product launches.

Next Steps

This is mostly a context sharing post to create a shared understanding of our major goals. The specifics of how the technical operations expert group should function are still a work in progress. We will need people to help us do this. Come talk to me if you think you can help!

9 Likes
2

Have experienced this within the last week and I quite like the effectiveness of this

This has the capacity to be the most underrated thing we do. These need to be here and hammered out clearly with the assumption that we will be having things worked on by people that aren’t familiar already with our code and processes.

There are many things we can do to ramp up capacity relatively quickly, but they potentially come at a high token cost and don’t necessarily last. How best can we thread the needle of putting something up on Gitcoin and make sure we maintain the culture the Coop currently fosters, one of earning our contributors?

5 Likes
3

Thank you, @dylan - this is an excellent follow-up that addresses many of the concerns I shared on Punia’s original post on Engineering and Product Constraints.

I am :100: percent on-board with this and believe it’s the key to unlocking products from many methodologists that @overanalyser has outlined here.

Please count me in on helping to resolve this problem specifically:

It is a must-have for @Kiba and I to be able to launch our proposed Data Economy Index (DATA) and I’m willing to pull out all the stops to make this happen.

I will be attending the EWG meeting this Wednesday (6/16) and would be happy to spearhead a Technical Operations Working Group to begin making progress on the required items you have you outlined:

4 Likes
4

Thank you to @puniaviision for starting this conversation here and to @dylan for continuing it here. It seems to me that sincere thought, time and motivation is now going into this process - from many angles. This is a positive development and it’s good to shine light into this issue, which I think is the most important issue in the DAO. This is probably not easy to surface and communicate and I think these ambitions are spot on:

Key Results - How You’ll Know if the EWG succeeds

Indices

  • Rebalances : Infrastructure to handle 10 rebalances a week safely w/o MEV, slippage issues
  • New Products : Preparedness to launch 2 index products a month
  • Maintenance : Ability to handle 100 metagovernance transactions a week

Leverage Indices

  • Supply Scalability : Infrastructure to scale ETH 2x FLI to 2,000,000 supply (400k today) for 10 products
  • Maintenance Scalability : Teams and infrastructure to support 10 products, their parameter updates, notifications, etc.
  • New Products : Preparedness to launch and support 10 products

I want to get a negative comment out the way first before moving on to more positive things and trying to move this conversation forward for all of our and our clients’ benefits.

First, it is very disappointing to get to such an acute choke point situation regarding engineering resources, period, and without much prior remedial action or future planning (unless I missed it). I personally find communication and especially planning regarding this most important resource in the DAO to be insufficient, and hard to understand given:

  • Set’s 28% long-term incentive in the DAO
  • A fast-growing community of people since many months ago who are committing real time and money to the Coop
  • Rapidly rising AuM
  • Great PR and product traction
  • And, general market understanding from thought leaders that we are beginning to win this space!

We have been operating since the 6th of October '20, had some small sparks of action regarding hiring more engineering resources with the engineering lead job post early in '21, but it seems little has happened since - and now we find ourselves here. If we were a tradfi Co - and I don’t think we’re sufficiently different to render this comparison obsolete - the spotlight would be on the product manager and engineering leads to rationalize this. (‘Why was this not foreseen and planned around more?’ ‘How can we learn from this and not make the same errors again?’). We are indeed all in this together, and as has been described, ‘three legs’ of a stoll (Set, DFP, IC community), but at times each leg should be open to direct, respectful and human feedback. The same applies to the IC community where feedback is due from time to time.

In my view, this situation is the largest short-term, strategic threat to the DAO, impacting in a number of ways.

  • We’re less able to launch products as fast as we’d like (on an absolute basis and relative vs competitors) to keep trying to win this space (‘speed kills!’)
  • Contributors are less able to launch their popular index product ideas within the 18-month Methodologists Reward scheme (mentioned elsewhere by @Thomas_Hepner. And, the 18-month’s timeframe being rather short in nature is even more spotlighted now). This seems inequitable
  • External partners are being impacted in a way not forewarned
  • Building engineering teams takes a good deal of time in tradfi, let alone DeFi where skills are even more sought after.

While it’s definitely helpful this issue is out in the open, until this issue is resolved it could also affect our ability to encourage talent to move from interested/part-time to full-time/soul invested. It’s affected my motivation. I’m aware and appreciative that Set being involved in IC since the beginning helped jump start things at the beginning (no doubt! - instant protocol to launch on), but the situation now is problematic and seems very foreseeable.

Second, and onto more positive things: how to move forward? This matters more and I think boils down to understanding these items, with crude direction but not perfect precision, and then solving for them.

  • How many engineers, with what skill sets, work on IC each month in Set? (Noting your comment @puniaviision to @Matthew_Graham: I wouldn’t want to seek hourly time sheets for engineers (waste of time and would p*ss engineers off), but crude direction as described. From experience in a different market, our engineering teams have broken down into: front end, full stack, back end, infra/devops, info sec, QA, data/quant, HFT, and other, who’s time on various projects and releases various a lot and is communicated via simple Google Sheets documents with GANT-chart-like representations (coloured cells for each resource) to the wider team. Over the last months I’ve heard huge variance regarding what Set’s resources we use: ranging from ‘one engineer week/month’ (seems like a problematically low amount or resource but could be a rumour gone viral) to ‘most of the Set team’s time’. What is the actual resource committed?
  • What is the demand profile for these skill sets in an average month or quarter? Ie - steady, surges, product-launch spikes, etc?
  • On what timelines can we realistically seek to off-board these skills and value adds from Set to IC engineering? Pros and cons of this approach and other options which we have available. I’m sure short-term, drastic measures bring some potentially undesirable risk and volatility. We’ve seen some communications regarding FLI maintenance and management, as well as rebalancing here (great to see!)
  • Are there gates and checkpoints to this process and how they work? Ie - Dylan/other approves first non-Set engineer to release code of ABC types to production. Smart contract code being the highest value and trust level here. How do we rationalize and communicate these gates/checkpoints to the wider community?
  • How many engineering resources of which skill sets do we need? Given we’re a DAO, it wouldn’t be a surprise to see a larger group (relative to tradfi) of a mix of full/part-time resources
  • What incentives do we need to consider to attract and retain this engineering talent? It seems a more formalized EWG is being planned, but I would be open to a LARGE funding of this WG - knowing that these peoples’ skill sets are valued very highly in the market (more than most peoples’ here!). Mission matters for sure, but when there’s a feeding frenzy for talent, rewards of USD/stables/$INDEX talk loudest.

I would assume there’ll be a post and poll on the more codified formation of this EWG. I’d encourage other Owls to keep their minds open to a potentially large funding cost of this WG given the market for engineers generally and especially for top notch DeFi engineers. Our products are the complete merge of finance and tech - much more so than Blackrock or Vanguard - which brings positives and negatives. Positives: efficiency and scale. Negatives: if we get the ‘tech’ piece wrong, the consequences are even more problematic - potentially devastating. (And I say this as a non-engineer!).

This is shared in the spirit of cooperation but also honesty - I’m amped up because I care and I’m upset - and I hope to see the DAO solve this most critical issue rapidly, to more effectively win this market and deliver potentially life-changing products to our index product investors and value accrual to community members and $INDEX holders. This is a very sub-optimal situation right now, but also an opportunity to right it - and in doing so set ourselves up to realize this much touted ambition to become the ‘decentralized Blackrock’.

How we respond and improve is critical.

12 Likes
5

+1 on this. The idea generation talent is already here, but it seems ideas are languishing once on the table. The INDEX.labs post seems like a practical way to approach this in the short run but both sides need to be advancing now (how to launch simple products quickly plus a way to launch ideas that require engineeering) recognizing the time it will take to develop the engineering team.

Also agree with this – Transparency on the resource draw might help on multiple fronts: 1) better understanding of the problems faced may lead to less frustration within the community, 2) clearer description of the type of talent needed might make it easier to identify relevant talent.

This could be an opportunity to keep advancing the discussions on varations of CTOP and UMA Options if large funding without strings concerns anyone, but I’m 100% onboard with your sentiment.

4 Likes
6

@0x_Dev @prairiefi
Thank you for your feedback, this kind of engagement is essential and very much appreciated.

I’ll keep this response brief because there’s more context sharing & data (re: engineering resources) coming down the pipeline that will allow us to start fixing these problems.

I share your frustration and am 100% accountable for not getting Index Engineering to where I know it could be today. Part of the reason for this is a flawed understanding of what Index Coop’s role in engineering should be. We’re learning now that Index Coop engineering autonomy is a much more urgent need than previously thought, not least for the transparency around engineering decision-making & prioritization that the Index Coop will get. A post detailing that change in thinking, and the engineering requirements to reach high autonomy are forth coming.

14 Likes
7

@dylan thank you for your response. This is a very honourable and big thing to say, though I think it goes beyond you too. :fist_left:t5:

I’m excited to see this:

A post detailing that change in thinking, and the engineering requirements to reach high autonomy are forth coming.

6 Likes