Deciding How to Prioritize Engineering Work [Poll]

Product Engineering
1

Authors

This article covers important topics at Index, and so for those who we have yet to spend time with, here is more information on the authors:

Co-Author: Edward Previously built engineering organizations from the ground up as CTO/VP Engineering for Web 2.0 startups. Spent the past couple months helping @dylan think through the path to engineering autonomy for Index Coop.

Co-Author: Cavalier Recently quit web2 for Index full time, now shadowing Punia to build out the PWG we need and want. Previously founded and scaled product startups to 150 people as Product Lead and Chief of Staff.

Summary

With limited engineering capacity, deciding between products is crucial to improve the quality and velocity of our product output. We’d like to discuss how to better coordinate between EWG, PWG and the wider coop, so that we can unlock our potential to launch the world’s best crypto indices before our competition.

This post is not meant to be a direct discussion of Engineering autonomy, although it does have some implications on that front.

Engineering @ Index: A brief history

It’s worth highlighting that the large majority of engineering undertaken to create Index products was done by Set directly, or Set employees seconded to Index. This is important to recognize for several reasons:

  1. The technology required for our products is extremely complex, and sits at the forefront of software and financial engineering.
  2. The small but growing team we have is only capable of so much work at any one time.
  3. Until we have sufficient expertise within Index Coop, every new smart contract must have its code reviewed by Set Labs. This is prudent for safety, and is a significant joint effort often requiring multiple rounds of code changes.
  4. Every product that is launched requires rebalancing and maintenance, much of which is being performed by Set Labs. For the FLI products the burden is larger, and shared amongst Set Labs and Index through the Leveraged Indices Pod.

With the hiring of @ncitron and @0xModene the Index Coop now has its own small, but fully functional core engineering team. This a radical change from just a month ago, when there was effectively zero in-house capability to build new products. In addition, @dylan and @edwardk will continue to marshall engineering resources from Set Labs. Now that there are two distinguishable engineering teams, synergy is the name of the game.

722×508 39.9 KB

A simple diagram showing where engineering capacity comes from. Not meant to be precise.

Engineering Growing Pains

Right now the internal engineering team has a number of organisational challenges:

  1. We have little visibility into what we should be working on next. Often workloads come up suddenly because communication is scattered. Right now the team is busy getting ready for a successful DATA launch (:owl: w00t!), but what about after that? Knowing roughly what’s in the pipeline will make it easier to allocate the right people to the right projects. It will help us know who we need to recruit. It will also go a long way in preventing burnout by making it easier to communicate around expectations.

  2. Engineers are being asked to do significant research and communication around products that have yet to be prioritized. There are already several new product proposals that will likely be put up for DG1 voting. If they pass (and most do), even more precious engineering time will be spent in feasibility conversations.

  3. There is a strong reliance on Set for complex R&D, technical oversight and final code review

  4. Engineering requests by IIP mean that “nice to haves” are being assigned priority by INDEX holders.

This means that the Engineers are being pulled in different directions and not being allowed the time to focus on deep work. Ultimately this means that overall progress is slower.

Product Pipeline (Unordered)

At the time of writing there are five product ideas that have passed DG1 and are being readied for a final DG2 vote:

In our current process, whichever one of the products in the pipeline makes it through DG2 first and has the most vocal support is probably going to be the product that gets built first. All other products will be blocked, regardless of subsequent voting outcomes.

In addition there are numerous other features or products that we would like to build but don’t currently have the contracts available:

  • Fully formed products on Polygon
  • Leveraged products on Polygon
  • Productive indices (PAY / LDI / YHI)
  • NFT indices
  • Issue and redeem fees on simple indices
  • Management fees on simple / productive indices

It’s amazing to see these high quality product ideas coming from within and outside the community. Within a year of inception Index has become the leading crypto index organization, and a formidable community has formed. Each week we attract new talent, new ideas, and more enthusiasm for what is possible.

The dose of realism is that we simply can’t build all of the products in our pipeline simultaneously. Aside from the large number of products, features like mint and redeem fees require significant engineering development.

How to Get Better

Like other Product organizations, Index Coop would benefit greatly from an agreed roadmap and priority for smart contract R&D and product launches.

The roadmap/ prioritization should consider:

  • What we can launch with little effort / zero R&D (see product feasibility)
  • Technical feasibility for new contract types
  • Return on investment for the work required
  • Business opportunity

This obviously needs input from both EWG and Set Labs, but also the wider Index Coop community to ensure we’re as aligned as possible on priorities. To reduce communication overhead and give engineers more time to focus on building, we’d like to suggest that EWG interface mostly with PWG and Set engineering. The communication flow would then look like:

Set Engineering ←→ EWG ←→ PWG ←→ Index Coop Community/Stakeholders

The question remains, how should PWG coordinate with the community to prioritize work?

The DAO organization structure is unique in some ways, but is not immune from the need for effective decision making, clear prioritisation and considered tradeoffs. We can learn and borrow from the extensive best practice of product management from web2. In traditional tech companies, product managers have influence without authority.

We believe that PWG should be empowered to bring together stakeholders and produce a near-term roadmap. Perhaps this could be in conjunction with the Simple and Leveraged Indices pods. Importantly, the new structure should help prioritize between products and work to be done, not just decide in isolation if that product or task is something the community wants to do.

Any approach to deciding how we prioritize products needs community buy-in. Please answer this poll question, and comment with your thoughts!

Do you agree with EWG that the Product Working Group should be empowered to prioritize products and collaboratively construct a near-term roadmap?
  • Yes
  • No

0 voters

13 Likes
2

This is desperately needed. Glad to see @edwardk and @Cavalier_Eth working on this - very happy to have you both at Index Coop!

7 Likes
3

I am a new member so trying to understand the current scenario. Is there any time estimation done for a project that engineers take on? That will help create a roadmap with a timeline.

4 Likes
4

Very supportive of this – it’s a much needed step to evolving our product and engineering efforts!

3 Likes
5

Welcome @blockdev! A rough estimate of engineering effort is taken into account for any new product. See Revised Product Prioritization Framework. There’s also an excellent review of the product onboarding process led by @catjam going on in Request for Feedback: New Product Onboarding Process.

Sounds like you have some experience with technical projects. Find me on Discord edwardk#9722 if you want to get more involved!

4 Likes
6

Very well written and insightful analysis of the situation. Having built out product organizations I am in full support of this proposal but want to add some comments:

Completely agree, I would add a few benefits. With a clear roadmap, it is much more straightforward to communicate progress/prioritization, which improves transparency and clarity for the DAO. This also enables ideation with a public roadmap, let me know if you want any suggestions for tooling for roadmap ideation if we want to go down that path.

True, but would it be correct to say the small size of the team is a major limitation to our bandwidth to deploy new products and this is currently the main bottleneck to get more products out? If so , what is limiting us from expanding our bandwidth (talent/money/community support)?

1 Like
7

So pleased to see this @edwardk and @Cavalier_Eth - strongly FOR this and voted accordingly. Thank you for the really well articulated problem set and core idea to solve it.

4 Likes
8

Engineering capacity isn’t a resource that can be expanded quite as easily as just throwing money at the problem. The main constraint here is actually mentoring capacity. New engineers (even very talented ones) take one to two months of mentoring to get completely up to speed and this is if they are working full time. This is the amount of time it takes to build up the required domain knowledge and context over the problem space. While they are being mentored up, the existing team will lose capacity until the new engineer has ramped up. This means striking a balancing act between expanding engineering capacity and building new products.

This would be the case at a normal fast-growing startup.

Indexcoop also is subject to another problem. Web3 engineers with production experience is also a much smaller talent pool compared to general developer talent. There’s just less to draw from.

Finally, due to the nature of working of a dao, it is easier for engineers to leave with the context that they’ve built up (unless of course their incentives were aligned with indexcoop’s success). As such, the codebase seems to operate under a hybrid model of an open source project (support from stable group of core contributors with random contributions from the public) and a start up.

Given these factors, for the long term health of Index Coop’s engineering health, there needs to be:

  1. A long term vision on the product direction that Indexcoop intends to head towards. This should result in a more predictable workload.
  2. A steady backlog of non-critical engineering work that will allow new engineers to be mentored and on-boarded into the code base. This relates to point no.1 as a prerequisite to having a new engineer be onboarded is having a steady stream of work.
  3. Commitment from existing engineering team to carve out time to mentor and onboard potential new contributors which is balanced out with critical product work. Long term, the dependency on Set Labs Engineering is removed.
  4. An incentive structure that keeps engineers aligned for IndexCoop’s long term success, yet is only given after a certain investment/time from the engineer has passed.

Perhaps some of these already exist and I’m just not aware of it.

7 Likes
9

Thank you @FlattestWhite for the detailed breakdown of the challenges in expanding the developer bandwidth. I completely agree and fully support your four points. Though I would like to add:

  1. Despite adding new product personnel in complex software projects being a short term drag on productivity (know all about it as I run a satellite API company…), we need to proactively be onboarding more engineering sources, if we in ~3 months want to increase velocity. Maybe we are actively doing this already and I missed it, but if not it feels to me like it would be a relatively high priority task for someone (happy to help out), to piece together a plan to increase bandwidth. Understand many priorities right now but definitely think in a year we will not regret address dealing with this early.
  2. Given that retention is challenging given the nature of work and tough competition for talent, as well as the narrow talent pool, it might be worth to attract someone for a Head of People/Talent role. This role would include identifying + filling our talent needs, retaining talent, being an independent person resolving incentive misalignment etc. Given our aggressive growth plans, it would imo be wise to onboard such a person sooner than later as the Coop’s people are ultimately what will make it successful.
3 Likes
10

Just wanted to circle back and thank everyone for their input and support, especially @FlattestWhite and @robdog for raising important points on scaling up the engineering team!

@Cavalier_Eth and I are looking forward to building on this conversation in future posts and the upcoming Future of Product and Engineering workshop.

7 Likes
11

Maybe an example of how prioritization is handled in a web2 tech company can give some new ideas. Prioritization is our biggest challenge. Resources are scarce and, with every release, we need to decide what to build and what not.

How priorities are decided?
There is no magic formula to identify a high-priority development. We try to decide that by asking ourselves what will bring more value to the customer. Value can be derived in many ways: financial pay-off, customers’ experience, time to market, tech complexity, and so on. We don’t use a fixed formula for this (maybe something that we can introduce?). Prioritization calls are made by taking into account all the previous factors.

Who decides the priorities?
This is the product team. The reason is that product represents in our organization the crossing between the business and the development side. They are the team that has the most context in what the market needs on one side and what can be built on the other side. They also have the right information to “calculate” the product added value, which is used to determine priorities.

How priorities are communicated?
We use a product backlog (a list of all the things that we plan to do no matter how far in the future). The product features are ordered in the backlog according to their value. The items at the top are the ones with the highest prio. while at the bottom you have the least priority. The backlog is publicly available. It makes clear for engineers what is the next thing to focus on as well as for other parties to know what is the next product/feature to be launched.

Happy to share more of my web2 product experience if you find it useful :slight_smile:

4 Likes
12

New engineer here! Your post was a super informative read. Thanks! A couple thoughts…

A steady backlog of non-critical engineering work that will allow new engineers to be mentored and on-boarded into the code base. This relates to point no.1 as a prerequisite to having a new engineer be onboarded is having a steady stream of work.

This is a great suggestion and could likely take the form of a Jira backlog or similar issue tracking product - even a free Trello board.

Commitment from existing engineering team to carve out time to mentor and onboard potential new contributors which is balanced out with critical product work. Long term, the dependency on Set Labs Engineering is removed.

Index Coop recommends completing CryptoZombies but not much else. Perhaps Index Coop-specific documentation/tutorials could lighten the burden on the existing team to mentor newbies.

2 Likes
13

Great post. Let me add that a lot of the methodologists have a quantitative background and so some of the math-oriented, non-programming work could be delegated to them. I’m thinking of liquidity analysis for example.