What we learned from the survey

Primary Authors: @helmass, @MaryQ
Review: @DarkForestCapital, @Pepperoni_Joe

We should all be extremely proud of what we have accomplished at the Index Coop.

Executive Summary

This post is extremely long, please make sure you block at least 30 minutes to read/reflect. We won’t judge for only reading the summary.

Our goal:

  1. Establish baseline data re: contributor engagement

  2. Determine areas of further exploration and improvement regarding
    – The current state “leadership” structure
    – The Index Coop’s core purpose
    – Ways of organizing work and people
    – Decision-making protocols
    – And, how the Index Coop might reinforce an inclusive, fair,
    and respectful culture

  3. Convey a narrative/provide context based on survey analysis,

  4. Display raw/objective/neutral commentary from the survey,

  5. Identify areas that we can improve on at the Index Coop and an action plan to progress as a DAO.

Here is an outline of the post:

  • Research methodology, potential data collection pitfalls, and how to improve the survey in future iterations
  • Research participation information
  • Convey a narrative and provide context of the Index Coop, while layering in survey responses and data gathered from leadership interviews
    • Values, principles, mission
    • A journey through the DAO and organizational structure
    • How we work together internally and externally
  • Workshop areas of further exploration and immediate action

What we do well:

  • We have establish a safe and welcoming environment
  • We trust each other
  • People are committed
  • We have a unique opportunity to change the trajectory of haves and have nots
  • We share common values
  • We have both heart & rigor
  • We continue to attract top notch talent
  • We embrace principles guiding our work
  • We built a culture of care
  • We are self starters
  • We’ve grown with strong servant leadership
  • We have proven results
  • We have deep intellectual & cultural curiosity
  • We offer a safe & welcoming environment
  • We find like-minded people here

The list goes on.

Areas of opportunity:

Anticipate developmental and planning milestones

  1. Mission, strategy & purpose
  2. A people/community first approach
  3. Culture, principles and values
  4. How work gets done

Actions recommended as a result of our findings/workshop:

You can find initial feedback and our post workshop synthesis here

Find the Index Coop Organization Assessment Workshop Video here.

Research methodology and focus

Our overarching goal of the research was to identify what we are doing well, areas of opportunity, and topics of further exploration and immediate action. The information provides a baseline set of data and can be used to monitor contributor sentiment and engagement over time.

Here are the themes we were focused on.

  • The current state of the leadership structure
  • The Index Coop’s core purpose
  • Ways of organizing work and people
  • Decision-making protocols
  • And, how the Index Coop might reinforce an inclusive, fair, and respectful culture

To uncover contributor perspectives we

  • Conducted 20, 60-minute 1-1 interviews with selected members of Index Coop Leadership team including Working Group Leaders, methodologists, key strategic partners from SET and DFP, methodologists, and active forum contributors. The discussion guide leveraged Roger Schwartz’s, Group Effectiveness Model and therefore included topics such as group context, structure, and processes. SCHWARTZ, R. M. (n.d.). Skilled facilitator: Practical wisdom for developing effective groups. Jossey Bass Wiley.

  • Implemented the August 2021 Pulse Survey (50 respondents) using a Likert scale rating system to understand contributor perspectives and attitudes. The survey was also designed to explore the same three key themes: group context, structure, and processes.

  • Throughout we evaluated the Index Coop and these topics at three levels; individual, group, and organization.

Who participated?

20 Index Coop leaders participated in 60-minute, 1-1 interviews including, in alphabetical order:

  • ​​AfroMac, BigSky7, DarkForestCapital, DeV, Dylan, Fallow8, Felix | {set}, Jo K | DeFi Pulse, JDCook, Kiba, Lanks, Mel.eth, Metfanmike, Mrvls_brkfst, Mr Madila, Over-analyer, Pepperoni_Joe, Puniaviision, Thomas Hepner | Titans of Data, Verto0912

50 contributors/leaders also completed the August 2021 Pulse Survey, representing 55% of those who submitted contribution sheets in August. We consider the level of participation in this inaugural Pulse Survey a huge success. Here is a more extensive representation of survey participants.

Length of time spent with the community

Hours devoted per week

Location

Discord tag/identifier

It’s important to note that the information above is displayed as Owl Level (or contributor identifier)/total survey participants. If normalized, data would likely convey an evenly distributed balance of participants across the Owl Levels.

What we do well; bright spots

There were many things to be proud of as a community. Here is some raw qualitative commentary that stood out to our team. This is just a sample, there were many positives to highlight.

  • Love it here. Ya’ll changed my life
  • Participatory, open, scrappy
  • Collaborative, constructive, diversity of thought, civil
  • The DAO is welcoming, inclusive, young, ambitious, collaborative, and growing
  • Bright, energetic, sincere, capable, cohesive, and driven on a singular mission
  • Vibrant, open, kind, welcoming. Full of knowledge and power
  • Inclusive, inspiring and a bit of a mess
  • Very active crowd of smart and kind people working (mostly) together
  • self organizing. rewards initiative, high ownership. cooperative. mostly positive sum. skeptical. optimistic. full of energy.
  • Open and community led. Everyone has a voice (both good and bad at times). High performing individuals

WE SHOULD BE PROUD OF WHAT WE HAVE CREATED AT THE INDEX COOP

A deeper understanding of the Coop: Values, and principles help shape organizational culture

Why organizational culture is important

Formal and informal influences can motivate employees in different ways. Formal influences such as compensation contracts can provide rewards or extrinsic motivation while culture helps create a desire to perform a task for its own merit. Every organization has cultural traits, and those cultural traits can have positive and negative impacts on the business.

People first

90% of the contributor/leadership base (45 of 50 respondents) said the Index Coop feels welcoming, while 84% (42 of 50 respondents) highlight that the Index Coop feels safe. Although there is room for improvement, these numbers screen positive.


Leadership felt less safe than the contributor base. 30% of the leaders felt unsafe, while 12% of the contributor base felt unsafe. We should monitor this in future survey iterations.

Q: The index coop feels safe?

Values and principles act as the pillars which define our culture

Values are beliefs that motivate attitude and actions. They help us determine what is important to us.

When asked which values are essential to successful DAO contribution, we found that our community values accountability, collaboration, adaptability, and trust the most.

Individuals could select up to five values. Leadership and contributor responses are combined.

The list of values we offered in this survey is from Barrett Values Center: Values Lists for Culture Transformation Tools. We chose to use this standardized tool to assure objectivity.

Our Guiding Principles

Our guiding principle, mission, purpose are all part of the Coop’s strategic toolkit. The principles establish and communicate our core business promises related to how we work together. These are in place to set expectations and establish an understanding of desired action, attitudes, and mindset. Here is a link to our guiding principles.

Here is a quick summary of these principles.

If our goals state where we want to be, then it’s our principles that decide how we get there. We think about these in three distinct sections:

  • Business
  • Community
  • Personal

The sections above include these values.

  • Business: shared goals, innovation, performance, data driven, community governed
  • Community: long-term thinking, people > profits, fairness, empathy, internal motivation, transparency
  • Personal: communication, autonomy, self-organization, growth, and improvement

Our values (from the survey results) generally align with values displayed within our guiding principles (as shown above), showing thematic overlap. Values stated in our guiding principles that up fewer times in the survey results include: fairness and long-term thinking.

When we asked the community if these guiding principles are clear and improve our work at the DAO. 86% of the contributor/leadership base (43 of 50 respondents) said that we have clear principles that guide our decision making and 84% (42 of 50 respondents) highlighted that our mission effectively guides our work.

Have we established an organizational culture?

When asked to describe the culture at the Index Coop, this is how we responded.

Note that data is in its raw form. It is possible that selection bias was introduced during this process, however, our objective was to show the raw contributor/leadership level data and to maintain a balanced/neutral view.

Positive responses

  • Welcoming, Intelligent, fast-paced, decentralized but collaborative and open-minded
  • The culture that exists in this DAO is of Business through a community lead initiative, reward-oriented contributions, diversity in inclusion, and equal opportunities to contribute
  • Positive & more efficient than a traditional organization. Hard workers are recognized within Index Coop DAO. A new joiner can quickly assume a senior position & does not need to ‘wait’ as is common in traditional organizations
  • Open
  • Lately, the culture is more combative, hostile, and focused on self-interest. For a long time, however, we did live by our guiding principles having a culture that was open, collaborative, empathetic, innovative and solution-oriented
  • Open and community-led. Everyone has a voice (both good and bad at times). High performing individuals

Neutral/unclear responses

  • The culture is preached as well at each meeting on an initial slide (Principles)
  • I think people are confused on what creates value and what the future state should look like other than to launch more products
  • Well-intentioned but it seems a bit stratified and hierarchical/hard to access
  • Large enough to have a wide spectrum within it. Generally, a safe space filled with meaningful commentary. Very short distance between idea and action. The distribution of power is skewed and there is also blatant dishonesty from DFP on the forum in not acknowledging the work done by the coop and the costs that they bear. It’s great to have open and direct connection with the leads of working groups but they value work arbitrarily and differently, IC needs more structure in this department to approach fairness even further.
  • Humanity in a technological age. Empathy & respect plays an important role. Ego & self should not belong in this culture
  • A weird blend of structured and chaotic. Envision a balloon that is being squeezed into a shape but the gaseous molecules inside are ricocheting in different directions 🡨 MIND BLOWN HERE :blush: and I REALLY LIKE HOW YOU WRITE
  • Lately, the culture is more combative, hostile, and focused on self-interest. For a long time, however, we did live by our guiding principles having a culture that was open, collaborative, empathetic, innovative, and solution-oriented.

Negative responses

  • Bias for Inaction, Political, Corrupt

Interestingly, describing the DAOs culture is a challenge for the DAO. Views vary widely.

How work gets done at the Index Coop

How we work together is highly important in determining our success as a DAO. We first seek to understand the individual experience of our contributors and leadership. Thereafter, we will investigate the working groups and key constituents within the DAO.

The journey we take at the Index Coop

The journey toward DAO contribution is challenging and there are barriers to entry.

These include:

  • Learning a new language (the language of crypto)
  • Learning multiple tools (discord, discourse (forums), twitter, snapshot, notion, google hangouts, google drive, etc…)
  • Understanding where and how to show up. (Essentially, navigating the DAO onboarding experience)
  • Balancing contribution and compensation (trial by error, standards of quality, transparency, and communication)
  • Managing boundaries
  • Offboarding

A people/community first vs. operational approach

We identified what attracts people to the Coop and crafted these insights into initial archetypes.

When designing our survey, we set out to explore the individualist perspective within the DAO. In doing so, we focused on questions with “I statements”. Collectively we best understand how our contributions are valued and how our contributions impact the DAO’s goals. We also feel like we can participate in decision-making.

90% of the contributor/leadership base (45 of 50 respondents) said that their contributions are valued and 88% (44 of 50 respondents) highlighted that they know how their contributions impact the DAO’s goals. 44 or 50 respondents also felt like they were able to participate in decision-making.

How the work gets done

Before diving into the results, we think it’s important to first empathize with our leaders.

Leadership is difficult. There is no playbook. Leaders bear responsibility. Leaders display vulnerability.

A few articles that resonate re: leadership.

To all the working group leads. Thank you.

Our findings

We looked at three variables to get a better understanding of our perspective on working groups at the Coop. The three questions we asked were:

  • Rate the value each Working Group (WG) provides the Index Coop
  • Rate your level of confidence that the leaders of the Working Group will carry the Index Coop successfully through the next nine months
  • And, rate your level of confidence that the Working Group is providing the right level of transparency

Value

Results were generally positive, with respondents (contributors + leadership) signaling WG value-add across the board. The average WG score (very high value + high value) was 85%, with Engineering (95% and Business Development (96%) ) scoring the highest. The lowest scoring groups were Language Operations (a new group) at 74% and the Funding Council at 75%.

Confidence

The community is confident in WG leadership capabilities. The average score across the group (measured by very high and very high confidence) was 85%, with Institutional Business and Analytics leading at (100%). The lowest-scoring groups were Product (76%), Autonomy (65%), and Growth (54%).

Transparency

Respondents believe we could improve transparency at the Index Coop. While some areas are well organized there is still a need for more processes as everything is so async. Here is some respondent commentary below.

  • Yes on the point of transparency. Our weekly calls used to be updates from each working group lead about what is going on. To increase efficiency of our calls, this has changed and it’s now only about tasks or initiative most pressing in that moment.
  • Having at least a monthly call that is used for each WG lead to dive into what they’ve been up to, and what they are planning to do, would be a good step to increase transparency. Maybe it’s because of the turmoil around the recent autonomy and fee split discussion, but I have no clue what for instance Analytics is doing the whole week, or what the guys from the design team are working on (the new website seems to be in the making for about 4 months now)

The average score across working groups (measured by high and very high confidence) was 63%, with POC (82%) and Creative & Design (80%) scoring the highest. The lowest scoring groups were Growth (44%) and the Funding Council at (46%).

Internal and external relationships

Finally, we looked at core stakeholders within the broader Index Coop community (both internally and externally). Internally we focused on contributor/leadership trust and externally we focused on our working relationships with Set and DeFi pulse.

Internally: We display strong levels of trust with 96% of respondents trusting peers and 86% of respondents trusting leadership.

Externally: We generally agree that we have a productive working relationship with Set, while our relationship with DeFi pulse is considered unproductive.

Areas of opportunity

Based on the qualitative and quantitative data gathered we have a good baseline understanding of the current context and narrative within the Index Coop. Using Schwartz’ Group Effectiveness model we analyzed our community through multiple perspectives (individual, group and, organization). Now it’s time to identify areas of opportunity, exploration, and actions. Insights gathered here inform our path forward.

Pillars of improvement

It’s time to learn and grow as a community. We identified four focus areas within the DAO to improve the quality

  1. Mission, strategy and purpose

Let’s circle back on the commentary above re: culture. Organizational culture is difficult to define and we struggle to define our culture at the Index Coop. We share similar values and are rooted in principles that guide us. Our community is generally safe. Our mission, however, is less clear. Bringing clarity here should ultimately allow us to better communicate our culture over time.

In referencing our guiding principles, it was interesting to see a mission statement buried in the body of the community portion of the document.

  • Our mission is to help everyone in the world achieve financial freedom

This was an interesting find and we don’t believe this is widely communicated within the Coop. When conducting leadership interviews, it was evident that the mission was unclear. Here are some qualitative highlights.

  • We’re a community focused on making crypto investing simple
  • We will be the BlackRock of Decentralized finance
  • Our mission is to be the best DAO. To attract and retain diverse thinkers, writers, builders, do-ers and challengers

Embracing a purpose-led, mission-focused stance should ultimately positively influence our community, allowing us to better define our organizational culture. A communicable, well-defined culture ultimately ensures that we continue to do the right things for the DAO in the long-run.

To reiterate, our values and principles are clear. Our mission is less clear.

Strategy is the formal logic of our goals

We scored poorly when asked if we have a clear strategy to achieve our mission. This question is two-sided in that our mission (or unclear mission) could impact the result, among other factors like lack of transparency and feedback. 29% of respondents believe that our strategy is unclear to achieve our mission. Results are aggregated, however, leadership and contributors shared a similar sentiment.

  1. People/community first, vs. operational, approach

A people-first approach takes into account the physical and psychological needs and motivations of the individuals who comprise the Index Coop. Although the Index Coop feels safe and welcoming for most, 13 of 50 respondents felt like the Index Coop does not feel fair.

One possible explanation of this could be due to our poor scores around feedback, extrinsic rewards, and a DAO-work/life balance. This should be explored further to understand the scores. 28% of respondents felt like they have a poor DAO-work/life balance. 72% indicated that they receive helpful feedback to improve their contributions and % indicated that they were rewarded appropriately.

One of the more interesting observations was our leadership’s perspective on a DAO-work/life balance. Leadership’s perception of a work-life balance was low. 50% of the leaders who responded indicating their DAO-work/life balance did not leave them feeling energized.

Q: I have a DAO-work/life balance that leaves me feeling energized

  1. Principles, values and culture

  2. How work gets done

How we work together is highly important in determining our success at the Index Coop. A few things which stood out as areas to improve include greater transparency among working groups and improving the relationships with our external partners.

As highlighted above, the average transparency score (measured by very high confidence) was 63%, with POC (82%) and Creative & Design (80%) scoring the highest. The lowest-scoring groups were growth (44%) and the funding council at (46%).

While unsurprising, one of the more concerning findings was the relationship the community has with DeFi pulse. In aggregate, the majority of our contributor/leadership base feels we do not have a productive working relationship with DeFi pulse.

Lessons Learned

How can we improve the survey going forward?

  • Measuring inclusion: One of the main sources of feedback we received was around identity insensitivity. Specifically, data collected around age, gender, etc. were overly narrow. For this survey iteration, we have not included this information within our analysis and agree with the sentiment around the concern. At this point, we have not decided how this will change in future iterations, however, we will 100% be more thoughtful and spend more time here.
  • Survey implementation: To segment insights from leadership and contributors, we created two surveys which “lived” in discord. Our forum post took members to the respective channels. Effectively we were trying to gate the survey so contributors could not fill out the leadership survey and/or community members could not fill out the contributor survey. This created extra steps which we hope to eliminate in future iterations.
  • Question improvement/fatigue: When asking questions in the future, we would like to reduce the lift on those filling out the survey. For instance, when looking at values there were many choices to choose from. This was by design during this survey, however, our goal is to make this more manageable in the future.
  • Analytical insights: We can further segment data collected, which could potentially provide more meaningful insights/understanding of our community. I.e. we could layer in more econometrics and statistical analysis in our findings. For the next iteration, we will be looking for someone to lead this effort. Please shoot me (@helmass) a message if interested.
  • POAPs: While this is not necessary for improvement, in the future it would be nice to drop a badge to those who complete the survey.

How can we improve the overall learning plan?

  • Difficulty to keep up with the speed of change, the number of meetings and information shared daily via Discord and the Forum are overwhelming. It was fatiguing and left me (@MaryQ) feeling behind and questioning if the work would be timely, accurate, and valuable. I believe the entire system may experience this fatigue.
  • Qualitative interviews help us understand why and how, where surveys help us know what. Twenty interviews provided an abundance of information and insight, but it took time. Every 60-minute interview conducted required at least 90 minutes of analysis and synthesis. We took on too much, given how busy the Index Coop is, how quickly things change, and American University practicum deadlines. While the process was very inclusive, doing less may have improved the quality of the analysis.
  • Index Coop seems to value quantitative data over qualitative/observational assessments. The Coop states its commitment to rational decision-making and gut reactions, reading the room, tone of voice is hard to capture in the survey but is critical data. No decision or communication is without some emotional influence; this is human nature. How might the Coop acknowledge and appreciate all aspects of available data?

Potential biases and errors

  • No research is objective: While we used a reputable organizational design model to guide our research, we also brought our curiosities and background to the learning plan, each interview, survey design, and data analysis. We made every attempt to ground our assessment in facts, and we made choices regarding what to include and what to leave out to keep the presentation crisp and engaging, we acknowledge that this can impact the narrative

  • Measurement error: The construction of the survey could have introduced measurement error. For instance, survey respondents might not have understood the question we were asking. Question presentation and flow could have introduced bias in respondents’ answers. Etc…

  • Selection bias: Data was not gathered through randomization. Respondents of the survey could be the ones that “drink the kool-aid” and/or take a highly contentious stance re: the Index Coop.

  • Variable correlation: Variables (or questions) can be highly correlated, which could impact validity of the survey.

  • Halo effect: Judgement carryover from one question to the next. This would result in measurement error.

13 Likes

Hey team - great work here. Lots to digest here - and some very good learnings across the board. I’m seeing a number of ways for us to take the information in this survey and use it to improve the operations of BDWG ( @Mringz ) . Work like this is what allows us to level up and grow as an organization.

5 Likes

Work like this is so necessary - the foundation for conversation vital to both our health, and the bottom line. Thanks for valuing this enough to shed some light on this topic and get things rolling!

4 Likes

@helmass and everyone involved, great work! interesting insights. On the point of contributors seeking feedback, the mentorship program that women in Index are piloting would definitely help. Also curious to see on the lower score on work life balance especially for leadership, what are the primary issues, how many hours do they work etc.

3 Likes

Thanks for taking the time to read @Alks4778 ! While the quantitative data wouldn’t show this, I believe qualitative conversations are pointing to overwork within leadership. Essentially driving a low work life balance score. Happy to see how the Women in Index can collaborate with us

2 Likes

This is epic @helmass ,

Thanks for all the work that has gone into this. Pivotal time to be conducting this type of research.

I’ve read through a couple of times and have some thoughts/questions/ideas.

Working in a traditional startup/business environment, this research is easier. We send the surveys to leadership, full-time staff, part-time staff and contractors, and segment the data accordingly.

Obviously, this is far more difficult in a DAO environment where there are hundreds of people contributing and operating on the fringes.

In (1) goal:

Would it be a fair statement to make, that the 50 pulse survey respondents aren’t necessarily representative of the broader contributing community? More-so the core contributors who’ve overcome the initial barriers to entry? Wondering how we better understand these metrics. Maybe the new owl level discussion will feed into this? @Pepperoni_Joe

If we have the following goals:

and:

I think we should be looking for ways to collect data and gauge community/contributor sentiment earlier in the contributor journey.

One immediate suggestion I have, is that we include a short survey for new contributors once they complete their initial onboarding and begin the contribution process.

This is an easy implementation we could plug into Data Studio. We’d have a lot of good data by the end of Q4. Interested to connect with yourself and @MaryQ to design the survey questions.

3 Likes

@helmass @bradwmorris Thanks for taking the time to review our data and to provide such thoughtful questions and suggestions.

I love the idea of having 2-3 questions surveys embedded into the owl journey to get immediate feedback. I’m not sure what it would take to do that. I’m also interested in understanding how important certain engagement activities are again the experience they have of them. For example, as you mentioned, Brad, onboarding may be very important and also considered a poor experience - we better address this immediately. Or onboarding could be considered not important and a great experience - lets leave it alone, we probably have more urgent/impactful things to address. Onboarding is just an example to illustrate how it may work.

Now that we have an MVP1 (Leadership interviews and August Pulse Survey) launched, we can start planning in advance for MVP2. How does Nov (3 months out) sound? Whaddya think?

1 Like

Hey @MaryQ thanks for the work you’re doing, watched and loved the workshop. Forgot to mention that in my initial reply.

I think the way we’re looking to evolve the onboarding, is to have more workgroup specific onboarding processes. So there would be a shorter initial onboarding form for those wishing to get involved, then an additional onboarding if the contributor decides they’d like to contribute to a specific working group.

We already ask for open-ended onboarding feedback in the initial form. When we setup the workgroup forms, we can easily add in a few short feedback questions. I’d just be interested to know which questions you think would be most valuable.

To give you an example, I’ve gone through a lot of feedback responses, and organised calls with New Joiners, and the most common themes are:

  • What is the vision, mission, goals of the coop.
  • How do I contribute/confusion about contributing.

This sounds great to me. We are going to have feedback questions in the forms, so anything you can think for us to add in now, let’s do that.

I’m not sure how we’d gauge this. I’m biased, and I’d make an argument that the onboarding certainly falls on the very important side. But maybe we’d need to run this past the existing contributors.

2 Likes

Agree about the importance of onboarding. The measurement is probably more relevant to other aspects of contributor experience which impact engagement and retention. Let me know if I can be of service, in any way.

1 Like

I agree with this. Onboarding is likely one of the first and primary ways folks start their journey with us. This needs to tell the story of who we are and Vision, Mission, Goals are a great way to make that story portable.

5 Likes

Are there a few simple questions you would want added into the work group onboarding forms to start gathering data now? I can add them in. We may need to tweak the questions at a later point. But we’re getting a lot of submissions, and the numbers are growing, so I think it’s worth we collect data now. At the least to improve the onboarding process.

At the moment, we’re asking basic questions before and after the new joiner call.

I’ve reviewed and categorised 200+ new joiner responses and completed a number of calls with new joiners, and there is clearly a demand for (1) clarity on how to contribute (2) clearly defined mission, vision, (3) product roadmap.

When asked if there were any unanswered questions:

I’m going to map the onboarding process soon so we can decide which questions to be asking at what point in the onboarding journey.

1 Like

Geez. NICE WORK. [insert standing slow clap gif]

1 Like

Cool! At some point, can you send the link to the latest onboarding form? The POC and @helmass are starting to prep for the next pulse survey and think it would be helpful to refine our tone/language for overall Coop consistency. Also refining what we need to learn and how we’ll use the information going forward. I’m super interested in how we bring people into our community (thanks for all the new onboarding work Brad and team) and I’m also committed to giving attention to how we retain quality people and talent. I often frame this as promise making and promise keeping - both are essential. I would love to see more about purpose, principles (behaviors), and values in the onboarding process too. Let’s connect in the next few weeks at least.