Death of #Agile #PMOT

OK, so maybe Agile isn’t dead yet. But I think it is certainly starting to suffer from the same disease that ultimately claimed Waterfall and other methodologies. As usual, Steve Jobs saw the issues years ago:

Content over process

Process over Content

Once the focus is on process over content, the patient starts to decline. Process is supposed to improve content by providing guidance and predictability. But if the process is followed blindly and no autonomy is given to team members to customize process, we have placed Process over Content. This happened late in the Waterfall days with additional focus on CMMI ranking and achieving other process certifications.

Most troubling is a lot of the discussion is the Agile world lately is about process and not content. We are talking about who is more Agile than whom. We are talking about absolutes as to how estimating is bad and that estimates should NEVER be done. We are talking about absolutes rather than compromises. You are ignoring Content when discussions gravitate toward absolutes. Is it a key indicator that you are no longer evaluating what needs to be accomplished or what value is. The discussion now is darn it, come hell of high water, we will be doing User Stories, with relative estimating, or not estimating at all. And it doesn’t matter what the clients think or what success is.

I know this because people are writing books about absolutes and not about how to compromise.

But Terry, we do listen to our clients you’ll say. We wouldn’t be doing our professional duty if we didn’t promote our preferred approach. We then customize after.

Fair enough. But your preferred approach is all about process and not content. Process enables content, but process is not content. Your focus is on the process….

I’ll let that sink in…

 

#1 skill for #agile and #traditional project team members – “Enough-ness Sense” #pmot #baot

I’ve had the good fortune on being on small, medium, and large traditional and agile teams. As I’ve worked on Agile teams recently, I’ve come to realize that both types of teams require one non-technical skill above all others. (In my humble opinion) Interestingly enough, this skill is required on both traditional and agile projects but in contrasting ways.

The one skill

What is the one skill I refer to? Enough-ness Sense. ūüôā

What is Enough-ness Sense? It is the skill to know whether the work has been done to the appropriate level for the circumstance.

Funny enough, (har-har) this¬†Enough-ness Sense plays contrasting roles on traditional projects versus agile projects. On traditional projects, the¬†Enough-ness Sense is used to sense when we have done enough planning and analysis and we should move to developing and other doing activities. Without this sense of¬†Enough-ness Sense, sometimes traditional projects get stuck in analysis-paralysis and an inability to move until all the intricate details are known. Professionals with¬†Enough-ness Sense know the difference between an important factor that needs to be defined and unimportant factors that can be determined at a later time. This can be seen in how detailed the analysis and design documents are. Chances are if your documents are greater than 50+ pages you don’t have enough¬†Enough-ness Sense.

On Agile projects, the effect of¬†Enough-ness Sense is the reverse. Here the¬†Enough-ness Sense is used to ensure we don’t start work until we have enough of an overall solution and design. I have seen some Agile projects jump into development too early without an end-to-end thought out design. These projects then usually suffer from large rework and scope increases later as key elements were not considered and planned for. If you start coding without a User Story Map or some general design work, you probably don’t have enough¬†Enough-ness Sense.

In both scenarios, this Enough-ness Sense is crucial. And it is not specific to one role. The Enough-ness Sense is used to determine is the plan, requirements, code, and tests are at an appropriate level? It is a critical skill for a person to be able to look at the risk inherent in the situation and determine the appropriate level of thought required.

The negative side of Enough-ness Sense

So what happens if you don’t have enough¬†Enough-ness Sense on your project team?’

On a traditional project, this will result in waste. There will be too much planning, too much analysis, and too much design. Funny enough, it usually doesn’t result in too much coding as that is the last activity being done. But in some cases of gold-plating it can result in too much code.

On an agile project, this will result in rework. There will not have been enough planning, analysis, and design to envision a cohesive solution. As such, changes occur on the project not due to changes client requirements but rather because the project team did not think something through. It is a fine line to embrace change but to not use it as an excuse to not think something through. Agile is about embracing client change,  not a lack of thoroughness on the team.

Summary

Enough-ness Sense is critical for success on both traditional and agile projects. Unfortunately I don’t know how much¬†Enough-ness Sense is enough. (har-har)

A #NoEstimates Second thought #PMOT #Agile

A while ago I wrote a blog ¬†connecting my beliefs with estimates to the Scientific Method. I still believe in the position I proposed, but I have had the unique opportunity at SDEC13 to be exposed to different opinions and alternatives. I attended a fantastic session by Chris Chapman title “The Great Canadian #NoEstimates Challenge”.

At the end of the session, I felt more informed of the validity of both sides of the discussion.

Recently I came across some reading that clarified my thinking on the subject. That reading was the distinction between Deductive and Inductive Reasoning.

Deductive Reasoning vs Inductive Reasoning

“Deductive reasoning (top-down logic) contrasts with inductive reasoning (bottom-up logic) in the following way: In deductive reasoning, a conclusion is reached reductively by applying general rules that hold over the entirety of a closed domain of discourse, narrowing the range under consideration until only the conclusion is left. In inductive reasoning, the conclusion is reached by generalizing or extrapolating from initial information (and so induction can be used even in an open domain, one where there is epistemic uncertainty.” – Wikipedia

My Belief System

I guess I believe in estimates because I believe in Deductive Reasoning. I believe that Deductive Reasoning can be applied to almost any domain. There are some exclusions in the pure Research and Development areas, but that is not the situation for most Software Development projects. I’m not saying that the Software Development domain is simple and routine like construction, but it also isn’t a pure Research and Development domain. For 99% of the projects it sits solidly in the middle. As because of that reason I believe that Deductive Reasoning should be used on Software Development projects.

It should be noted I don’t believe in Deductive Reasoning only for estimating. I also believe Deductive Reasoning should also be used when designing the solution and architecture and scheduling the project. ¬†If some Deductive Reasoning is not applied for the solution architecture, ¬†we do not have “general rules that hold over the entirety of a closed domain of discourse”.¬†Put in a different way, we run a greater risk of having an inconsistent solution and architecture. If some Deductive Reasoning is not applied for the creation of the project schedule, we will not provide the external visibility that will be required to co-ordinate with other people. (Yes – that includes budgeting)

Story Slicing

The latest trend in the No Estimates land is not to estimate stories, but just to count stories. This is still estimating with a different formal system. So I agree 100% that this is a viable alternative. It is a great example of having “”general rules that hold over the entirety of a closed domain of discourse”

Summary

I guess the question is whether the negative aspects of estimating (and there are quite a few), are offset by the benefits of Deductive Reasoning.

For my money, I believe they are as predictability and consistency provided by Deductive Reasoning is very valuable to clients and external teams. Although no estimates can help the development team perform better, we must be careful not to sub-optimize and not provide the predictability and consistency required by the entire team.

When we are asking whether we estimate we must remember that the value is defined by the entire team, especially the client.

#Agile #Sketch – A process for #Agile solutioning

I was discussing how I wanted to approach an Agile project with my team the other day and came across a pattern that I typically find comfort and clarity in. I like to Sketch. Not the drawing type of sketching – but the project type of sketching.

I like to draw and sketch everything at a high level and then fill in the sketch as I go along on a project. I find these sketches extremely valuable. I also use various types of sketches:

  • Schedule – high level project plans
  • Architecture – high level design
  • Strategy – high level objectives and scope
  • Estimate – high level effort plans

I’m sure there are others.

The concept is two-fold.

1) Create a sketch so that it provides internal and external clarity in regards to what the team is doing and creates a shared vision.

2) Allows the team to think through the sketch. Sometimes oversights, inconsistencies, and errors are only found when you dive down into the details.

The situation

In this situation on the project, I was proposing we create a sketch of the test solution we were going to create. I felt this would be valuable as a method to confirm all the business scenarios that we felt this project needed to address. We would not define all the individual details about the business scenarios, but confirm at a high level, the X business scenarios we would have to ensure we do address and define in more detail in the future.

A colleague of mine asked whether this was a waterfall process.

It caused me to pause and think. Was this a remnant of waterfall processes I had used in the past?

In retrospective, my preference for these sketches seems to be to be very Agile. I am doing what I feel to be the minimum amount of work required. Although these sketches do imply some serialization, I don’t think serialization automatically implies waterfall. I believe excessive and extremely detailed serialization implies waterfall. Some limited serialization to create shared vision and mitigate risk is very Agile.

I viewed what I was proposing to be very similar to a User Story Map. A small amount of time to confirm vision and gain agreement with the team and clients on what the requirements are.

I also saw an analogy back to my video game days where you struggled to get to the next checkpoint. Once you got there you wouldn’t lose the progress you had made. I felt these sketch checkpoints were extremely valuable to spend a small amount of time to confirm the vision and think through the solution. If we had a disagreement we could reflect back to the sketch as we confirmed we had agreement at that point in time. In this way it very much is a risk mitigation strategy.

Flow

I think my comfort with the process of Sketching can be related to my concern on whether a project can be done fully in Flow without some up front planning. I believe that Flow can and should be used 100% on repeatable and stable processes, but as we know that sometimes projects can be called neither stable or repeatable.

Rather than having scope, design, or architecture organically grown from a collection of User Stories, I do think that some planning should be done to confirm visions and think the solution through. Whether you call these constructs sketches, User Story Maps, Plans or widgets is  not important.

There are two critical aspects

1) Do some level of upfront planning/sketching/mapping

2) Do as little planning/sketching/mapping  as possible

Summary

We need to be very careful in the Agile community that we find the appropriate solutions that mix Agile with Traditional approaches.

How to strategically prioritize user stories #Agile

One of the reasons I love to read books on Agile is because of the wealth of great ideas out there. It seems like every book I read allows me to discover one or two news ideas. The book I am currently reading is Ken Collier’s “Agile Analytics”. While reading this book I was introduced to the idea of prioritizing project requirements using the Purpose Alignment Model. (The model was proposed by Pixton et al in the 2009 book “Stand Back and Deliver”) Although the principle of getting the clients to prioritize your project requirements doesn’t change, the method does provide a structure and context that can inject new insights into the prioritization process.

Purpose Alignment Model

The Purpose Alignment Model proposes that you separate your project requirements into 4 categories based upon their strategic importance to the business. The categories proposed are:

Business Differentiating Activities – These are project requirements that support activities that are both marketing-differentiating AND mission-critical. These are strategic activities that will add to the growth of the company. These activities are usually aligned with new opportunities. The company risks the loss of growth if these activities are not done.

Business Parity Activities – These are project requirements that support activities that are mission-critical but not market differentiating. These are operational activities that will allow the company to maintain their current position in the market. The company risks losing their position in the market if these activities are not done.

Partnering Activities РThese are project requirements that support activities that are not mission-critical but are market-differentiating.  These are typically partnering opportunities in the market to create new opportunity and growth. Unlike the Business Differentiating activities, this activity is looking to find a partner to share the cost of this market-differentiating opportunity.

Who Cares Activities – These are the project requirements that support activities that are neither market-differentiating or mission critical.

Conclusion

I’m planning to use this structure on the next opportunity to prioritize User Stories in the backlog. I believe it will allow for interesting insights into where the development team is spending their efforts in each iteration. I also think the clients would find the information interesting. Especially if we are spending more than 50% of our time consistently on developing functionality to support “Who Cares Activities”.

 

#Agile Enterprise Data Model – #holistic and #iterative

I’ve recently been working on a Data Warehouse project that I am trying to do in an Agile way. Unlike Agile Software Development, Agile Data projects have some complexities that cause you to adapt the common Agile methods and practices. I am a big fan of User Story Mapping and creating a User Story Map to guide your development of an application and to create and prioritize your backlog. But I have felt that a Data Model tends to not allow stories to be sliced as thinly as application functionality. In many cases a group of tables must be taken together to make any sense at all. Unlike application components that can be segmented easier, it sometimes is quite difficult to segment a data model into parts that can be worked on independently.

I have been involved in other Data Modelling projects where intense analysis was done to try to understand all the intricacies of the data before the application development started. This analysis sometimes lasted 3 months, 6 months, or even years. It goes without saying that this is not an efficient was to organize a project.

If you try to segment application user stories, individual reports, and data requirements as stories they end up being a bit too fine-grained from database point of view. Creating a data model from a single user story or report can lead to a data model that has to undergo a great amount of re-factoring as new user stories, reports, and data requirements are considered. Frequently a new user story that enhances application functionality requires an entirely different data model. ūüė¶

Then I started to think that this can happen to most Agile projects. They slice the User Stories thin from an application point of view and that works as the application components can be somewhat independent, but what happens when you try to line-up the data components?

I am sure this impedance has led to the more drastic database re-factoring that has caused it to have a bad reputation and caused people to write books on the subject.

Epiphany

Then it hit me. Although the User Story Mapping exercise does allow for an end to end view of the application that can then be iterated on, it does not create an end-to-end view of the Data Model or the Data Domain. In fact, some critical components of the Data Model may not be encountered until much later. This can cause significant rework based upon the design that has been done earlier.

So what could be the solution?

Agile Enterprise Data Model

An Agile Enterprise Data Model is a thin data model of the Data Domain that can be done as a two-week technical spike. This Agile Enterprise Data Model provides a User Story Map for the data. It provides a Data Model with placeholders for discussion and further refinement. The concept it not to do detailed Data Modelling but to understand all the data components at a high level that will need to be considered by the project. Used in conjunction with a User Story Map, it can be a valuable tool to allow the application and data model to be worked on in true iterations.

Typically an Agile Enterprise Data Model is a conceptual data model with 20-30 entities. There are minimal attributes and discussion and understanding revolves more around entities and relationships than anything. The Agile Enterprise Data Model can then be refined to create the application database, Operational Data Store, and Data Warehouse.

Summary

I have used the Agile Enterprise Data Model with very good success. It also works best in a Data Domain you are unfamiliar with as it allows you to understand a Data Domain holistically initially and then deliver in iterations. The best of both worlds.

How an #Agile Data Warehouse leveraged an Innovation Game #agile2012 #sdec2012

On my latest project I have been doing a lot of research and learning on how to conduct a Data Warehouse project in an Agile manner. Actually I’m really honoured to be able to combine my two passions in my career: Data Modeling and Agile.

This investigation has led me to what I feel is a unique approach to an Agile Data Warehouse project. Many of the Agile Data Warehouse projects I hear of and read about typically seem to follow a somewhat standard process. This process usually involves building a Data Warehouse in increments by working with departments within the corporation to understand their data requirements. (in many situations, this is a very detailed analysis of their data requirements) In this way, the Data Warehouse is built up over time until an Enterprise Data Warehouse is created. While this is a far cry better that trying to analyze the data requirements for the entire corporation, I do believe there are some limitations with this approach.

Limitations

1) If the Data Modelers are not experts in the existing data domain, this process may involve significant rework as data modeled for one department may need to be modified as new information is learned and existing data structures need to be integrated with data structures designed in subsequent releases. This risk can be significantly reduced if the data modelers are experts in the data domain, but this can be a real risk for consultants that are not experts.

2) We are still not implementing a Data Warehouse in an iterative fashion, only incrementally. While this is a step in the right direction, it is still falling short of being able to implement a Data Warehouse iteratively and get even quicker feedback. Since we are also essentially time-boxing implementing data requirements on a department by department basis, we are also only working on the most important data requirements on a departmental basis. We are not working on the prioritized data requirements for the entire corporation. If every department receives the same time allocation, we may work on some data requirements while other more important ones are left in the backlog. And finally, the data requirement for each department can get quite detailed. In many ways, we are still doing Big Design Up Front – in chunks.

Our Approach

My background in Data Modeling has always been aligned with Bill Inmon’s thoughts rather than Ralph Kimball’s. So to no surprise I felt I needed a good understanding of the data domain for the Enterprise before I could start to model the data. To address this I proposed we create and Agile Enterprise Data Model. An Agile Enterprise Data Model is an Enterprise Data Model that takes 6 weeks instead of 6 months to create. (or 6 years) Its purpose is to validate the entities, relationships, and primary attributes. It is not intended to drive down into every single attribute and ensure alignment. In my experience, this excessive detail has derailed other Enterprise Data Model projects. (as consensus cannot be reached or the data is always evolving) But in creating an Agile Enterprise Data Model, we now understand enough of the enterprise’s data so that even though we may only be modeling one department’s data, we know what integrations and relationships are required in the future.

I felt this addressed the first limitation.

The second limitation was much harder. I felt it was much harder to engage clients in small slices or iterations of their data requirements in a light-weight fashion that wouldn’t require reviewing their current data requirements in detail. How could we get a picture of the data requirements for the corporation at a high level? Many of the clients were concerned that if the project operated in iterations they would not get all of their requirements satisfied. In addition, our current project had a list of existing reports that the clients would like to be migrated to the new environment. How could we engage our clients and work with them to determine what their corporation’s data requirements were in an iterative light-weight manner? How could we provide visibility of what the entire corporation ¬†data requirement were?

I turned to Innovation Games.

I am a fan of Innovation Games and any light weight process that can help to visualize data and engage the clients better. I was searching for a way to use a variant of the ‘Prune the Product Tree’ game to use. But I didn’t feel that just placing data requirements on the Product Tree would truly help the engagement of the client and provide the visibility required. What we really needed was a variant of the Product Tree that helped the clients ‘see’ their data requirements to confirm that we got it right and that showed the requirements for the entire corporation on one wall.

We ended up merging a ‘Prune the Product Tree’ game with a ‘Spider Web’ game with some unique additions.

Here is what are doing:

1) We are seeding the visualizations with 30-40 enterprise level reports that are used internally and externally.

2) We have asked for the top 20 reports from each department to further seed the visualizations. This will help to populate the visualization and I believe help the clients to generate other data requirements that we are missing.

3) We are capturing data requirements or reports in the simplest mode possible. We decided to leverage the Goal, Question, Metric method proposed by Victor Basili. Our data requirement stories will specify the following:

  • Noun : The object of primary interest
  • Question : The main inquiry about the noun
  • Reason : What is the business reason for the question
  • Frequency : How often do I need the answer?

An example is provided below:

Noun Question Reason Frequency
Claims Amount > $1,000 To generate audits Monthly

We will capture these additional data requirement stories in Silent Brainstorming sessions with users. This Silent Brainstorming will occur after we present the visualizations and review the data requirements that have seeded the visualization.

The final piece of the puzzle

Even with those first three pieces, it was still unclear how we could visualize the data and reporting requirements and gain engagement with the clients. Then during an Innovation Games course that was educating us on how to customize and create our own game, it struck me. The Product Tree is the wrong metaphor for our data requirements. We need something that visualizes the data requirements themselves. We iterated through a spider web metaphor, to a 3 dimensional axis metaphor, until we ended up on a hexagon. I’d recommend you find the shape that best fits your data domain. The hex and its six dimensions fit our data domain¬†nicely.

The hex diagram below is our customized Innovation Game we will use to seed data requirements and generate new requirements.

The Concept

Usually for all data requirements or reports there is a person object at the centre. This is no exception. We will have a data hex for each type of person that the corporation needs to know about. The six accesses were chosen by understanding the data domain and by reviewing existing reports that are used by the client. The existing data requirements and reports will be translated into the data requirement stories and placed on the hex. Sessions will be facilitated to add additional stories onto the data hexes.

Once all of the stories have been placed on the data hexes we will prioritize the stories by moving the data requirement stories based on priority. Data Requirement stories that are high priority will be moved near the centre. Stories of less priority will be moved to the outer rings.

The Rationale

Using this metaphor we are able to do the following:

1) Visualize the hundreds of data requirements and reports of the corporation at once and prioritize them.

2) Ensure we are working on the true data requirements across the entire corporation.

3) Work in a true iterative fashion in how we deliver data requirements and ultimately build the Data Warehouse iteratively.

4) Use a light weight method that limits detailed design up front

Summary

If you want to hear how this process worked, the results will be presented at Agile 2012 in Dallas and at SDEC2012 in Winnipeg! I’ll also have a subsequent blog post that publishes the results. So far the feedback has been encouraging and we will be expanding the use of the process over the next few weeks.