Four Information Technology Roles over time #agile #innovation

As I was evaluating the current role of Information Technology in our Industry I looked back and it seemed to be that there have been four distinct roles that Information Technology has played with business over the years

    1) Recording and Reporting 

This role focused mainly on Risk Mitigation. In the early years of computing, there was great value to be gleaned by entering data in a consistent way and minimizing errors. There was also great value realized by then having all the data in one location that could then be reported on to provide key indicators for the health of the business.

Information Technology implemented systems to record the current state or what ‘IS’.

    2) Automation

Soon after Information Technology had implemented solutions to help in Recording and Reporting, focus turned to larger problems like:

1) Automation of workflows/larger processes

2) Scalability

3) Efficiency

4) Availability

I often refer to this as the golden age of computing as we turned the power of the computer to problems that were much easier to scale and make more efficient using a computer rather than adding more people. The benefits were also realized in many workflow projects where Information Technology assisted people in guiding what they were supposed to do in the workflow process. In this phase, the increased computing power was also brought to the forefront to provide processing power that was only dreamed about 5 years ago.

In this phase, Information Technology implemented systems to take business beyond ‘IS’ and towards what business ‘COULD’ do with automation.

    3) Agile

With the Agile Manifesto, Information Technology ushered in the new phase of Agile projects. Now we were focusing not only on the outcomes of the projects, but also HOW the projects were being executed to maximize value. This was done through reducing Cycle Time, Maximizing Client Engagement, and through Visual Project Management.

In this phase, Information Technology implemented project processes that showed how projects should be run.

These first three phases focused on improvement, refinement, assisting the business to get better. We are now on the cusp of the fourth phase where Information Technology helps the business grow.

    4) Innovation

There is much talk about Innovation and how people must innovate, but it remains somewhat elusive. What is clear is that Information Technology has a role to play in helping clients Innovate.

Information Technology is uniquely position to leverage visual tools like Innovation Games, Empathy Maps, Business Model Canvas to provide business the brutal visibility that Information Technology provided previously about the projects in the Agile phase. What is clear is that the trend that Agile started toward visual requirements must continue. Specifications have improved from Specification documents to Use Cases to User Stories to A Business Model Canvas. Each one more visual that the prior.

What is also clear is that asking people what they want and documenting the response in words is a very risky endeavor.

For the first time, Business Models can be captured on one sheet and what-if-analysis can be done like in the first spreadsheet programs to evaluate options and alternatives.

In this phase, Information Technology is analyzing and recommending business processes that show how business can grow.

We won’t just be helping companies to get better, but to be different.

Hang on… This should be fun…

How an #Agile Data Warehouse leveraged an #InnovationGame – Iteration 2!

A few weeks ago I authored a post that explained how were leveraging an Innovation game on my current Agile Data Warehouse project. You can find the original post here.

Iteration 2

One of the aspects I love about Agile is the freedom it allows you to be bold as it acknowledges it is impossible to get it perfect off the bat. I find that this encourages people to take a chance and try new things. If it doesn’t work? No problem, we will just adjust and get better as we progress.

My initial blog proposed using a visual Innovation Game with Visual Report Boards to allow for brutal transparency and the management of the data requirements for an Agile Data Warehouse project. Each Visual Report Board had an object at the centre with 6 dimensions around it that illustrated how the object could be reported on. (I affectionately refer to these boards or diagrams as Hexes) Our Objects were the people who the corporation were interested in. It turns out that this method has been successful. It has allowed us to create a visual backlog of data requirements and have the clients prioritize the work and guide the work according to the overall business priorities.

Why am I writing this blog then?

Well it turned out that once we started placing the data requirements or reporting stories on the Visual Report Boards, it became apparent that our objects and dimensions of the diagrams required tweaking.

Initially I had placed a Person object at the centre of the diagram. The dimensions where then aspects of how we need to report on that person. Although this object and dimensions may work for other projects, it did not work for our project. (Actually I think that most Data Warehouse projects would probably make the same change we did, but I’ll let you decide that for yourselves)

Our experience was that most of the reports or data requirements were very clustered on the Visual Report Boards and the diagrams did not allow for the visual communication of what the report was or what data was required. I was starting to worry that this process might not provide the brutal transparency that allowed for the efficient creation of a Data Warehouse in an Agile way.

Just the Facts

After reviewing the requirements, it became apparent I had the wrong objects at the centre! Rather than people at the centre, the objects at the centre should be transactional data. The centre objects needed to be the actual data that was summed, aggregated, filtered, sliced, and diced. Once I made this change, the value of the Visual Report Boards increased exponentially. Now they communicated the content and purpose of the data requirements.

The real indication that I was on the right track is that the Revenue and Expenses Hexes I now had were also the first two Fact tables that were needed in the Data Warehouse! This method of visualization and analysis was aligned 100% with the Data Warehouse design process. Of course the Hexes were Fact table. This made perfect sense. I imagine more Hexes will be needed in the future as we discover the need for more Fact tables.

In addition, we created one more Master Data Hex as some reports and data requirements are not related to transactions.


I am convinced the use of these Visual Report  Boards and the related use of an Innovation Game enable the creation of a Data Warehouse in an Agile manner. We are executing in Iterations and not increments and the clients are thrilled with the control, visibility, and value they get every 2 weeks. I’ll post another blog once we have created enough of the Fact tables to provide more lessons learned.

How an #Agile Data Warehouse leveraged an Innovation Game #agile2012 #sdec2012

On my latest project I have been doing a lot of research and learning on how to conduct a Data Warehouse project in an Agile manner. Actually I’m really honoured to be able to combine my two passions in my career: Data Modeling and Agile.

This investigation has led me to what I feel is a unique approach to an Agile Data Warehouse project. Many of the Agile Data Warehouse projects I hear of and read about typically seem to follow a somewhat standard process. This process usually involves building a Data Warehouse in increments by working with departments within the corporation to understand their data requirements. (in many situations, this is a very detailed analysis of their data requirements) In this way, the Data Warehouse is built up over time until an Enterprise Data Warehouse is created. While this is a far cry better that trying to analyze the data requirements for the entire corporation, I do believe there are some limitations with this approach.


1) If the Data Modelers are not experts in the existing data domain, this process may involve significant rework as data modeled for one department may need to be modified as new information is learned and existing data structures need to be integrated with data structures designed in subsequent releases. This risk can be significantly reduced if the data modelers are experts in the data domain, but this can be a real risk for consultants that are not experts.

2) We are still not implementing a Data Warehouse in an iterative fashion, only incrementally. While this is a step in the right direction, it is still falling short of being able to implement a Data Warehouse iteratively and get even quicker feedback. Since we are also essentially time-boxing implementing data requirements on a department by department basis, we are also only working on the most important data requirements on a departmental basis. We are not working on the prioritized data requirements for the entire corporation. If every department receives the same time allocation, we may work on some data requirements while other more important ones are left in the backlog. And finally, the data requirement for each department can get quite detailed. In many ways, we are still doing Big Design Up Front – in chunks.

Our Approach

My background in Data Modeling has always been aligned with Bill Inmon’s thoughts rather than Ralph Kimball’s. So to no surprise I felt I needed a good understanding of the data domain for the Enterprise before I could start to model the data. To address this I proposed we create and Agile Enterprise Data Model. An Agile Enterprise Data Model is an Enterprise Data Model that takes 6 weeks instead of 6 months to create. (or 6 years) Its purpose is to validate the entities, relationships, and primary attributes. It is not intended to drive down into every single attribute and ensure alignment. In my experience, this excessive detail has derailed other Enterprise Data Model projects. (as consensus cannot be reached or the data is always evolving) But in creating an Agile Enterprise Data Model, we now understand enough of the enterprise’s data so that even though we may only be modeling one department’s data, we know what integrations and relationships are required in the future.

I felt this addressed the first limitation.

The second limitation was much harder. I felt it was much harder to engage clients in small slices or iterations of their data requirements in a light-weight fashion that wouldn’t require reviewing their current data requirements in detail. How could we get a picture of the data requirements for the corporation at a high level? Many of the clients were concerned that if the project operated in iterations they would not get all of their requirements satisfied. In addition, our current project had a list of existing reports that the clients would like to be migrated to the new environment. How could we engage our clients and work with them to determine what their corporation’s data requirements were in an iterative light-weight manner? How could we provide visibility of what the entire corporation  data requirement were?

I turned to Innovation Games.

I am a fan of Innovation Games and any light weight process that can help to visualize data and engage the clients better. I was searching for a way to use a variant of the ‘Prune the Product Tree’ game to use. But I didn’t feel that just placing data requirements on the Product Tree would truly help the engagement of the client and provide the visibility required. What we really needed was a variant of the Product Tree that helped the clients ‘see’ their data requirements to confirm that we got it right and that showed the requirements for the entire corporation on one wall.

We ended up merging a ‘Prune the Product Tree’ game with a ‘Spider Web’ game with some unique additions.

Here is what are doing:

1) We are seeding the visualizations with 30-40 enterprise level reports that are used internally and externally.

2) We have asked for the top 20 reports from each department to further seed the visualizations. This will help to populate the visualization and I believe help the clients to generate other data requirements that we are missing.

3) We are capturing data requirements or reports in the simplest mode possible. We decided to leverage the Goal, Question, Metric method proposed by Victor Basili. Our data requirement stories will specify the following:

  • Noun : The object of primary interest
  • Question : The main inquiry about the noun
  • Reason : What is the business reason for the question
  • Frequency : How often do I need the answer?

An example is provided below:

Noun Question Reason Frequency
Claims Amount > $1,000 To generate audits Monthly

We will capture these additional data requirement stories in Silent Brainstorming sessions with users. This Silent Brainstorming will occur after we present the visualizations and review the data requirements that have seeded the visualization.

The final piece of the puzzle

Even with those first three pieces, it was still unclear how we could visualize the data and reporting requirements and gain engagement with the clients. Then during an Innovation Games course that was educating us on how to customize and create our own game, it struck me. The Product Tree is the wrong metaphor for our data requirements. We need something that visualizes the data requirements themselves. We iterated through a spider web metaphor, to a 3 dimensional axis metaphor, until we ended up on a hexagon. I’d recommend you find the shape that best fits your data domain. The hex and its six dimensions fit our data domain nicely.

The hex diagram below is our customized Innovation Game we will use to seed data requirements and generate new requirements.

The Concept

Usually for all data requirements or reports there is a person object at the centre. This is no exception. We will have a data hex for each type of person that the corporation needs to know about. The six accesses were chosen by understanding the data domain and by reviewing existing reports that are used by the client. The existing data requirements and reports will be translated into the data requirement stories and placed on the hex. Sessions will be facilitated to add additional stories onto the data hexes.

Once all of the stories have been placed on the data hexes we will prioritize the stories by moving the data requirement stories based on priority. Data Requirement stories that are high priority will be moved near the centre. Stories of less priority will be moved to the outer rings.

The Rationale

Using this metaphor we are able to do the following:

1) Visualize the hundreds of data requirements and reports of the corporation at once and prioritize them.

2) Ensure we are working on the true data requirements across the entire corporation.

3) Work in a true iterative fashion in how we deliver data requirements and ultimately build the Data Warehouse iteratively.

4) Use a light weight method that limits detailed design up front


If you want to hear how this process worked, the results will be presented at Agile 2012 in Dallas and at SDEC2012 in Winnipeg! I’ll also have a subsequent blog post that publishes the results. So far the feedback has been encouraging and we will be expanding the use of the process over the next few weeks.