June 10th, 2013 by Daniel Woods

Recently, I worked on an ETL/BI project for a non-profit client where task changes inside of each iteration consumed me. I became so focused on tasks that were creeping into each iteration that we had to push demonstrations into future iterations. The fact is, even though iterations were scheduled with features to be delivered I allowed additional tasks to creep into the feature sets we had planned to deliver. The client was very happy in the end (even giving us a testimonial), but the kind of scope creep I was struggling with haunted me.   Isn’t approaching BI in an agile fashion supposed to rid our projects of scope creep?

Creeper's Gonna Creep

Creeper’s Gonna Creep

Could I improve my approach to agile business intelligence to secure the delivery of value in each iteration?  After some reflection, I came to see this as a special kind of scope creep that can happen with tasks in Agile and I had to think through solving and preventing these kinds of problems specifically in analytics projects. Scope creep is defined by Wikipedia as uncontrolled changes or continuous growth in a projects scope. Generally speaking after a project’s

initial scope, the project owners will continue to pile on existing tasks without adjusting resources. This leads to projects failing to meet deadlines and unhappy customers. It is traditionally found in software development but it applies equally to business intelligence or analytics projects with database ETL development.

Clear User Stories

Clear user stories are important because they will provide what the role would like to do and more importantly why they want to do it. They why is very important because it tells you the specific need or business value the user will get from the completion of the story. I recently read a great description of what a user story was from Mountain Goat Software. They stated: User stories are short, simple description of a feature told from the perspective of the person who desires the new capability, usually a user or customer of the system. They typically follow a simple template:

As a <type of user>, I want <some goal> so that <some reason>.

In a good user story, you can easily discern the business value while also encouraging some feedback and collaboration.  A bad user story will not clearly identify a business value; contain multiple user stories inside of the single story; provide too much detail and not allowing for the developer to use their creativity; and finally not clearly identifying the user, using either  too vague or too specific a role. An example of a bad user story would be something like “As a NOC user, I want a report which highlights troubled calls in red, possibly issues in yellow, etc. to address quality control”. A better way to write this example would be “As a NOC user, I want to identify troubled calls to address quality control over particular trunks lines”.

Move Scope Creep to New Stories

Eventually in every project, the owner of the project will ask, “Can we add this data point or this report in this iteration or release?”  In order to make the client happy, I will say sure no problem and attempt to finish the new task all the while trying not to miss the deadline on other tasks. Sometimes this works and you are able to get it done, other times you end up working overtime to produce. Instead of accepting the new work into the existing iteration, a better response is take the requirement, phrase it as a clear user story, and set it on the stack for review at the start of the next iteration.  If it is high priority, you can always bring it in to the current sprint if there is extra capacity.  If there isn’t, it would be a mistake to add the story into the current sprint plan.   Not completing what you promised at the beginning of the sprint harms trust, and there are good reasons to fight to keep your stakeholder’s trust including the ability to self-manage. Saying that, sometimes clients have business critical needs that must be addressed and you must figure a way to fit them into the sprint. It is very important though to distinguish those business critical needs and small one offs that should be moved to other sprints because all clients will always try to get a little bit more into each iteration. This can cause you to fail to meet deadlines on more critical items while also eating at that trust you are trying to build.

Clear Resolution

At the end of the iteration, a set of user stories should be completed and presented to the client. You present the result of the work to the client so they can grab onto something. Sometimes it can be difficult to explain what you have working on for them, but if you can actually show them, trust will be build and they walk away knowing they have gained some value from the project.

The developer will demonstrate the result of the story. In BI, maybe a report is generated or chart is designed. In a data project, maybe a BI tool can be used to visualize the new data points that were created. The client must either approve or deny that story. Avoid comments such as text changes or style changes. These types of changes will inevitably lead to back and forth with a client prolonging the deployment. If a story fails to meet the clients need, it goes back to the drawing board in the next iteration.

Had we used the approach described above during the project for the non-profit, we could have completed many more items. This is not to say that we did not provide value to the customer. They are now able to report on data points they couldn’t in the past. This is to say, we could have provided more value over the short engagement.



  1. Cohn, Mike (2004, October 8). Advantages of User Stories for Requirements. Retrieved from http://www.mountaingoatsoftware.com/articles/advantages-of-user-stories-for-requirements
  2. Erickson, Aaron (2013, May 13). Big Data, Powerful Insights. Retrieved from http://www.youtube.com/watch?v=53XKYa1vT90
  3. Collier, Ken (2012, February 24). Organizational Friction Impedes Agility. Retrieved from http://theagilist.com/2012/02/24/organizational-friction-impedes-agility/