To warn you, if the last blog post made you uncomfortable, this one is going to make you downright cranky. Last time I talked about pulling back the curtain on your enterprise data warehouse (EDW) and business intelligence (BI) solutions in order for people in your organization to be able to see what data is available to them and how everything is calculated. This improves the quality of your data because you have more eyes validating definitions, it improves efficiency because people don’t spend time spinning their wheels trying to find data or having redundant reports developed, and it improves customer satisfaction because people have greater understanding and trust of the data. This time we’re going to take the idea of pulling back the curtain a few steps further to actually allowing people behind the curtain!
So often as developers of EDW/BI solutions we work with the data so much that we start to think it’s ours. We’ve spent hours working with business sponsors to build the patient registry just so. We’ve spent more hours poring over measure specifications to get the OR block utilization calculation working perfectly at all levels of summarization. We want to protect the investment we’ve made in this asset and we don’t want anyone coming in and even questioning our work much less being allowed to change it and potentially mess it up. But the data doesn’t belong to us. As data architects and BI developers, our job is to enable others to utilize data to do their jobs in the most efficient and effective way possible.
One way to significantly increase the efficiency and effectiveness of the EDW/BI team, especially in a climate disinclined to adding new FTEs, is to allow others outside of the team to share the development load. After you revive from fainting at the thought, let me explain how that can work.
Let’s say you take a development approach in which data elements such as “Most Recent A1C Lab Value” are implemented in a single set of code (sql statement) that only returns that data element. The technical owner who has been identified for that data element works as an analyst in the diabetes service line, and works closely with a clinician in defining measures pertinent to diabetes care. Between the analyst and the clinician, they know the subject matter best, and certainly better than an individual on the EDW/BI team who isn’t able to focus on a single condition. They know the specific definition of diabetes that is accepted in the organization, they know which lab codes relate to A1C, and they know the valid range of A1C lab results in order to spot invalid values.
Given all of the subject matter expertise, why not have the analyst go a step further than what you would traditionally see in an EDW/BI environment and actually maintain the code for “Most Recent A1C Lab Value” as well as develop code for new data elements relevant to diabetes? Maybe you think analysts won’t learn sql, however in my experience even the staunchest holdouts come around once they see the opportunity to really own data elements that relate to their area of expertise as opposed to playing a game of telephone with the EDW/BI staff to get the data elements built.
Perhaps you think it’s dangerous to allow non-data-architects to develop anything “in production”, and you’re right to an extent, however we often inflate that risk far beyond what it really is. Still, what I’m suggesting isn’t the Wild West. A data architect should always review the code to make sure it will run and run efficiently before it is implemented, and analysts writing code for new data elements should (and will!) use previously developed data elements (via the metadata tool) as a starting point. But that final check by the data architect (on code that already came from a template) is much more efficient after the analyst has already been able to research, run, tweak, and validate the code and the resulting data ahead of time.
EDW/BI teams often are seen as a bottleneck, and requiring absolutely all development to come from a limited number of people plays into that significantly. With proper (but the minimum necessary) controls in place, we have seen this EDW/BI “crowdsourcing” model work well. It lowers the burden on overtaxed EDW/BI teams, and in the end produces a better product since those closest to the subject matter are the ones creating the content.
Next time, in our final installment in this series, we’ll discuss what regular check-ups can do for your EDW/BI solution.
