The job function, market, structure, and role definition in the world of Business Intelligence had big changes throughout the past few years. The big change from traditional BI to self-service BI is not just about tools, but also about roles, and jobs, and organizational structure. In this post, I am sharing my views on this change with you. Like any other change, there are resistance and opportunities. I hope this post, helps you to find your way among those lines.
Introduction
Disclaimer: I have nothing against the BI team, BI developer or anything in the traditional BI space. In fact, I have been working as these roles for many lovely years. I am just pointing out a very important change here in this article.
To understand the change that happened with the appearance of self-service BI, it is good to have a look back at traditional BI first.
Traditional BI
Business Intelligence (BI) is not a new term. Companies have been hiring for their BI team and BI roles for many years in the past 20 years. Most of the major players in the market had their services, tools, and offering for traditional BI in the market for many years. In the world of Microsoft, that has been SQL Server Analysis Services (SSAS), Integration Service (SSIS), Reporting Services (SSRS) and a few other services and components.
SQL Server Analysis Services easily has a background of over 20 years of development. we have seen some major updates around 2005, 2008. Same for SQL Server Integration Services, this started with DTS in SQL Server 7 and 2000 and enhanced a lot in 2005.
Developer Tools
For traditional BI tools, there were tools to build the model, data transformations, and reports. The main tool for Microsoft tools were BIDS (Business Intelligence Development Studio) for a long time, which is now rebranded and updated as SSDT (SQL Server Data Tools)
If you have worked with SSDT or BIDS you know that the tool is an edition of Microsoft Visual Studio with great development features and capabilities.
BI Professional
The appearance of tools and services for the world of BI demanded people with that skill to work with those tools. SSDT or BIDS was a developer tool. Not a C# or application developer tool, but a new type of developer who should know how to write T-SQL scripts, work with SSIS packages, build SSAS models, and create SSRS reports in that environment. Learning such tools and services, needed a big learning curve, and that created a profession: BI professional.
There were tons of books and courses under this subject, even Microsoft had a set of five exams to achieve a certification named as MCSE: Business Intelligence.
Organizational Structure using traditional BI team
Having a team of BI professionals in most of the organizations were a necessary part of getting analysis and reporting. This ended up having a BI team in many organizations, a team of developers with the skillset to build analytical reports from existing data. This team usually received the analysis requests from all other teams in the organization.
Bottleneck, and the Disconnect from the Business Understanding
The structure above had pros and cons, one of the main disadvantages was the bottleneck of requests to the BI team considering the limited resources in the team for development. Organizations started to grow the BI teams. In many big organizations, we had big and even huge BI teams. No matter how big this team was, still, the number of requests was more than what the team could process, so there was a delay in building reports from the time of the requirement. When the report was built, sometimes it was far after the requirement, that the requirement was no longer valid.
Another problem was the disconnect between this team and other teams. BI team is built from developers who are good at their analytical skills, but they do not necessarily have business knowledge, they are not business analyst! They need to sit beside a member of other teams (Sales, directors, HR, finance) to understand their requirements and build something based on that. This, not only slowed down the process of analytics, even more, it also ended up with sometimes not desired output, because when the two parties are not in the same team, there is always a disconnect in between.
The Appearance of Analysts
By growing the number of requests, and the long delays to receive the response, which was not always 100% what the requester wanted (because of disconnect as well), each team in the organization, started keeping their own version of analytics, mainly in spreadsheet tools such as Excel. You can easily see these days analysts in each team who are really good at Excel and keep many analytical calculations and information in Excel files.
In fact, these people (green in the above diagram), been in their teams and been working with Excel even before the appearance of the traditional BI. The hope, however, was that the BI team could help them in their day to day job significantly, which did not happen. Mainly because the analytical requests of these analysts are so agile, that they can’t be easily answered through the process of requests from the BI team.
New Tools: Self-Service BI Tools
With the growing demand of analysts and tools that they can work with even without having deep developer knowledge, we see the self-service BI market to grow. Microsoft came up with the fantastic Power BI tool. Power BI is built exactly for analysts who want to build their analytical reports but don’t have deep developer knowledge.
analyst job became much easier using this tool. This gave them more power, faster and even easier.
Analytical Requests Can be Solved Internally
The era of self-service came with great enhancement in all teams in the company structure except the BI team. Now every team can build their reports and analytics using self-service BI. the number of requests to the BI team decreased significantly and only limited to “how do I connect to this data?” or “can you set up a gateway for me?”. The BI team gradually shrank as the result of this change, and the analysts in each team became the BI analysts.
The Change
The change struck many organizations, companies and BI teams. If you haven’t yet hit by this change, there might be a bunch of reasons for that. one of the reasons, for example, is that; other teams in your organizations, might not yet have figured out the power of self-service BI tools, and not familiar with tools such as Power BI. another reason can be; the structure that the BI team created over many years, was so embedded and in-use in the organization that other teams still use it that way. They do prefer to get data from the BI team because it is more reliable.
You might not yet be struck by this change, but this is the change that I believe is happening: BI teams are shrinking more and more, and analysts are growing more and more.
But what this change means? does it mean you have to retire and leave your job? Of course not! Change happens, change is part of life, don’t panic, just move with the change.
The New Jobs
If you are in a BI team, if you are a BI developer, BI architect, BI professional, BI consultant or any roles like that, this is how the change is happening in your world.
Roles are Changing, the BI team is shrinking, and it might even come to the stage that it doesn’t even exist anymore, but there is something else happening. Analysts need a better understanding of the self-service BI tools. Although these tools are easy to use at the beginning, but there are still complex scenarios that need a bit of help. I believe the BI-related roles in the world of Self-service are coming into separation as below:
Analyst
An analyst is a person with a good understanding of how business works, this person, not only knows how the business works. but also knows how to build some reports and dashboards for internal analysis and analysis of the team that he/she is working in. The analyst is part of the team that does the function of business in which he or she is analyzing the data (Not IT, or BI team), Analysts might be in the finance team, HR team, or in the sales team.
The majority of Power BI and self-service report builders are analysts. They are part of a team and want to build an analysis solution for that team (either internally, or as a contractor)
Administrator
Although analysts can take care of the reporting needs, however, they are not fully competent to do the server configuration. Things like setting up the gateway on a failover cluster, or installing, setting up and configuring the Power BI Report Server, are things that need someone with a good understanding of BI tools and IT background. This person is not a person who knows how to build a report, this person might be part of the BI team or IT team, and support analysts in all other teams to build their reports from the administration point of view.
Architect
The role of an architect in the new self-service BI world is critical and probably the most important. Without a good architecture and governance in using Power BI or other self-service BI tools, soon every organization will see the rise of hundreds of silos of data analysis. Like back in the old times that everyone had their own version of an access database! An architect is a person with a very good understanding of all components of Power BI and can advise the right way of implementing it throughout the organization. The architect might be part of the BI or IT team, or in many cases, it can be even an external consultant.
Consultant
Like many other jobs in business, there are scenarios that it is more cost-effective for stakeholders to get an external consultant to come and help with the reporting. However, Consultant in the world of self-service BI, is mainly focusing on helping with the issues, rather than implementing the entire project. There is a small job market for a mid-level consultant in the world of self-service BI. The consultant should have deep knowledge of the tool, components, and services, and help the internal analysts on an ad-hoc basis with their questions.
Instructor or Trainer
Power BI or any other self-service tool needs training. if this is end-user training, usually the analyst in the team can do that. If this is a developer training, then either a consultant or a trainer can do the job. Because consultants have a deep understanding of tools and services, they usually are good candidates for this job too. This is usually an external role, and an external consultant or trainer can work with the organization on a contract basis for it.
What is the Future of Your Job?
We talked about all of these to get to this stage and see what happens to your job. I have seen personally in the past few years that many people in the BI team or even consultants lost their job. They became redundant because their role is not needed anymore. There are a set of analysts in each team that can do their job.
This even happens for big consulting firms. With traditional BI, they supposed to get clients with multi-million dollar projects over multiple years to work on. But now that their clients can do their own analysts, their self-service BI projects are a month’s length or two, or even less. So they also end up shrinking their BI Consulting practice.
Is this the end of the BI team or consulting on BI? of course not. Here is what I think you should be doing for the next step of your job;
If you consider yourself very competent
If you see yourself very good at Power BI, then you can play the role of consultant. You can work as an independent or part of a consulting firm (that part doesn’t matter). The main thing is that people would need your skills and experience. There are many analysts that need to get help from you on an ad-hoc basis. and You can be available to them.
If you are good at analysis, but not the best
Try to become an analyst. Try to work in a team and understand their business requirements and solve their challenges. The traditional BI team might have shrunk over the past few years, but those skills are now required in each team separately, to be part of that team, but still doing the analysis job is what your next step can be.
If you see yourself good at the administration of IT, or training
If you see yourself good at administration, then you can take the Administrator role. If you, on the other hand, have a good understanding of tools and services, and a passion for teaching others, then you can take the role of trainer.
The Structure of Jobs and Roles in the Self-Service Scenario
Wrapping up on whatever I mentioned so far. This is how the analysts are spread out in each team, building reports and analytical solutions with the help of Administrator, Architect, Trainer, and Consultant (sometimes, in smaller organizations all of these four roles might be one person).
As you see in the above, it looks like there is still a BI team. However, it is not like the past. Now report builders are separated and part of each team themselves. There is no disconnect from business understanding to building the report. there is no bottleneck to building the report too. All the IT skilled roles (Consultant, Administrator, Architect, and Trainer) can be either out-sourced or provided from the resources in house, and it can expand easily.
Bottom line is that the new structure is to help businesses to achieve easier, faster and more effective data analysis, and make better decisions faster. You can be definitely part of this structure, just try to find your role. and, if you are interested to see how much the salary of a role is, have a look at the 2020 Power BI Salary Report.
If you are interested to know what things you need to know to fit into each role, or even what you need to learn for it, have a look at my training requirement questionnaire.
Check out the video here
Reza is author of more than 14 books on Microsoft Business Intelligence, most of these books are published under Power BI category. Among these are books such as Power BI DAX Simplified, Pro Power BI Architecture, Power BI from Rookie to Rock Star, Power Query books series, Row-Level Security in Power BI and etc.
He is an International Speaker in Microsoft Ignite, Microsoft Business Applications Summit, Data Insight Summit, PASS Summit, SQL Saturday and SQL user groups. And He is a Microsoft Certified Trainer.
Reza’s passion is to help you find the best data solution, he is Data enthusiast.
His articles on different aspects of technologies, especially on MS BI, can be found on his blog: https://radacad.com/blog.
Very good topic and article! To add, I think there are two key roles in the new BI world that are not being mentioned here, and they are data modeler and data stewards. Business analysts in both the old and new world still lack the key data modeling skills such as structuring data into proper star schemas, setting up data pipelines, maintaining golden datasets, etc.Maybe some of this tasks can be supported by the architect role but data architects are usually limited to one or two persons in most organizations and won’t be able to handle the data modeling workload. The need for data steward probably goes without saying. This can be same persons as the business analysts but needs to be clearly defined with data ownership in the organization.
Hi Jerry
Thanks.
The role of analyst also includes being a data modeler and data steward. In many scenarios, it won’t be one person being the only analyst in the finance team for example, there will be a few of them, and among those. one or two would take the role of taking care of the data modeling and stewardship and the rest visualization. However, because that is all still about building the analysis solution. I called them all Data Analyst. Of course, that can be split down into more detailed functions as you mentioned.
What should be (and will be anyways) avoided, is a central data modeler or data steward who is responsible for data modeling and stewardship of all analysts from all teams in the company, because that brings again the traditional BI structure and problems of it into the world of self-service.
Cheers
Reza
Don’t you need a central data modeler if you want to combine data from different teams?
Hi Jan,
It is usually rare that you need to combine data from different teams, because the requirement usually comes from the team itself, not across multiple teams. that is how it was in the traditional structure and the BI team.
However, inside each team, there is a need for a central data model. Amongst the data analysts in each team, one or two usually takes the field of modeling and data preparation and the rest doing the visualization. This might be different based on the size of the team or the number of analysts
Cheers
Reza