Reclaimed Time For Data Personnel

Evidence and data. Deploying Monte Carlo allowed interviewees’ organizations to detect data quality issues earlier via health monitoring — an efficiency that resulted in productivity improvements for personnel across their data estates. Data analysts used the platform’s AI-driven anomaly detection and recommendations for early issue detection so data engineers could remediate issues before they became larger and less localized. Organizations’ data analysts and data consumers were less prone to data and AI downtime, which allowed them to remain on task, contributed to faster decision-making, and saved time.

• The AVP of product strategy at the stock exchange noted that their data quality issues often originated at the beginning of a month and were not detected until later, which put a major burden on the organization’s data engineers to remediate. The interviewee said: “[Our engineers] would have to go to the trading system teams and have them rerun trade data transformations. It is an involved process, so that was a big pain point. There's a lot of manual labor to do all that.” The same interviewee also emphasized the benefit of detecting data quality issues earlier with Monte Carlo, as fewer personnel resources were required for remediation and resolution time was cut to a fraction. The interviewee explained: “We’ve caught issues that would [previously] take the team three to five days to resolve. Now we catch them on day one and resolve them pretty much in place.” Since deploying Monte Carlo, their organization frequently catches and resolves errors on the same day they occur, which previously required tens of hours of data engineer or analyst time, avoiding an estimated 90% of the previous remediation effort.

• The same interviewee highlighted additional observability efficiencies for pipeline development, noting: “We spent a lot of time making large, backlog-like changes that we pushed out once every six months versus being much more agile and getting feedback sooner. So our development processes were just much longer, which was not ideal from both an analyst point of view and from an engineering point of view.”

• The senior manager of data analytics and architecture at a food processing organization explained that their data analysts and data engineers could focus more on their core responsibilities given the features within Monte Carlo. They estimated a 50% to 60% efficiency gain for their organization’s data personnel since their analysts could set data quality rules in Monte Carlo, and engineers now had data lineage tracking capabilities to approach data quality issue resolutions from a more informed position.

• Interviewees at the airline noted that since onboarding Monte Carlo, data quality issue escalations that required the effort of data personnel dropped dramatically. Employees catch issues sooner with fewer resources required to resolve them. Data personnel now spend an estimated 10% to 15% of their time monitoring and acting upon data quality issues, whereas the interviewees estimated this effort to be significantly more before Monte Carlo.

• Interviewees also highlighted that although data personnel reaped a significant share of the efficiency benefits of Monte Carlo, data and AI downtime impacted other personnel as well. The AVP of product strategy at the stock exchange noted that finance and accounting personnel were frequently subject to delays in their monthly closes resulting from data reconciliation efforts at the end of the month.

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Six data quality issues per month require manual intervention.

• Eight data engineers are typically responsible for data quality issue remediation. Each data engineer spends an average of 10 hours on remediation, normalized across all issue severities.

• Data quality issues are caught sooner and resolved faster with Monte Carlo, improving resolution time (and reducing personnel effort) by 80% to 90% from Year 1 to Year 3.

• The fully burdened hourly rate for a data engineer is between $66 and $68 from Year 1 to Year 3.

• There is a 75% productivity recapture for data engineers, as not all time reclaimed will be repurposed toward value-added work.

• Twelve data analysts are responsible for detecting quality issues along with their other responsibilities.

• Each downtime incident costs a data analyst 3 hours in downtime.

• The fully burdened hourly rate for a data analyst is between $39 and $41 from Year 1 to Year 3.

• There is a 50% productivity recapture for data analysts, as not all time reclaimed will be repurposed toward value-added work.

Risks. This benefit will vary among organizations based on:

• The baseline number of data quality issues within an organization relative to the potential for improvement with Monte Carlo.

• An organization’s data estate relative to the severity and traceability of undetected issues.

• The skill and capacity of an organization’s data engineers and data analysts.

Results. To account for these variances, Forrester adjusted this benefit downward by 10%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $647,000.

Avoided Losses Due To Data And AI Downtime

Evidence and data. Interviewees provided examples where unchecked data quality issues resulted in excess costs or lost revenue. Deploying Monte Carlo reduced the likelihood of small data quality issues becoming larger over time and having similar outcomes.

• The VP of BI analytics and strategy at a media organization explained that data and AI downtime resulting in broken or incorrect dashboards for their business users had dramatic effects on decision-making. As a media organization where ad revenue is approximately 30% of annual revenue, decisions about advertising inventory are made by several teams every day. Before deploying Monte Carlo, data quality issues that affected these dashboards resulted in suboptimal decision-making with millions of dollars in implications. With Monte Carlo, it is inherently less likely that data quality issues persist long enough to affect these dashboards and decisions as frequently as before.

• Despite efforts to validate data quality before sending reports to internal and external clients, the AVP of product strategy at the stock exchange noted that customers sometimes detected data quality issues. They summarized: “We did do basic record checking, but we didn't do a whole lot of row level validation or anything like that. What would end up happening relatively frequently is we would send a report, and they would come back to us and say, ‘Hey this doesn’t look right.’ So then we would have to go investigate what was going on from there.” The interviewee also added that Monte Carlo helped their staff detect issues that would have resulted in regulatory fines if left unchecked and unresolved, including an issue where customers in one market would have been overbilled by $1.5 million. The interviewee concluded: “When we're tracking our revenue internally, we’re potentially misrepresenting it. But with our customers, if we overbill them even once, then the trust dissolves.”

• The same interviewee noted that some data quality issues resulted in customers receiving rebates that they were not entitled to, leaving revenue opportunities on the table. The interviewee concluded: “The more trust the business has in the data, the more likely they are to make more money. They may try things like pricing changes to optimize how we think about collecting revenue.”

• The senior manager of data analytics and architecture at a food processing organization said that their team quickly caught a significant data quality issue about product availability due to Monte Carlo. They explained that the issue nearly caused incorrect product availability information to populate for their retail customers, potentially jeopardizing large orders due to the appearance of stockouts. While quality issues that affect the bottom line occur infrequently, the interviewee acknowledged that the consequences of just one issue getting through would be dramatic.

• The manager of data products at an airline recalled an example where a small deployment to make a change to a marquee database object (a table used by analysts to understand flight schedules and operations) used across hundreds of dashboards resulted in an accidental truncation of the table. Due to Monte Carlo alerts, the issue was resolved before it could result in major disruptions.

• The same interviewee noted that their airline’s data team used Monte Carlo to further support the business by setting up custom alerts. The alerts are triggered by certain data points within a set that correlate to a particular business condition or outcome. The interviewee explained: “We’re not just setting up monitors to check whether the data is trustworthy, but we’re setting up monitors to indicate whether a particular event has occurred so someone can go and act. I think this is an untraditional way of using Monte Carlo, but valuable in enabling those business workflows.”

• The director of engineering at a cybersecurity organization noted that it provides reporting for customers based on data queries from its cloud data warehouse. If a query is broken or takes too long to complete, it will time out and the customer will not have their report. The interviewee explained that Monte Carlo alerts on-call staff to these issues or timeouts so they can proactively remediate them.

• With Monte Carlo providing trust and consistency, the product line lead for data platforms at a pharmaceutical organization indicated that the business was less prone to data and AI downtime. Business units that rely on this data, such as manufacturing, were also less likely to experience downtime that could affect operations or revenue. They explained, “Monte Carlo is behind the scenes enforcing that trust through our data mesh dashboard where there’s visibility into whether the data products meet the fitness metrics.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Fifteen percent of the organization’s total data quality issues are customer facing to some degree.

• During customer-facing data and AI downtime, 5% of the organization's hourly revenue is impacted.

• With Monte Carlo, data quality issues are resolved 80% to 90% faster from Year 1 to Year 3, minimizing the lost revenue of customer-facing data and AI downtime incidents.

• While the value of minimizing customer-facing data and AI downtime incidents may be different across organizations, avoided revenue and profit loss has been quantified for the composite organization based on the interviews.

Risks. This benefit will vary among organizations based on:

• An organization’s industry or business specifics relative to the impact or severity of customer-facing data and AI downtime.

• The baseline number of data quality issues within an organization relative to the potential for improvement with Monte Carlo.

Results. To account for these variances, Forrester adjusted this benefit downward by 20%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $1.3 million.

Improved Efficacy Of Internal Decision-Making AI Models With Better Data Quality

Evidence and data. Interviewees explained that deploying Monte Carlo to monitor their organizations’ data estates allowed their AI models, which supported key business decisions, to leverage higher quality data and facilitate more efficient outcomes.

• The AVP of product strategy at the stock exchange noted that trust in data with Monte Carlo allowed them to apply AI modeling to strike optimization in the options market and determine which of an estimated 245 million strikes were most likely to trade and make money, a job previously required of humans. The interviewee continued: “Our AI model looks at volume metrics, trends, and things like that and determines which strikes are going to be the most likely to trade, and trades are how we make money. We’re putting trust into a machine for something that humans used to do, and having Monte Carlo lends that trust to the business that AI is going to do the right thing, at least based on the data. When we first launched strike optimization, we got a 5% increase in revenue. The person [who managed strike] could go off and do something else. So it's an operational efficiency, but we're picking the right strikes, getting more trades, and increasing revenue.”

• The senior manager of data analytics and architecture at a food processing organization said that with Monte Carlo, there was a level of trust in the data where key decisions could be made through AI models across the organization. The interviewee noted that AI models fueled by Monte Carlo-monitored data were being used in most functional areas, including supply chain (routing, warehouse management) and sales (price optimization). The interviewee explained: “[Monte Carlo] gives us trust in the data for the models we run behind the scenes. By making optimized routing decisions for loading our trailers for instance, we save significant money.”

• Interviewees at an airline shared that their organization trusted Monte Carlo-monitored data to power AI and ML models for decision-making. Models supporting dynamic pricing decisions for in-flight products, as well as models supporting to-the-minute flight logistics and operations were inherently more trusted with Monte Carlo-monitored data. The director of data engineering at the airline summarized: “We haven’t had any issues where [bad data quality] has affected these models, but if it were to happen, we've got the technology in place with Monte Carlo that would flag it for us.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Between 0.75% and 1.25% of the composite organization’s total revenue is influenced by AL/ML models, representing a journey from a medium-high AI/ML maturity level.

• Monte Carlo helps the composite organization improve its supply chain operation AI models with better quality data, leading to efficiencies between 0.25% and 0.75% annually.

• While the composite organization achieves revenue improvement via operations cost savings, this example was selected to illustrate the value that better data quality with Monte Carlo can have on AI/ML use cases. Organizations in different industries with different AI/ML use cases may see value manifest in other ways, as highlighted by the interviews.

Risks. This benefit will vary among organizations based on:

• The specific AI/ML models or use cases that Monte Carlo-monitored data feeds at an organization.

• The number of AI/ML models or use cases at an organization.

• The baseline data quality at an organization relative to the potential for improvement with Monte Carlo.

Results. To account for these variances, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $585,000.

Internal Collaboration Benefit From Improved Data Trust

Evidence and data. Before deploying Monte Carlo, interviewees noted that distrust in data across teams in their organizations was common. Teams would recreate reporting based on their own data or spend extra time validating data from other teams. With Monte Carlo monitoring the organizations’ data estates, interviewees spoke of improved  data trust across their organizations that resulted in more frequent collaboration across teams, less rework or redundant efforts, and more decision-making using mutually agreed-upon data.

• The VP of BI analytics and strategy at a media organization explained that before deploying Monte Carlo, each division of the company managed its own data function and teams rarely shared data or reporting. Implementing Monte Carlo as part of its greater decentralized data strategy provided business users with trust in the data that allowed for a self-service data model. Business users across every division could self-service the data they needed for decision-making, and data projects that used to be division-specific were now enterprisewide for the same cost. The interviewee summarized, “Monte Carlo has helped us bridge the gap between the business users across the organization and our data team.”

• The same interviewee noted that a centralized, nine-member data team now supported the entire organization despite the data estate growing three times larger. Before deploying Monte Carlo as part of their data strategy overhaul, more than 30 people supported the organization’s business users across different divisions.

• The senior manager of data analytics and architecture at a food processing organization cited increased collaboration within their business and technical communities as a major benefit of Monte Carlo. Data traveled across multiple projects owned by multiple people and every change could potentially impact another project, model, or team. The interviewee summarized: “Observability gives visibility to our technical community too. Our models are much more stable, and that trust and collaboration becomes much stronger because we have a tool that is observing all of the data interactions.” This interviewee noted that a previous lack of data trust led to the creation of redundant data assets by data scientists across the company.

• The interviewees at a pharmaceutical organization noted that Monte Carlo enabled trust in the data in their business community, which was especially important with the decentralized self-service data mesh approach their data team had worked to foster.

• At the airline, organizational trust in the data had encouraged more users to make decisions based on data. The director of data engineering noted: “We used to only have about 30 to 50 people that would actually go into the data warehouse, write queries, and build reports to show to senior leaders, which is a pretty small number given our company size. We have grown that number tremendously. We have over 1,000 users in [our data warehouse] now, and I think that growth can only really happen when you have people that trust the data you're putting in their hands.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• Six teams staffed with an average of four FTEs each create and consume data products.

• With Monte Carlo, shared trust in data across the organization fosters more centralized efforts to create and share data products. The composite avoids 65% of the effort previously spent creating team-specific data products as it breaks down silos across the organization.

• The average fully burdened annual salary for an employee who creates data products is $110,000.

• It achieves a 75% productivity recapture on avoided effort, as not all reclaimed time will be repurposed toward value-added work.

Risks. This benefit will vary among organizations based on:

• The number of teams or employees creating and consuming data products within an organization.

• The degree to which data assets are shared or trusted within an organization.

• The skill and capacity of personnel tasked with creating data products within their respective teams.

Results. To account for these variances, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $400,000.

Avoided Cloud Data Storage And Compute Costs

Evidence and data. Expanding data estates at interviewees’ organizations caused significant cost increases due to limited visibility into query or pipeline efficiency, the types of data going into storage, and issues resulting in data backfills. Monte Carlo provided the organizations with views into where and how data was being used. The platform also allowed them to prioritize data assets while avoiding storage and compute costs associated with poorly performing queries or unused/less valuable data assets.

• The director of data engineering at an airline told Forrester that Monte Carlo enabled them to better understand opportunities for optimizing data engineering workload queries or self-service analyst queries, allowing the queries to run faster and saving on compute costs with the cloud data warehouse provider. Alongside other custom dashboards and monitors built by the data operations team, the organization saved more than 40% in data warehouse compute costs over the past year, some of which was attributable to understanding gleaned from Monte Carlo.

• The director of engineering at a cybersecurity organization explained that they used Monte Carlo to investigate and save money on their most expensive queries. The interviewee explained, “With the [Monte Carlo] performance dashboard, we can actually see what our most expensive queries are so we can bring down the costs.” They continued: “We spend close to eight figures with [our cloud data warehouse], so Monte Carlo can help us save over 1% on that bill, and sometimes a poor-performing query can be that or more. And that’s being conservative.”

• The product line lead for data platforms at a pharmaceutical organization explained an issue that was caught early with Monte Carlo before there was any downstream impact to their data (or excessive costs), noting: “We started ingesting thousands and thousands more records than what those tables were supposed to bring in normally. Once they get into the warehouse and start transformation and processing all the data that's all junk, costs will go up. If you detect an anomaly and you stop the pipeline up front, it can save money.”

Modeling and assumptions. Based on the interviews, Forrester assumes the following about the composite organization:

• The composite organization spends between $3 million and $4 million annually from Years 1 to 3 with its cloud data storage provider.

• It avoids 2.5% of this spend with Monte Carlo, as insights, monitoring, and alerts contribute to better data hygiene and data warehouse allocation. Performance dashboards allow the composite organization to identify and improve its worst-performing and most expensive pipelines and queries.

Risks. This benefit will vary among organizations based on:

• An organization’s total spend with a cloud data storage provider. 

• The baseline quantity and performance of an organization’s queries and ability to act upon insights from Monte Carlo.

Results. To account for these variances, Forrester adjusted this benefit downward by 15%, yielding a three-year, risk-adjusted total PV (discounted at 10%) of $180,000.

Unquantified Benefits

Interviewees mentioned the following additional benefits that their organizations experienced but were not able to quantify:

• Improves customer experience and reputational benefit. With a Monte Carlo partnership, interviewees noted that bad data and AI affected their end customers less often. This outcome resulted in less customer-facing data and AI downtime (as quantified in benefit B) and contributed to a better customer experience overall.

  • The AVP of product strategy at the stock exchange noted how observability on Monte Carlo facilitated documentation about how they test data for the Securities and Exchange Commission, which contributed to approval for new processes and products.
  • An interviewee at an airline noted that internal Net Promoter Score℠ (NPS) scores for their data team improved several points after implementing Monte Carlo.
  • The director of engineering at a cybersecurity organization noted that catching data quality issues before they manifested in reporting to end customers helped protect their platform’s reputation.

• Uncovers new product and revenue opportunities. Some interviewees cited their Monte Carlo partnership as a data product value-add for their end customers that contributed to additional product opportunities and customer retention. The AVP of product strategy at the stock exchange noted their ability to win significantly more exchange-traded funds (ETFs) after building trusted data pipelines with Monte Carlo. They explained: “We could launch a product to help with our ETF business because we could build trusted pipelines with Monte Carlo. Now our winning in the ETF space is up 75%.”

• Empowers the organization with data. Interviewees told Forrester that data monitored with Monte Carlo was inherently perceived as more trustworthy and “agreed upon” across their organizations, leading to cross-team willingness to use data or data products from other teams or sources. Multiple interviewees noted that the Monte Carlo deployment enabled a decentralized self-service data strategy and allowed higher quality data to be used by more employees more frequently, contributing to better results across the business.

Flexibility

The value of flexibility is unique to each customer. There are multiple scenarios in which a customer might implement Monte Carlo and later realize additional uses and business opportunities, including:

• The downstream impact of decision-making on better quality data. Interviewees expressed optimism that monitoring their data with Monte Carlo would contribute to better quality data and contribute to better decision-making overall, especially as AI/ML decisioning became more prevalent within their organizations. The AVP of product strategy at the stock exchange noted that in addition to the AI/ML use cases already in practice, more were on the horizon and included trading behavior A/B testing historically managed by employees, which would save personnel time and potentially lead to better business results.

Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Total Economic Impact Approach).