Backand Supports the HP Data WarehousePosted by Itay Herskovits on Jan 08, 2015
When HP had a problem with their internal business intelligence processes, they came to Backand for a solution. Using their Backend-as-a-Service offering built off AngularJS and Twitter Bootstrap, Backand was able to rapidly address the deficiencies in HP’s internal data warehousing process, allowing the organization to quickly deploy a new application that both replaced and enhanced the former manual data processing tasks.
In the following HP case study, we go more into detail about what HP was looking for. Where their initial estimates ranged from months or years to develop a tool internally, Backand was able to provide them with the tools to build their own solution in well under two months. By using Backand, HP was able to increase the robustness of their internal processes without resorting to reliance on an external vendor, developing a new enterprise-level application in the process with minimal guidance. While just one example, this case study calls attention to how quickly Backand can make an impact in your organization, providing enterprise solutions on a fraction of the budget.
Providing Support for HP’s Enterprise Data Warehousing
Many companies have business processes that could use streamlining. In some cases the processes are too complex, requiring special training to complete the multitude of constituent steps. In others the processes may be very fragile, either breaking easily or, even worse, quietly failing to function. HP unfortunately had both issues with some of their data warehousing processes. In this case study we’ll explore how Backand helped HP to streamline their business intelligence process, reducing complexity while at the same time improving the final quality of the data.
HP came to Backand with a problem. As a part of their Business Intelligence (BI) activities, they were performing a process they called Extract Transform Load, or ETL. This process was focused on consolidating data from multiple sources into a set of three Microsoft Excel spreadsheets, which were then uploaded into their data warehouse. The spreadsheets contained information from three locations – one spreadsheet per location – covering customer information, maintenance logs, and product life cycle data among myriad other data points in their Enterprise Resource Planning (ERP) system. Each day, dedicated staff at each of the three locations (one in Western Europe, one in Central Europe and one in Israel) spent a significant amount of time updating these spreadsheets to include all of the data from that business day. Once they completed their tasks, they would combine all of the spreadsheets from the three locations into one master spreadsheet, and members of their IT department would write and run scripts that parsed the master spreadsheet, sending the data to a central data warehouse.
The entire process was fraught with problems. There were data cleanliness issues brought on by the manual portions of the process, with the data suffering from formatting errors, typos, incorrect sequences, and other concerns. On top of this, each location continually updated a centralized master spreadsheet, meaning that all relevant changes needed to be made multiple times by hand. One of the problems with Microsoft Excel is that it doesn’t allow concurrent user access, meaning that many of these updates were being made in “read only” mode. Users were finding their own ways around this issue, leading to excessively large workbooks with multiple in-progress spreadsheets being created and maintained by different employees trying to circumvent the user locks. As a result, the master spreadsheet grew very fragile and frequently caused problems with the ETL process itself. The smallest discrepancy or typo would crash the scripts, causing IT to waste time hunting down and fixing the offending error before being able to restart the process.
The end result of the process deficiencies was that the data from the ETL process was frequently delayed, slowing down each of the three participating branches as they fought against the underlying process. As long as things persisted, HP’s BI data would be woefully inaccurate and out of date, creating an incorrect picture of the operations at each of the three locations. An enterprise-oriented solution was sorely needed, and HP came to Backand for help.
HP had, at a glance, two options for resolving the issue.
Option 1 – Modify the ERP System
The first option was to modify the ERP system to perform the necessary data warehousing activities. While this solution would have been optimal in terms of achieving the end goals, modifying the underlying software was both cost- and effort-prohibitive. Implementing the modifications would have been an extensive project, as HP would need to take the needs of many different departments into account. Given the global nature of the departments involved, simply coordinating the efforts would have been an expensive effort.
To compound the situation, the departments involved didn’t have the resources to devote to revamping the ETL process. “This would have meant creating an Enterprise-level solution,” stated one of the managers involved in the process, “which would need to operate on a longer time cycle. The proposal would need to go through cost analysis and budgetary proposal submissions, and the process to get final approval could take as long as the development effort itself.” During the two year development cycle, the three locations would be forced to continue using the same broken ETL processes, continuing to deliver unclean and misleading data.
Option 2 – Develop a Management Application
The second option was to build out a solution using either Microsoft’s .NET framework or a smaller application built off of Microsoft Access. According to one of their IT managers, “HP had already attempted this option, and while we knew that it would be a smaller effort with more reasonable costs, taking much less time to build out, we lacked the talent to perform the tasks in-house and would have needed to contract with an outside vendor.” Doing so would have proportionally increased the cost of the effort, as HP would need to provide funding for everything from process consultation to development and quality assurance.
Even with a modest estimate of $20,000 for two month’s worth of work to implement the system, HP would have had one remaining significant problem: end user change requests. Every time a user wanted a change in the application to better support their business needs, HP would require more interaction with the external vendor developing the tool, increasing time to delivery and associated costs. HP was looking to break the cycle of vendor dependence, and trying to find a solution that was cheaper to build, cheaper to maintain, agile, and (most importantly) entirely in their control.
Backand’s software supported importing Excel spreadsheets natively, and was easily configured to send the data to a second warehousing location. Backand worked with HP’s IT department to get an initial test system built in only two days. The IT department, using Backand’s tools, was able to build the entire system themselves very rapidly with only minor input and support from Backand’s engineers. Since Backand’s service is very easy to use – and very configurable – HP was able to own the entire product from the very beginning, using the knowledge they already possessed in-house.
Following the rapid development of the prototype, HP deployed the tool to internal customers for a two-week trial. “We thrive on customer feedback at Backand, and when we received the additional requirements and observations from the initial HP test group we went to work,” stated Itay Herskovits, CTO at Backand. “We pride ourselves on our agile processes at Backand, and we were able to move very quickly to incorporate the suggestions into our product offering.” After only a day of work, Backand delivered the updated product to HP. Backand maintained a focus on creating an Excel-like experience, in order to best leverage the experience of the people involved with the old ETL process. This ended up reducing training times involved, as anyone familiar with Microsoft Excel could get up and running on the product very quickly. Thanks to the Backand product being highly configurable, the Backand team was able to emulate the experience that HP was expecting as closely as possible.
The pilot system received highly positive feedback, and users rapidly engaged with the system. After the two-week pilot, only two issues arose – both related to the ETL process itself, rather than the new product. The first of these issues centered around email notifications for any significant changes to the data, which Backand was able to very quickly implement. The second issue was focused on data-integrity, particularly with multiple users working on the system simultaneously – an issue promptly addressed using the secure and robust back office, with redundant hardware allocations that made the system very stable. HP finally had something they had never had before with their prior ETL process – a single data location with 100% data integrity.
Following the initial two-week trial, the tweaks were made and the system was deployed a few weeks later. The new system has been running successfully ever since. Internal customers have been ecstatic about the replacement. To quote one of the IT people involved: “After using Backand, our deployment process greatly simplified. Instead of distributing software via IT, we are able to simply distribute a URL and update the application as we need – without end-user involvement.”
Through leveraging the Backend as a Service (BaaS) and it’s automated back office, Backand was able to deliver a solution to HP’s ETL process that was performant, secure, and addressed all of the deficiencies of the enterprise-level process. The entire process was rapid – from concept to deployment in under two months – and secure from day one. The enterprise-level application greatly simplified the collaboration among HP’s three locations across Europe and Israel, solving extremely quickly a problem that had seemed insurmountable with the current processes.
Learn how Backand can help your enterprise build Angular apps, both new and ones that are migrating. – Get started now.