Complex challenges, successful outcomes

nFolks' engagement with the Commonwealth of Massachusetts (COMA) Executive Office of Health and Human Services (EOHHS) Health Safety Net (HSN) turned a lost cause a client preparing to return IBM software into a win and an expanded IBM footprint.

The challenge

COMA EOHHS HSN was grappling with an aging, roughly 20 year old DataStage system and the challenge of integrating it into a modern IT infrastructure on IBM Cloud Pak for Data (CP4D). Initial apprehension toward CP4D and an inclination toward AWS compounded the difficulty, leading to complaints and demands for a refund or credit for CP4D.

Our solution

nFolks built an action plan around migrating the legacy DataStage jobs and a series of working sessions with the EOHHS project lead, to demonstrate how integrating DataStage with CP4D could add substantial value. Using the client's own data and N-Cube, our flexible sandbox packed with third party tools that replicates complex customer landscapes we showcased a migration approach for an unsupported version of DataStage alongside an MVP, shifting COMA's perception of CP4D through architectural discussions, demos, and proofs of value.

Results

COMA reconsidered CP4D and adopted it as the new standard within EOHHS an affirmation of nFolks' approach and execution. In the process, the team prevented a potentially disputed IBM HSN software return in Q1 2022, protecting both the customer relationship and IBM's footprint.

The bottom line

The engagement epitomizes nFolks' ability to maneuver through challenging technology landscapes solving hard problems, rebuilding client confidence, and helping IBM win.

City of New York HRA was upgrading their infrastructure from older versions of DataStage to DataStage 11.3 which was implemented by IBM and as a part of the project, which was a very short engagement, worked closely with NYC HRA to Install and configure IBM InfoSphere Information Server 11.3.

Implementation of IBM MDM v11.6, DSC, RESTful, Bamboo, Maven & Oracle 12c, Kafka for major financial institution in Australia.

The implementation consisted of two parts: The Global Registry (TGR) and Australian Operational Customer Master (AOCM).

The Global Registry (TGR) was a combination of virtual MDM (Matching Hub MH) and physical MDM (Operational Hub OH). MH provides matching entity link based on outcome of algorithm defined with probabilistic matching, while OH persists with the member data for enriching additional data such as hierarchy and relationships. Data changes for every member are notified to downstream systems by triggering notification messages. nFolks designed and developed ‘maintain party service’, ‘publish derived record’, ‘data model’ and ‘algorithm’ to accommodate entity attributes. Configured Bamboo jobs for automating build, deploy and test processes.

Australian Operational Customer Master (AOCM)

AOCM is a master slave database to consolidate customer data from multiple data sources and to serve the required data to downstream applications or systems. Java Server Faces web framework-based Data stewardship application is customized to integrate the new attributes, and used for searching customers based on user-defined criteria. AOCM is capable of emitting event notifications for changes to ‘party profile’ and ‘customer-to-account relationship’ to downstream systems.

On behalf of IBM, nFolks designed and developed standalone Java application ‘Missing Data Remediation’ for remediating missing data in AOCM or in source systems, and also migrated source control repository to GIT and validated the generated EAR works. In addition, nFolks also upgraded AOCM from MDMv10.5 to MDMv11.5, and included Oracle migration from 11g to 12c. By ensuring that AOCM MDM migration did not affect functionalities, customer data and web services, nFolks ensured business continuity for the client with minimal disruptions.

Louisiana Department of Corrections implemented the DB2 migration by creating the DB2 Data Store as their first IT enterprise modernization. The department planned to extract the data from their existing Mainframe Mapper and Notes legacy databases and move the data to DB2 database as the basis for the enterprise “database of record” after going through the compete cycle of profiling, cleansing and transforming the legacy data for web-OLTP using IBM Information Server Suite of products as well as query and reporting using IBM Cognos BI tool.

The issue for the client was extracting the data from legacy hand written very old databases such as Mapper and Notes for different database objects/tables & applying rules on top of it to load data into DB2 Database. All this was targeted to be completed in limited budget and fixed timeline. In addition, the Information Server was deployed on Windows Operating system with limited resources sitting on a VMWare for which support model from vendors was not existing. As these were legacy databases parent/child relationship between tables was also not present. As a result this check needs had to be made in ETL code and records which did not meet the requirement had to be captured and emailed to Data Owners.

nFolks Data Solutions took this as a challenge and used IBM Information Server suite of tools provided by client by coming up with different solutions such as using the correct ODBC drivers as extracting data from legacy database was not easy with limited drivers supporting it partially. nFolks used a combination of Parallel and Server jobs as it was creating lot of processes on windows server & parallel extract or huge extract would break the connection with legacy databases Scripts were written to monitor the hanged processes & clean them frequently as the no of resources are limited. The data was also Unicode and special characters had to be handled carefully so that data retains its original form and loaded into DB2 database.

Following the implementation, the department made the process seamless. Programs & scripts written by nFolks Data Solutions made the process seamless and in time delivery of project within limited budget by addressing all issues which made the migration of legacy databases to DB2 database successful.

Smart Connect Data Warehouse project is one of the high priority projects for client connecting large number of data points from their customer meters. The Metadata Warehouse repository is one of the important modules of this project being implemented to generate different reports related to impact analysis, data lineage and Business lineage.

The client, as part of Smart Connect project, wanted to implement the Metadata Warehouse repository across systems to find the impact analysis for any field changes, find data lineage and business lineage using IBM Business Glossary, Metadata workbench and other IBM foundation tools. The challenge was to integrate the solution with the large number of lot of end-to-end systems including vendor technologies such as SAP Business Objects reports, Erwin data models, Teradata, Oracle, DataStage code, stored procedures, extended mappings and others.

Representing IBM at client site, nFolks Data Solutions implemented the project by importing all the metadata assets using IMAM for Erwin data models, Teradata, Oracle, DataStage ETL jobs and others. nFolks also created Business Categories, terms and other Business assets using Business Glossary, and mapped the IT assets to Business assets in Business Glossary to generate data lineage, business lineage & impact analysis using Metadata Workbench. Worked closely with IBM Israel Labs to address the product issues, we offered help in including right features which would provide more granular information for the client.

Client was able to find impact analysis if any IT asset such database column or ETL jobs, generate end-to-end lineage from source system to SAP BO reports, generate business lineage. In addition, both function & technical teams are able to use Business Glossary.

Cleanroom project was a sales merger project between two major US based airlines, which were in the process of merging their operations, including sales data. The target for the implementation was to generate market shares of the two airlines with reference other airlines for improving sales.

To major airlines in the US decided to merge. Prior to the completion of the merger, the airline companies were not allowed to run combined sales reports, and the same was outsourced to ZS Associates where nFolks Data Solutions was asked to help on the ETL part for loading sales data across different routes, sources, destinations, legs, seats and others compared to other competing airlines.

nFolks Data Solutions used various data tools to assimilate data from the two airlines while adhering to legal requirements for data privacy prior to the merger. Following the solution provided y nFolks, client was able to seamlessly generate market share reports for improved business planning for activities such as route optimization and sales planning.

Client is a distributor for a large Japanese automotive company in the US. Client operates with 150+ dealerships in 5 states. Data Migration project was one of the high priority projects for the client wherein they were migrating their systems from IBM InfoSphere DataStage 7.0 version to IBM InfoSphere 8.7.

Client wanted to migrate the code from DataStage 7.0 version to DataStage 8.7 version but most of the jobs were server jobs with few plug-ins being used which were no longer available in DataStage 8.7 version and these needed to be converted to parallel jobs. Overall Architecture assessment was based on different factors such as volume assessment, server assessment, topology design, resource estimation, performance tuning before installing the DataStage 8.7 version and migrating the code from lower version to higher version.

nFolks Data Solutions performed server assessment and upgraded the code from DataStage 7.0 version to DataStage 8.7 version by converting all the server jobs to parallel jobs whose plugins were depreciated. The migration was successful with minimal disruptions to the operations of the client. The implementation helped the client achieve Business Continuity by staying abreast of the latest data technologies.

ProfitMax NextGen objective was to calculate Profit and Loss metrics for the wholesale line of businesses at various levels like Customer, Relationship, Officer, Accounting Unit (AU), and Organizational Unit (OU) by building a Profitability Data Warehouse. The ProfitMAX NextGen is a rewrite of current legacy application, which is a customer and product profitability tool used by the Wholesale Banking division of the client to view customer / product profitability. 3000 users were estimated to access the solutions and would enable them to view and measure profitability at an officer, AU and product level.

The issue for the client was that the Data Warehouse modeling was done with very poor practices & no best practices had been in place by previous implementation partner, leading to large number of issues, resulting in inaccurate data with missing relationships between target tables. The issues resulted in an unstable code and the reporting results were affected.

nFolks Data Solutions, along with another implementation partner, were requested to fix the issues caused by the previous poor implementation. With best Solution Developers and Architects being deployed with right strategy, we were able to fix the code issues as well as the design issues & developed a solution which resulted in generating reports at 20 levels down granular reports across the whole sale line of Business Services and products for the financial institution, thereby meeting the primary end-point requirements. With the implementation, client was able to save lot of money and got and effective solution, which resulted in generating reports granular reports for 20 different at 20 levels, thereby providing streamlined data for improved decision making.

Client was a holding company for popular retails clothing and accessories brands in the US, and was merging the brands under a single entity for better business management and decision making. The merger process was to be completed in two phases and all the major brands under the portfolio were being migrated to upgraded systems.

Client wanted to merge all the 8 brands in the portfolio in two phases by migrating data from different sources such as SAP R/3 4.7, files and DB2 to SAP ECC 6.5 so that these brands are managed from one SAP ECC system of record.

Representing IBM at client site, nFolks Data Solutions implemented the project by creating the DataStage jobs with best practices and performance tuning steps to merge the data for different brands from SAP R/3 4.7 System to SAP ECC 6.5 system. Following the implementation, client was able to access all the brands data from one SAP ECC 6.5 system as opposed to individual systems or different system of records for different brands