Already a successful big data visualization platform, mosaicOA from CJC is now enhanced with machine-led analytics – developed in partnership with T-DAB.AI (T-DAB). The use of Principle Component Analysis (PCA) in mosaicOA enables clients to more comprehensively realise the predictive power of IT Operations Analytics.
Firms across the capital markets spectrum typically have multiple bespoke and specialist IT infrastructures distributing real time data, both internally and to global counterparties. These are critical to running their market operations and disruption comes at a high cost.
IT Operations Analytics (ITOA) provides essential information on infrastructure performance to enable:
- Diagnosis of bottlenecks
- Analysis on outages and the impact of market movement
- Capacity management forecasting of device failure
- Impact analysis of new technology or software upgrades
MosaicOA, a SaaS-based big data visualization platform from CJC is designed to support IT Operations Analytics and deliver insight via visualisations, analytics and reports from clients’ mission critical IT infrastructures.
A typical capital market server can easily create 50 million statistics every day – from CPU, networking and application level. This data is recorded in real-time from the entire infrastructure to a time series database running in CJC’s private cloud, then presented to the client via an interactive front end.
Users are typically IT operations and engineering teams, as well as service and capacity managers.
Human to machine led analytics
The mosaicOA platform provides excellent human led analytics, presenting users with the ability to interrogate data and create meaningful visualisations. For example, the visualisation below shows a selection of different manually created views that shows the relationship for market data message rates and the total number of processes used to manage:
This visualisation demonstrates a key challenge around the correlation of different data sets. Clearly, the steady increase in market data update rates had an impact on the total number of processes. However, the number of processes ranges from just over 100 to over 600 with market data updates as a numeric between 0 and into millions. Placing these into a single view would result in number of processes being rendered invisible, so for human led visualisations, several views are required to show all the interrelated values.
CJC wished to offer advanced machine led analytics, particularly in highlighting metrics which did not typically correlate. They also wanted the final presentation to be actionable, so users could immediately see and understand system behaviour. The team needed specific expertise to address the project and structure their database to deliver the enhanced insight. They called on T-DAB.AI (T-DAB) to provide that expertise.
Principal component analysis
T-DAB presented Principle Component Analysis (PCA) as the solution for mosaicOA. PCA identifies co-linearity in collected infrastructure metrics, enabling a reduction in the number of dimensions represented. These metrics are then normalised from their base units of measure into a Performance Index; a value between -100 (under-utilised) to +100 (over-utilised).
In the first instance this was applied to 5 separate dimensions, CPU, Memory, Disk usage, Network and Disk IOPS. Once these metrics are normalised, they can be blended together into a Performance index which allows us to build a Hot and Cold view of a set of servers.
The performance index can be accessed via the front end mosaicOA GUI. A list of servers is simply inputted, and any additions will then be compared to each other against a time frame.
Adding personas to PCA
Creating personas is a tried and trusted method of data sorting. For mosaicOA, a persona is a group of servers that perform specific roles to the organisation. For example, physical servers could be separated from virtual, Windows and Unix operations systems, server’s specification and type (Dell / HP / Virtual / Cloud) and, importantly, the ‘Chinese wall’ separation of business units, departments and regions commonly found in regulated capital markets.
The personas are separated by individual windows, which clearly show the utilisation of that persona. Aggregation is carried out by the client and the algorithm then works out, in real time, which servers are outside the bounds of normal performance for that selected persona, over a given time frame. T-DAB provides continuous technical project support through a senior data engineer and data scientist.
This approach allows firms to quickly identify the most used (and underused) servers, allowing them to proactively manage resources and make cost savings and decisions on their infrastructure.
T-DAB and CJC
T-DAB helped CJC structure the software development process into two-week sprints and the tools for configuration management were built in a central location to reduce the support overhead. MosaicOA uses a continuous integration server with automated testing that allows not only automated regression tests for releases, but also automated stress tests to help define specific use-case versions of the offering.
CJC now have enhanced data competency with T-DAB.AI as a partner, enabling them to push the boundaries of data innovation. Indeed, as users of mosaicOA themselves (see Global Head of Product Management, Steve Moreton’s post here), the enhanced tooling was used on their own environment to find a 40% capacity reduction.
