The continuous increase in the volume and detail of data captured by organizations, such as the rise of social media, Internet of Things (IoT), and multimedia, being utilised alongside traditional transactional information has produced an overwhelming flow of data in either structured or unstructured format. This data is increasingly being used in virtually real time and combined with historical data for both Operational and Predictive Analytics. The growth is relentless as is the desire to keep the data available and useable at relatively short notice for business purposes.
The physical infrastructure and operational costs to maintain this ever growing lake of data, online and immediately available for analysis, is becoming a burden on both in-house datacentres and cloud based environments. Tier 1 online data storage is, typically, chosen to be fast so that the query results can be returned in as short a time as possible. Those faster access storage devices are more expensive compared to other storage types suc as Network Storage or Archival systems. When the data volumes grow from gigabytes to terabytes and petabytes, the online stoage costs grow steeply high. Users require to have solutions that provide access to the data on demand with minimal latency but also to minimise the overall storage cost. To mitigate this rising cost organisations keep most frequently accesed data “closer” to the applications on faster and more expensive storage and then move the less frequently accessed data to a lower cost storage. However, when they need to access the less frequently accessed data they want access to that to be transparent and seamless with preferably zero operational intervention.
The solution to this challenge is to utilise an intelligent connection layer to bridge the data access requests to the specific data elements stored in a lower cost storage medium such as an archive platform or tiered storage infrastructure.
Often, in the industry, people use the term hierarchical or tiered storage and it generally means having Solid State Drive for faster data access and SAN or NAS storage for 2nd tier storage. In this case, we are talking of very large data (100’s of terabytes and petabytes). So, we need to have a more intelligent solution to manage that large volume of data.
In an iFusion Intelligent Archive deployment users will adopt a 2-tier storage architecture.
Within this users plan to move data older than a certain time (for example, 3 months or 6 months) to archival storage, while maintaining the ability to query the online data and archival data in a single submission. iFusion Analytics provides an Intelligent Archiving connector to access the specific data elements irrespective of the layer they reside on and return these back within the single result set. This is managed by the Archival Coordinator at the initiation of the requestor, to move the data to one of the supported tier-2 storage devices. When the data needs to be queried, the iFusion data virtualization process recognizes which data is on what type of storage (tier-1 or tier-2), and uses federated query processing techniques, to query the data from each storage tier and combines the returned values to give the final results to the requestor / user.
For Compliance reasons or when data has no further business use it is necessary to completely remove this from the system. The iFusion Archival Connector manges a retention policy. This means that after a period (say 1 year or 3 years) the data stored in archival storage this is purged and will no longer be retrievable. This is a fully auditable procedure that allows organisations to meet the demands of local regulators and data protection agencies.
By being aware of both the business requirements and operational cost of large amounts of data iFusion Analytics provides not only a best in class platform to deliver insight to the organisation but a cost effective mechanism to collect, store and delete the data over time. This two dimensions ensures that maximum value at minimal cost is incurred by our customers.