Tag Archives: Object

The Evolution of Object Storage


It’s a truism that the amount of data created every year continues to grow at exponential rates. Almost every business now dependends on technology and the information those businesses generate has arguably become their greatest asset. Unstructured data, the kind best kept in object stores, has seen the biggest growth. So, where are we with object storage technology and what can we expect in the future?

Object storage systems

Object storage evolved out of the need to store large volumes of unstructured data for long periods of time at high levels of resiliency. Look back 20 years and we had block (traditional storage) and NAS appliances (typically as file servers). NAS – the most practical platform for unstructured at the time – didn’t really scale to the petabyte level and certainly didn’t offer the levels of resiliency expected for long-term data retention. Generally, businesses used tape for this kind of requirement, but of course tape is slow and inefficient.

Object storage developed to fill the gap by offering online access to content and over the years has developed into a mature technology. With new protection methods like erasure coding, the issue of securing data in a large-scale archive is generally solved.

Object stores use web-based protocols to store and retrieve data. Essentially, most offer four primitives, based on the CRUD acronym – Create, Read, Update, Delete. In many instances, Update is simply a Delete and Create pair of operations. This means interacting with an object store is relatively simple — issue a REST-based API call using HTTP that embeds the data and associated metadata.

This simplicity of operation highlights an issue for object storage: Applications need to be rewritten to use an object storage API. Thankfully vendors do offer SDKs to help in this process, but application changes are required. This problem points to the first evolution we’re seeing with object: multi-protocol access.

Multi-protocol

It’s fair to say that object stores have had multi-protocol access for some time, in the form of gateways or additional software that uses the object store back-end as a large pool of capacity. The problem with these kind of implementations is whether they truly offer concurrent access to the same data from different protocol stacks. It’s fine to be storing and retrieving objects with NFS, but how about storing with NFS and accessing with a web-based protocol?

Why would a business want to have the ability to store with one protocol and access via another? Well, offering NFS means applications can use an object store with no modification. Providing concurrent web-based access allows analytics tools to access the data without introducing performance issues associated with the NFS protocol, like locking or multiple threads hitting the same object. The typical read-only profile of analytics software means data can be analyzed without affecting the main application.

Many IoT devices, like video cameras, will only talk NFS, so ingesting this kind of content into an object store means file-based protocols are essential.

Scalability

One factor influencing the use of object stores is the ability to scale down, rather than just scale up. Many object storage solutions start at capacities of many hundreds of terabytes, which isn’t practical for smaller IT organizations. We’re starting to see vendors address this problem by producing products that can scale to the tens of terabytes of capacity.

Obviously, large-capacity hard drives and flash can be a problem here, but object stores could be implemented for the functional benefits, like storing data in a flat name space. So, vendors are offering solutions that are software-only and can be deployed either on dedicated hardware or as virtual instances on-premises or in the public cloud.

With IoT likely to be a big creator of data and that data being created over wide geographic distributions, then larger numbers of smaller object stores will prove a benefit in meeting the ongoing needs of IoT.

Software-defined

Turning back to the software-only solutions again for a moment, providing software-only solutions means businesses can choose the right type of hardware for their environments. Where hardware supply contracts already exist, businesses can simply pay for the object storage software and deploy on existing equipment. This includes testing on older hardware that might otherwise be disposed of.

Open source

The software-defined avenue leads on to another area in which object store is growing: open source. Ceph was one of the original platforms developed as an open source model. OpenIO offers the same experience, with advanced functionality, like serverless, charged as a premium. Minio, another open source solution, recently received $20 million in funding to take its platform to a wider audience, including Docker containers.

Trial offerings

The focus on software means it’s easy for organizations to try out object stores. Almost all vendors with the exception of IBM Cloud Storage and DDN offer some sort of trial process by either downloading the software or using the company’s lab environment. Providing trials opens software to easier evaluation and adoption in the long run.

What’s ahead

Looking at the future for object storage, it’s fair to say that recent developments have been about making solutions more consumable. There’s a greater focus on software-only and vendors are working on ease of use and installation. Multi-protocol connects more applications, making it easier to get data into object stores in the first place. I’m sure in the coming years we will see object stores continue to be an important platform for persistent data storage.



Source link

Object Storage: 8 Things to Know


Object storage is one of the hottest technology trends, but it isn’t a particularly new idea: The concept surfaced in the mid-90s and by 2005 a number of alternatives had entered the market. Resistance from the entrenched file (NAS) and block (SAN) vendors, coupled with a new interface method, slowed adoption of object storage. Today, with the brilliant success of Amazon Web Services’ S3 storage system, object storage is here to stay and is making huge gains against older storage methods.

Object storage is well suited to the new data environment. Unstructured data, which includes large media files and so-called big data objects, is growing at a much faster rate than structured data and, overall, data itself is growing at a phenomenal rate.

Experience has taught us that traditional block systems become complex to manage at a relatively low scale, while the concept of creating a single pool of data breaks down as the number of appliances increases, especially if the pool crosses the boundaries of different equipment types. Filers have hierarchies of file folders which become cumbersome at scale, while today’s thousands of virtual instances make file-sharing systems clumsy.

An inherent design feature of object stores is distribution of objects across all of the storage devices, or at least into subsets if there is a large number of devices in the cluster. This removes a design weakness of the block/file approach, where failure in an appliance or in more than a single drive could cause either a loss of data availability or even loss of data itself.

Object stores typically use an algorithm such as CRUSH to spread chunks of a data object out in a known and predictable way. Coupling this with replication, and more recently with erasure coding, means that several nodes or drives can fail without materially impacting data integrity or access performance. The object approach also effectively parallelizes access to larger objects, since a number of nodes will all be transferring pieces of the object at the same time.

There are now a good number of software-only vendors today, all of which are installable on a wide variety of COTS hardware platforms. This includes the popular Ceph open source solution, backed by Red Hat. The combination of any of these software stacks and low-cost COTS gear makes object stores attractive on a price-per-terabyte basis, compared to traditional proprietary NAS or SAN gear.

Object storage is evolving to absorb the other storage models by offering a “universal storage” model where object, file and block access portals all talk to the same pool of raw object storage.  Likely, universal storage will deploy as object storage, with the other two access modes being used to create a file or block secondary storage to say all-flash arrays or filers. In the long term, universal storage looks to be the converging solution for the whole industry.

This trend is enhanced by the growth of software-defined storage (SDS). Object stores all run natively in a COTS standard server engine, which means the transition from software built onto an appliance to software virtualized into the instance pool is in most cases trivial. This is most definitely not the case for older proprietary NAS or SAN code. For object stores, SDS makes it possible to scale services such as compression and deduplication easily. It also opens up rich services such as data indexing.

Continue on to get up to speed on object storage and learn how it’s shaking up enterprise storage.

(Image: Kitch Bain/Shutterstock)



Source link

Object Storage: 8 Things to Know


Object storage is one of the hottest technology trends, but it isn’t a particularly new idea: The concept surfaced in the mid-90s and by 2005 a number of alternatives had entered the market. Resistance from the entrenched file (NAS) and block (SAN) vendors, coupled with a new interface method, slowed adoption of object storage. Today, with the brilliant success of Amazon Web Services’ S3 storage system, object storage is here to stay and is making huge gains against older storage methods.

Object storage is well suited to the new data environment. Unstructured data, which includes large media files and so-called big data objects, is growing at a much faster rate than structured data and, overall, data itself is growing at a phenomenal rate.

Experience has taught us that traditional block systems become complex to manage at a relatively low scale, while the concept of creating a single pool of data breaks down as the number of appliances increases, especially if the pool crosses the boundaries of different equipment types. Filers have hierarchies of file folders which become cumbersome at scale, while today’s thousands of virtual instances make file-sharing systems clumsy.

An inherent design feature of object stores is distribution of objects across all of the storage devices, or at least into subsets if there is a large number of devices in the cluster. This removes a design weakness of the block/file approach, where failure in an appliance or in more than a single drive could cause either a loss of data availability or even loss of data itself.

Object stores typically use an algorithm such as CRUSH to spread chunks of a data object out in a known and predictable way. Coupling this with replication, and more recently with erasure coding, means that several nodes or drives can fail without materially impacting data integrity or access performance. The object approach also effectively parallelizes access to larger objects, since a number of nodes will all be transferring pieces of the object at the same time.

There are now a good number of software-only vendors today, all of which are installable on a wide variety of COTS hardware platforms. This includes the popular Ceph open source solution, backed by Red Hat. The combination of any of these software stacks and low-cost COTS gear makes object stores attractive on a price-per-terabyte basis, compared to traditional proprietary NAS or SAN gear.

Object storage is evolving to absorb the other storage models by offering a “universal storage” model where object, file and block access portals all talk to the same pool of raw object storage.  Likely, universal storage will deploy as object storage, with the other two access modes being used to create a file or block secondary storage to say all-flash arrays or filers. In the long term, universal storage looks to be the converging solution for the whole industry.

This trend is enhanced by the growth of software-defined storage (SDS). Object stores all run natively in a COTS standard server engine, which means the transition from software built onto an appliance to software virtualized into the instance pool is in most cases trivial. This is most definitely not the case for older proprietary NAS or SAN code. For object stores, SDS makes it possible to scale services such as compression and deduplication easily. It also opens up rich services such as data indexing.

Continue on to get up to speed on object storage and learn how it’s shaking up enterprise storage.

(Image: Kitch Bain/Shutterstock)



Source link