Re: HSM

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Looking at the potential applications of HSM, an implementation based around the object store/radosgw may also be highly desirable.

Spectralogic recently announced their DS3 API which adds bring online/offline operations to S3.

In particular,

- People using S3/SWIFT think in terms of bulk per-file operations rather than POSIX semantics. Updates are a problem for tape based systems. It can be done but recovering from a bad migration of active data is not easy.

- Containers group related files together. The expensive operations on tape are mount and seek. Thus, the aim should be to maximise the number of related files on a tape for ease of recall. When moving files around from one media for another (repack operations), container level grouping ensures that the number of recall mounts is minimised.

- Keeping the tape drives busy js always difficult… tape drives are now regularly exceeding 250MB/s on a single stream so the storage system needs to be able to maintain a high data rate. Tape drive performance drops rapidly when the drives have to stop and then restart as the buffers fill up again.

Tim

On 9 Nov 2013, at 09:33, Sage Weil <sage@xxxxxxxxxxx> wrote:

> The latest Lustre just added HSM support:
> 
> 	http://archive.hpcwire.com/hpcwire/2013-11-06/lustre_scores_business_class_upgrade_with_hsm.html
> 
> Here is a slide deck with some high-level detail:
> 	
> 	https://jira.hpdd.intel.com/secure/attachment/13185/Lustre_HSM_Design.pdf
> 
> Is anyone familiar with the interfaces and requirements of the file system 
> itself?  I don't know much about how these systems are implemented, but I 
> would guess there are relatively lightweight requirements on the fs (ceph 
> mds in our case) to keep track of file state (online or archived 
> elsewhere).  And some hooks to trigger migrations?
> 
> If anyone is interested in this area, I would be happy to help figure out 
> how to integrate things cleanly!
> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux