Architecting Transparent Cloud Tiering: How to Solve the Unstructured Data Crisis

Download CloudTier Storage Tiering SDK Setup File
Download CloudTier Storage Tiering SDK Zip File

As enterprise data grows at an exponential rate, IT departments face a recurring nightmare: the “Storage Trap.” You buy a high-performance SAN, it fills up with “cold” data (files not accessed in over 60 days), and suddenly you’re back to requesting budget for another expensive hardware expansion.

But what if you could treat the Cloud as an infinite extension of your local disk?

The answer lies in Hierarchical Storage Management (HSM) and Transparent Cloud Tiering. In this post, we’ll look at how to build a solution that moves data to the cloud while keeping it fully accessible to users—without changing a single line of application code.


The Problem: Why “Move and Replace” Fails

Traditional archiving usually involves moving files to a secondary server and leaving a “shortcut” behind. This fails for three reasons:

  1. Broken Links: Applications (like Excel or specialized CAD tools) can’t follow standard Windows shortcuts.

  2. User Friction: Users get frustrated when files aren’t where they left them.

  3. Manual Labor: IT teams spend hours manually identifying “old” data to prune.

The Solution: The Magic of “Stub Files”

The modern approach uses Kernel-Mode File Virtualization. Instead of a shortcut, we use a Stub File.

A stub file looks, feels, and acts like the original file. If you have a 1GB video file, the stub file will show a size of 1GB in Windows Explorer, but its “Size on Disk” will be 0KB.

How the Architecture Works

The secret sauce is a Windows File System Filter Driver. This driver sits between the User Application and the NTFS storage layer.

  1. The Intercept: When a user or app tries to open a stub file, the filter driver intercepts the IRP_MJ_READ request.

  2. The Transparent Recall: The driver realizes the data is missing (it’s a “reparse point”). It triggers the CloudTier SDK to fetch the specific blocks of data needed from Amazon S3 or Azure Blob.

  3. The Delivery: The data is streamed back to the application in real-time. The user never sees a “Downloading” bar; the file simply opens.


Implementing a Policy-Based Strategy

An effective SDK shouldn’t just move data; it should manage it. With the CloudTier SDK, you can automate the lifecycle of your data:

  • Tier by Age: Automatically “stub” any file not accessed in 180 days.

  • Tier by Size: Move only files larger than 50MB to maximize local space.

  • Tier by Type: Keep your active .docx files local, but move .iso and .bak files to cold cloud storage.

Security First: Encryption and Compliance

Moving data to the cloud shouldn’t mean compromising security. A professional SDK ensures that data is encrypted before it leaves your local network. By using AES-256 bit encryption at the driver level, your data remains “dark” to the cloud provider, ensuring compliance with HIPAA, GDPR, and FINRA regulations.


The ROI: Why Build with an SDK?

You could buy a “black box” cloud gateway, but those often come with heavy subscription fees and proprietary formats that lock you in.

By using the EaseFilter CloudTier SDK, you are building a custom, high-performance engine into your existing infrastructure. You get:

  • Total Control: You own the logic and the data format.

  • Performance: Kernel-mode drivers offer lower latency than user-mode applications.

  • Cost Savings: Stop paying for “Tier 1” local storage for “Tier 3” cold data.


Ready to Modernize Your Storage?

The transition from on-premise silos to a hybrid cloud model doesn’t have to be a headache for your users. With transparent tiering, you get the best of both worlds: local speed and cloud scale.

The Code: Creating a Stub File in C#

With the CloudTier SDK, turning a local file into a cloud-resident “stub” takes only a few lines of code. Here is how you initialize the filter and create a reparse point (stub) that points to an external storage location:


using EaseFilter.CloudTier;

// 1. Initialize the CloudTier Config
CloudTierConfig config = new CloudTierConfig();
config.ConnectionKey = "Your-Access-Key";
config.ProviderType = CloudProviderType.AmazonS3;

// 2. Define the local file and its remote "Cloud" destination
string localPath = @"C:\Data\HugeVideo.mp4";
string remotePath = "s3://my-archive-bucket/HugeVideo.mp4";

// 3. Convert the file to a Stub (Reparse Point)
// This keeps the metadata local but offloads the 1GB payload to S3
bool success = CloudTierApi.CreateStubFile(localPath, remotePath, config);

if (success) {
    Console.WriteLine("File tiered successfully. Local storage reclaimed.");
}