Cloud Xtender – some detailed info!

Ok, since releasing our last blog post, we have had a flood of emails regarding some of the features seen in the demo video. So I though I would take everyone through some of the screens seen in the demo, and detail what each feature does. I should also note that this post was written a while ago, and I’ve only just got around to publishing it… so some info may be a little out dated.

First we have the connection tab, this basically allows you to connect to a specific cloud service.

provider_screen

For the most part, the options here are provider specific, except :-

1) Cloud connection: This is the connection (or provider) that will be used for this connection.

2) Connection name: Simply a name you give this specific connection instance.

The next tab is the placement rule tab. This is where you can define how Cloud Xtender places files on the cloud services you have added.

rules_screen

The key options are :-

1)  Placement rule name: Simply a name you give this specific rule.

2) Placement selection: This determines how Cloud Xtender distributes files between the different connections. There are currently 4 options :-

  • Single connection: This simply writes all files to the specified provider (or when * is specified, the first provider)
  • Multiple connections, cascade (default mode): This writes to the first connection in the list (if * is specified in the list, then the order is based on when the connections were added). Once that connection’s usage reaches a pre-set level, the next connection is written too… and so on. It should be noted that some cloud services do not have a fixed storage size (aka Amazon S3 etc), in these cases Cloud Xtender allow you to set an arbitrary fixed size.
  • Multiple connections, free space: This writes to the connection with the most free space.
  • Multiple connections, percentage of free space: This writes to the connection with the highest percentage of free space.

3) Cloud connection: Here you can add/remove connections that are to be used by the placement selection (previous option). You can specific individual connections, and the order, or you can use a wildcard that will use all existing, and any new connections added in the future.

4) Connection sync method: This allow you to define how files are sync’d against the connection(s). The options are :-

  • Mirror: In this mode, Cloud Xtender monitors the local file and syncs any changes to the cloud connection. In the mode there is a local and remote file maintained, although any changes made against the remote files will not be detected.
  • Remote: In this mode, Cloud Xtender maintains a ghost image of the remote file, so there is no local file (Cloud Xtender caches some portion of the file for a period). If this file needs to be modified, then you can request this file to be pulled down (see the Mirror -> Remote mode) via File/Folder management. Simply find the file, right click and select “Cache locally for a limited time”
  • Bidirectional: In this mode, Cloud Xtender monitors the local file and remote files, and syncs between the local machine and the cloud connection. In the mode Cloud Xtender needs to monitor the cloud connect, and as such this mode can impose a network load (the frequency of the checking is determined by the next option).
  • Mirror -> Remote: In this mode, a files starts out in Mirror mode, then after a period, it is changed to Remote mode.

The final tab is the files/folder management tab. From here you are able to assign specific folders to specific placement rules, query a files to check it’s sync status, resync the file (in either direction) and you can even request a file to be cached locally for a period (for files sync’d using the backup mode). I should note that all this functionality is also available directly from Windows Explorer.

file_folder_mgr

Finally… we are close to the beta release. Our intended date slipped (surprise surprise) because of a number of bugs in one of the providers, however we have now got on top of these and are close to a beta release.

Oh… and we have unified our Twitter accounts, to get the latest news on Cloud Xtender, PoolHD or Drive Bender, follow us at @_divisionm_

Are we there yet?

We have been receiving heaps of mail regarding the Cloud Xtender beta release date, the continuation of Drive Bender v2 development and PoolHD v2. So I thought it best to touch base and give everyone the run down.

Cloud Xtender
The Cloud Xtender beta has been our core focus over the past few months, the interest in this product has been overwhelming… so we have been hammering away in an effort to get it out the door. We have a working version running in-house, and are now just fine tuning how Cloud Xtender interacts with different cloud providers. At this stage we are looking at mid November as the first beta release… and providing there are no show stoppers, the final release will happen shortly thereafter. Although we have cloud storage providers for AWS (Amazon) S3, Azure, DropBox, Google Drive and OneDrive (formerly SkyDrive) we are planning on having only AWS S3 and Google Drive in the first beta. Reason being is that we really want to get the first beta out the door… as it stands we are well behind our initial release schedule (hmm who would have thought!), so in an effort to get the first beta out, we have decided to exclude the other providers for the first beta. To give taste of what is to come, I have posted a demo video of an earlier build, this shows the interface, and how we create and add cloud drives etc. You can check it out here.

Drive Bender v2
This leads me into Drive Bender v2’s cloud duplication support. We are planning to continue with the v2 development when Cloud Xtender is sorted, mainly because the cloud duplication feature uses much of the Cloud Xtender code. I will also mention VSS, we are getting a small number of users asking what happened to this feature in v2. As a refresher, here’s what I wrote about this in a blog post last year…

VSS Support
I thought I would touch on this much requested feature first up. For those not familiar with VSS (aka shadow coping), let me point you to http://en.wikipedia.org/wiki/Shadow_Copy. With v2 we have implemented VSS support and have this under internal testing. We are able to snapshot a pool, and access these individual snapshots independent of the pool itself. This all works great, however one of the key requirements is external application support for doing this, known as requestor support (the requestor is the calling application such as Windows Backup). Again we have this working, however during testing we have run up against a number of issues. The most notable is reliability, for some reason the VSS requestor (i.e. Windows Backup), cannot always successfully snapshot the pool, and the entire backup fails with an error. Also the success of this seems to be very machine dependent, and we have no idea why… unfortunately good technical documentation on VSS very limited to non-existent. I should also mention that the internet is filled with stories of VSS failures… this is one finicky beast!

So what does this mean for VSS support? Well our original timeline had full VSS support in beta 1, and this was meant to be available mid August, however given the issues we have faced, this has not happened (obviously). At this stage we have decided to move on with the other stages of the v2 development and beta releases, while we continue to work on these VSS issues in the background. At the very least v2 will support pool snapshots, snapshot mounting and snapshot management. If requestor support continues to allude us, then we have a plan B (I will post on plan B when we come to a final decision).

We have revisited VSS a number of times since, only to run into the same lack of reliability… however we have not given up, and will keep plugging away when time permits.

PoolHD v2
Version 2 of PoolHD is also set to be released very soon. PoolHD v2, is based heavily on the Drive Bender v2 code (as v1 was), which is why we have been delaying… we had been chasing some odd permission issues in Drive Bender versions prior to v2.1.9.0, so we didn’t want to move forward with PoolHD v2 until these were sorted. Stay tuned!

Cloud Xtender, what, when, where and why?

Well, we are getting close to our first Cloud Xtender beta release. We have had a lot of interest in Cloud Xtender, and have received many questions about the product features and when it will be released. So I thought I would throw up a quick post to answer some of these questions. First up the features… well, while I do not want to give to much away, what I will do is lay out some of the prominent features and give a basic overview of how Cloud Xtender works.

Cloud Xtender allows a users to create a local drive on their machine that points to one or more cloud drives. The number and type of cloud providers is not limited, and you specify rules that determine file placement. For example, a user could create a drive that points to 10 different free Google drives accounts, giving a total of 150 GB of free cloud space (of course you can add more accounts later). The user then specifies a rule that define where files are stored, which might be either an “even” rule (cloud drives are filled evenly), or “cascading” rule (cloud drives are filled one after the other). Now let’s say the same user want to store a photo on an Amazon S3 Glacier drive, he then creates folder in his Cloud Xtender drive, then creates a rule for that folder that points directly to the S3 drive!

One of the most commonly asked questions about this is “Are the files stored locally and sync’d to the cloud, or are they just in the cloud”. Well the short answer is, you can have either. Files can be a) stored in the cloud only (no local copy), b) stored locally and sync’d bidirectionally. One of the key issues with cloud only storage is performance, parsing the cloud every time you browse the file system is a performance killer, we have solved this with some very clever caching technology. Browsing a Cloud Xtender file system is as fast as browsing your other local drives.

And finally, the question that is most often asked, when is this going to be ready? Well soon is the plan… we have most of the technology in place now, as a lot of the Cloud Xtender code is shared with Drive Bender’s new cloud duplication technology. From here we will be releasing a beta, taking feedback and suggestions, then moving pretty quickly on to a release.

Finally I just want to mention the Drive Bender v2 release… so far, so good. We have had some excellent feedback (everyone seems to love the new UI), and we must have done something right because sales are the best they have been this year! There are a couple of minor issues we are sorting, and plan to post an update in the next day or so.

Drive Bender v2 ER1 release date, and Cloud Xtender news

We now have a date for the release of Drive Bender v2 ER1. Providing all goes to plan (i.e. there are no show stopping bugs), we will be releasing the official version of ER1 on Friday the 30th of May. If you have not purchased a license at the discounted rate, you need to do so now as this discount will end the moment we post the new version.

Further exciting news… Cloud Xtender, our pooling product for the cloud, is set to see its first beta very soon. This beta will support a number of cloud services including Amazon S3, Dropbox, Google Drive, Microsoft Azure and SkyDrive.  If you want to be a part of the official Cloud Xtender beta team, send a support request via http://support.division-m.com. Make sure you select Cloud Xtender as the product and the request type to beta. If you want to be kept upto date with Cloud Xtender development, make sure you follow Cloud Xtender on Twitter https://twitter.com/cloudxtender

Drive Bender ER1

We are moving ever closer to the release of Drive Bender ER1. That said, we are getting a lot of questions around this ER1 release, and what this means for the full featured Drive Bender v2 release.

Well for those that may not already know, ER1 stands for “Early Release 1”, and is basically a production release of the new v2 code, but with some of the features yet to be added.  At the start of 2014 we looked at our timelines for the Drive Bender v2 release, and the number of fixes and updates made to the v2 code over and above v1… in view of this, it was decided that we should feature lock the v2 code, test and refine what was in place to the point that we could release a v2 version that would replace the current v1 release.

Why… why not wait until all features are in place? Well there are two main reasons, 1) The feedback on the new interface has been incredibly strong, and we wanted to get this out to our users ASAP, and 2) The new code base offers much more in the way of speed and reliability over v1, not to mention a the few niggling v1 bugs had been quashed!

So whats next? We are in the final stages of ER1 testing and bug fixing, and we now have a great platform to bolt on the balance of the v2 features. Here’s a list on the new features coming in ER1.

Drive Bender v2 ER1 (release soon)

New user interface
Drive Bender v2 brings with it an entirely new interface, this is to improve usability and overall interface performance. With this we are also improving the notification engine and introducing client notification popups.
dark_server_2012
T
he new Server 2012 R2 server addin interface

dark_win_81
T
he new Windows interface

Under the hood
Under the hood Drive Bender v2 is a vastly different beast, we have re-architect most of the core application, the result is a much faster, robust and scalable platform. Features such as NFS support, pool GPT partitioning and better handling of drive disconnection are just some of the new features of this new code.

File system health monitor improvements
The performance of the file system health monitor has been improved, along with improved duplication issue detection (such as repeated failure of duplication creation etc) and general drive health.

Network drive support
This new feature is straight forward, you can now add a network path to the pool, and it is treated just like a physical hard drive.

Landing zone support
This is the ability to set a specific drive to be used for all writes to the pool, then at a later time, these files are to be moved from the landing zone drive to the rest of the drives in the pool. The idea here is that a fast drive can be set as the landing zone to improve overall write speeds to the pool.

Changes to file balancing (more to come)
We have improved the “even balancing” method and have add limited “cascade balancing” support (if you disable balancing, Drive Bender now reverts to cascade balancing).

Post ER1 release

Obviously the are a number of major v2 features still to come (you can see a list of v2 features here), and we’ll be working on these as fast as our little fingers will carry us. We are hoping to delivery a number of these in the first post ER1 beta. Finally I can hear you asking “what is happening with VSS support”? Well this is proving to be quite the challenge to say the least. The VSS framework put in place by MS is, well, to be kind, poorly documented and mostly held together with gaffer tape… however our intention is to deliver this after v2 ships providing we can get it to work reliably.

Finally I’d like to make a shout out to the users that have helped with beta testing, your feedback has been invaluable, thanks!

The road ahead in 2014

Well 2014 has arrived, and straight out of the gate we have been flat out. Over the Xmas and new years period, development has been forging ahead with the first beta of the year being released a week or so ago. On the surface there may not look to be many changes, other than the manager and server addin, however under the hood approximately 80% of the entire code base has been either updated, or entirely rewritten… our experience with v1 over the past couple of years has given us a good insight into what we need to change for v2, and what we need in place for future development.

While talking about v2, we have been receiving many requests regarding the release date, while I would love to cement a date, the world of software development makes hard deadlines difficult to hit (for those wanting to know how hard, read a book called “Dreaming in Code”). However there are many fixes, updates and additions to the current v2 beta, so we are planning on releasing “Drive Bender v2 ER1”. ER1 (standing for Early Release 1), is a production ready snapshot of v2 that will become our release version while the rest of the v2 features are being bolted on. I should also make mention of Cloud Xtender, this is also under development and we are hoping to get a beta out when Drive Bender v2 is close to being sort.

In other exciting news, we have just acquired another pooling product called PoolHD (www.poolhd.com). For those not aware of PoolHD, it is a pooling product very similar to Drive Bender, except it’s target market is more desktop focused. I can hear the punters asking “why do you want another pooling product”? Well under normal circumstances, this wouldn’t be high on our list of things to do, however PoolHD is close to our heart… the app started life as a custom solution for a company to pool proprietary file systems and databases, from here they modified it was branded it as PoolHD. However over the past 12 months, this proved to be difficult for them to maintain… so we landed it for the right price!

So what happens next? We will be working on getting v2 ER1 out the door, and continuing with the v2 final. With regards to PoolHD, because it shares a large portion of its code base with Drive Bender, we will be releasing PoolHD v2 soon thereafter… onward and upwards!

Drive Bender beta 2 update

It has been a busy time since the first beta was released, thanks to the testers in the interactive beta program, many bugs have been identified and sorted, not to mention some great suggestions on interface improvements, thanks all!.

Originally I had hoped to have the second beta out by now, however it was decided to release an interim version with many of the fixes and changes as reported, and to also include our next generation of Drive Bender driver. All this and the fact that a number of non Drive Bender related interruptions have happened, has meant we are not where we would like to be with regards to milestones… who would have thought such a thing could happen in software development!

Anyway we are finally back on track and are working towards an updated beta which should be available some time this week (week of the 18th).

Finally, again a big thanks to those that have been testing.

Drive Bender v2 beta 1… what’s it all about?

Well after a feverish development cycle, we got the first beta out, and we are very happy with the results! There are a bunch of changes, with many of them being “under the hood” so to speak, however the most obvious is the new client interface. Earlier this year we prototype’d a new “compact” interface (reference – http://forum.division-m.com/index.php?topic=1520.msg5074#msg5074), this was the basis of our re-design. The big change is that the old tabbed interface is gone, the v1 interface used a number of WPF controls, which while looking cool, could really bog down the loading times… in v2 there is now only a single view that is much nippier.

The new Drive Bender v2 interface under Windows

The new Drive Bender v2 interface under Windows

This new interface has also found its way to the WHS addin, doing away with the tab interface has also improved the load time of the dashboard.

Drive Bender under WHS 2011

Drive Bender under WHS 2011

This beta release also bring with it the ability to add networked drives to a pool, so you can effectively add any drive or device that you can reference using a UNC address.

Adding a networked drive to the pool

Adding a networked drive to the pool

Adding the networked drive is very straight forward, enter the UNC detail and then perform a test of the target address, if the test passes you can add the drive to the pool.

Finally the plan was to release the new “Landing Zone” feature with beta 1, however this was pulled at the last minute… so we will be releasing beta 1.1 in coming days with this new feature enabled.

If you would like to check out the beta for yourself, you can download it from http://www.drivebender.com/drive-bender-beta-downloads-sn

Stay tuned, there is heaps more to come!

Drive Bender v2 beta program

Over the years we have run an open beta program for each Drive Bender release… this has, and has not worked. Let me explain.

Drive Bender is a product that deals with one of your most precious resources, your data! To install a beta product to handle said files is a tough ask, and we understand this. However there are a number of users that have suitable redundancy, or specialized environment(s) that allow them to test beta versions of Drive Bender, and our plan is to harness and focus on these users during the beta development phase. So we have come up with the “interactive beta program”. The idea being that there will be a limited number of users that will have a dedicated forum, and more direct access to the lead developer on the project… err, me! So how do you get in?
1) Well first and foremost you need to ensure you are not installing the beta on the machine that contains the only copies of your family photos that go back 100 years! FYI – We do not release anything that we do not consider to be “file safe”, however this is still beta code.
2) Next you need to be happy to test specific areas of functionality and provide feedback.
3) Finally you need to be a little technical, although each beta release will come via an installer, there may be instances where we need to upgrade one or two components manually. This will simply involve copying files to a specific folder and starting the Drive Bender service.

Additionally we are encouraging these users to provide feedback on features and usability during the beta phase.
If you are keen, log a support ticket at http://support.drivebender.com and ensure you select “DBv2 beta programs” from the type of request list, also include the operating system you intend to test upon, and your full name in the description field.

As I mentioned, this is going to be limited to a small number of users, however for those that miss out or are not interested in taking part, don’t worry, the beta will still be made available to all users regardless.

So what is the release schedule? Beta 1 is set to be released on the 8nd of October and will have the following new features
1) A number of major internal upgrades with regards to file duplication and file system health monitoring.
2) The ability to add networked (UNC) drives into the pool.
3) Phase 1 of the new interface.
4) Landing zone, the ability to specify a drive that files get written to, then redistributed around the pool at a later time (this has been added after a number of requests). Note because of the late addition of this feature, it may not make it into beta 1.

Drive Bender v2

Well we are hard at work on Drive Bender v2, and are excited with the progress we have made to date. Our plan is to roll out a number of staged releases (i.e. v2.0, v2.1 etc) with beta milestones in between… so lets take a look at what is in the works.

VSS Support
I thought I would touch on this much requested feature first up. For those not familiar with VSS (aka shadow coping), let me point you to http://en.wikipedia.org/wiki/Shadow_Copy. With v2 we have implemented VSS support and have this under internal testing. We are able to snapshot a pool, and access these individual snapshots independent of the pool itself. This all works great, however one of the key requirements is external application support for doing this, known as requestor support (the requestor is the calling application such as Windows Backup). Again we have this working, however during testing we have run up against a number of issues. The most notable is reliability, for some reason the VSS requestor (i.e. Windows Backup), cannot always successfully snapshot the pool, and the entire backup fails with an error. Also the success of this seems to be very machine dependent, and we have no idea why… unfortunately good technical documentation on VSS very limited to non-existent. I should also mention that the internet is filled with stories of VSS failures… this is one finicky beast!

So what does this mean for VSS support? Well our original timeline had full VSS support in beta 1, and this was meant to be available mid August, however given the issues we have faced, this has not happened (obviously). At this stage we have decided to move on with the other stages of the v2 development and beta releases, while we continue to work on these VSS issues in the background. At the very least v2 will support pool snapshots, snapshot mounting and snapshot management. If requestor support continues to allude us, then we have a plan B (I will post on plan B when we come to a final decision).

New user interface
Drive Bender v2 will bring with it an entirely new interface, this is to improve usability and overall interface performance. With this we are also improving the notification engine and introducing client notification popups.

Network drive support
This new feature is straight forward, you can now add a network path to the pool, and it is treated just like a physical hard drive. With the addition of the “Drive specific duplication” feature (see below), you can specify if this drive is to be used for primary files, duplicate files or both primary and duplicate file storage.

Landing zone support (feature added after user demand)
This is the ability to set a specific drive to be used for all writes to the pool, then at a later time, these files are to be moved from the landing zone drive to the rest of the drives in the pool. The idea here is that a fast drive can be set as the landing zone to improve overall write speeds to the pool.

Pool to Pool replication
File replication (of primary files) between two or more pools on the same network is now supported. You simply point the Drive Bender replicator to another Drive Bender pool, and all files will then be mirrored between these pools.

Drive specific duplication
This feature allows a user to specify which drive or drives are to be used for primary file storage, or duplicate file storage. In addition we have introduced “lazy duplication”, which optional allows a file to be duplicated after the fact (as opposed to the current real time duplication).

Cloud specific duplication
Files duplicated in a pool can now be stored in the cloud using any number of cloud providers. Currently we support Google Drive, SkyDrive and DropBox (with more to come), all of which can be implemented on the same pool. The duplicated files is move to the cloud in an asynchronous manner (using the new “lazy duplication” feature) to ensure pool storage performance is not compromised. We are also looking at implementing file encryption for duplicate files moved to the cloud to ensure “End to End” file security.

File system health monitor improvements
The performance of the file system health monitor has been improved. We are also improving duplication issue detection (such as repeated failure of duplication creation etc) and general drive health.

Hard drive performance monitor
One of the key support issues we face is a pool stalling or locking up for extended periods. This is almost always the result of a hard drive either failing, or suffering some file system structural issue. The new Hard drive performance monitor will supervise and report on any drive not meeting the required performance targets.

Enhanced file balancing options
Yet another much asked for feature, we are improving the existing balancing option, as well as adding a number of new file balancing options.
Even Balancing: File are balanced evenly across all drives in the pool. This is very similar to the v1.x balancing feature, however we are making a number of changes to improved this modes performance.
Folder level balancing: With this option, files are balanced based on their parent folder to ensure that a specific folder’s contents are not scattered across a number of drives.
Cascade balancing: This option basically fills a single drive at a time, when the drive becomes full, files are stored on the next drive and so on.

Multilingual support
Yes that’s right, we are finally going multilingual! To start with we will be supporting the following languages – English, German, French and Spanish.

Under the hood
Under the hood Drive Bender v2 is a vastly different beast, we have been re-architecting most of the core application for some time, the result is a much faster, robust and scalable platform.

Whats next?
The next step is to get beta 1 out the door, all going well this should happen in late September. Lets us know what you think, and feel free to comment either here or on the forum.
I should point out that none of these features are “carved in stone” so to speak, and are subject to change… either via user requests, or we just change our minds ;-).