Why Delta Sync Doesn’t Matter
The Hype
One of the common requests we often hear is for us to implement delta sync. Since beginning our implementation, we’ve found that it really doesn’t help as much as many are expecting. Many people expect that delta sync has a large impact on syncing speeds, often times because others in the sync space have heavily advertised this feature. Many have seen demo videos like Dropbox’s where a large image is edited and only the small change made needs to be uploaded. The video says that because of delta sync, over 80% of the bandwidth was saved because of delta sync.
Claims like these are just plain misleading. As the example below shows, the savings a normal user would see is actually 0%. Looking at the types of data we see users synchronizing, the fact is most people won’t see much benefit from delta sync.
What is Delta Sync?
For those unfamiliar with delta sync, it is a technology designed to detect and send only the parts of a file that have changed. If you have a large two megabyte file and change only two bytes in it, rather than re-upload the entire two million byte file, delta sync allows you to send just the two bytes that changed. While not as useful for most small files (the bookkeeping and header information for chunk tracking starts to eat into any benefit), it appears at first glance that delta sync would seem to provide a huge benefit for large files.
A Small Example
The reason delta sync doesn’t help as much with large files is that almost all large file types are stored compressed. Videos, music, digital photos, photoshop files, PDFs, you name it — the files you deal with day-to-day are all stored compressed. Unfortunately, compression negates any benefit from delta sync. When a file is stored compressed, in the process of saving the file, the file is run through a special process that finds duplicate data and removes it. This means any change to the file, no matter how minor, changes the entire file.
So how do we get these claims of bandwidth savings for large files? With a bit of slight of hand and a some contrived circumstances. To use a concrete example, I’ll use the Dropbox demo itself. In it, a picture of a platypus is drawn over with a white X and we see the claim of 80% bandwidth savings due to delta sync. A detail that is somewhat glossed over is the fact that while a file being shown before the edit happens is a JPEG (.JPG – digital image format used by almost all digital cameras), the file actually edited is a bitmap (.BMP – an uncompressed and uncommonly used type).
I took an almost identical graphic and made a similar edit. Using the rsync tool, which is the same rsync algorithm Dropbox uses, I measured the bandwidth savings between edits made on the compressed JPG and uncompressed BMP files. The difference is striking.
For the uncompressed bitmap file, a 65K difference was generated for a 476K file, a 86.4% savings — inline with the demo.
For the compressed JPEG file, a 0K difference was generated for the 76K file, a 0% savings. There were no bytes saved versus uploading the entire file.
While the Dropbox demo doesn’t lie, it also is quite misleading. While the 86.4% savings is nice, it neglects to mention that no normal end-user uses bitmaps to store their images and the bandwidth of 65K required to send just the changes is almost as large as the entire file of 76K when it’s stored in a proper file format.
Try this with a music file, a video, photoshop files, or any other large files and you’ll find that almost none of large files commonly benefit from delta sync.
What Delta Sync is Good For
While delta sync doesn’t help much for most people’s common day to day files, it is incredibly useful for cases in which large files are stored uncompressed. The most typical case is log files for system administrators. These gargantuan text files of things like web server accesses are often hundreds of megs to many gigabytes in size. Data is just appended onto the end as additional activity is logged. This is a perfect case for delta sync.
Relative to the normal user however, use cases like these are the exception rather than the rule. Next time you see delta sync marketed about as a way to save gobs of bandwidth, definitely take those claims with a grain of salt.
How I Tested
If you’d like to try repeating the results for yourself, you can download the files I used here.
I used rsync version 2.6.9 with the following commandline to force a delta sync:
foreach f (*Edited*)
rsync –stats -e ’ssh’ $f localhost:tmp/`echo $f | sed ’s/_Edited//’`
Customer Satisfaction Survey
We recently surveyed our Syncplicity users to see how we are meeting their needs. Based on the survey outcomes, we want to share how we plan to address each of the below issues in our upcoming product development schedule.
How many people does your company employ?
Two thirds of respondents told us they worked in companies of 1-10 employees. Syncplicity is aiming to provide a solid, reliable syncing and collaboration tool for small and medium-sized companies. We’re happy to help small and medium sized team users work more closely together
What would you likely use as an alternative if Syncplicity was no longer available?
This survey result made it very clear who are competitors are. When you know who the competition is, it’s very easy to know where you need to do to be better than the other guys. As Syncplicity’s Support Guru, I’m here to promise you that while other companies have great products, my personal goal is to blow them out of the water with Syncplicity’s customer support. Customer support is just one of the areas we plan to amaze you in, and next week we’ll talk more about how we plan to compete with other companies.
How easy is it to set up and use the Syncplicity service?
Only a third of respondents told us that setting up Syncplicity was extremely easy. While an additional half of respondents told us it was still easy, our goal is to improve these responses – A new online user manual (look for it next week!) is one of our first steps towards better documentation and instructions.
Please indicate the importance you place on the following Syncplicity offerings:
Three items received a “High” ranking in response to this question, and we are happy to see that our user requests are in line with our product roadmap.
- Ability to add extra storage: If you’re using extra storage, we take this as a sign of success – you’re using our product a lot, and we should make it easy for you to continue doing that! We are currently investigating ways to make it unbelievable easy to expand your storage space with just a few clicks.
- Large file support (>2GB): Most files fall well under the 2GB size limitation, but that doesn’t mean we’re giving up on stretching those limits. We are investigating the synchronization of even larger files to make working with Syncplicity appropriate for all your files.
- Centralized user management and administration: I’m pleased to say that the last item is already something we are actively working on. If you are a business user, we’d love to have you join our beta team and give us feedback. You’ll get early access to the new tools, the chance to help us develop this new product, and free service in exchange for your feedback. To join in, email sales@syncplicity.com and tell us a bit about your business and how you might use this tool in your organization.
What would you like to receive from Syncplicity on a regular basis?
Our users let us know that announcements about the product – service messages, new features, and company updates – were important. We’ll improve our blogging and Twittering schedule to keep our users up to date.
Listening to our users is a critical and intentional act in everything we do at Syncplicity. Based on this survey, 75% of respondents would recommend Syncplicity to a friend, so hopefully that means you like what we’re doing. Thank you for everyone who completed the survey!
Mac Client Withdrawal
Today, we’ve made the decision to withdraw our Mac Beta Client. As of today, the Mac Beta Client is no longer available for new users to download. Existing users of the Mac Beta Client can continue to use it until July 31st at which time service will be discontinued.
Syncplicity was founded to provide a rich and reliable synchronization, sharing, collaboration, and backup service. One of the key cornerstones we aim for is an exceptional client experience.
One of the main purposes for a Beta is to help perform full testing and validation in real world scenarios. What has become clear to us is while the Mac Beta Client can run well in many scenarios and computer configurations, there are also many for which it doesn’t work well. Over the past few months we’ve been taking a close look at the unexpected issues that some of you have experienced. Unfortunately what we found is that the fixes entail almost a rewrite in many cases — a process that would take many months to a year.
We want everything we release to be outstanding and something we’re extremely proud to offer our users. It’s clear now the Mac Beta Client isn’t there yet, so we will be withdrawing it for reengineering. We will reassess our schedule for redevelopment next year.
For those of you who have signed up for a yearly account, you will be able to cancel and receive a full refund of your $99 subscription fee. For those who have signed up for a monthly account, you will be able to cancel and receive a refund for the most recent month. Details on how to do so will be e-mailed directly to you in the coming days, but you must request a refund no later than December 31st, 2009.
If you have any questions, we’ve set up a dedicated e-mail for you: macbeta@syncplicity.com.
Obviously like all companies, our intention is for Beta software to always become Final Release software. But it’s just as important to recognize and admit when software doesn’t meet our high quality bar. Our decision to withdraw the Mac Beta Client will allow us to step back to do things much better in the future. While it isn’t an easy decision for us, it is the right decision. We do want to express our apologies for any and all inconveniences this may have caused each of you. We do hope that you give us another chance to deliver future products that meet the level of excellence reflective our existing ones.
Sincerely,
Leonard
–
Leonard Chung, CEO
News from Syncplicity Support
At the end of June, I joined the support team as a Support Guru. This role is a unique challenge for me – no existing dedicated support team, low ticket volume, but technological needs that are detailed and complex. This blog series will follow me as I build the support team and services at Syncplicity to match the needs of our users.
What are your first steps?
One of the questions asked in my interview was “What will you change here in your first 30 days?” My answer? “Nothing.” I plan on spending my first 30 days with Syncplicity learning about our users, investigating the product, and learning about support patterns and current support tools. After that 30 days, I will be able to start moving support in a direction that is in line with the Syncplicity user base and company direction – but I know the learning doesn’t stop at 30 days. It is critical to have a constant presence with the support queue and user base to make sure that we continue to grow with our user base and their needs.
What is your hypothesis for how support will grow?
- Support Ticket Tracking
Support ticketing volume is fairly low at Syncplicity – about 20 tickets a day. However, we do have an active forum and Twitter account. Because the nature of our support interactions require more one-on-one communication, it’s time to move support into a standard tracking system rather than a public forum. To do this, we will use a highly customized version of Salesforce to handle support interactions. We will also have a dedicated support Twitter account to help troubleshoot and announce status updates to our user base. - Support Business Hours
The support team should remain relatively stable for the next few months and 8×5 support should cover our bases. However, after assessing the user base, we may want to offer first 8×7 support (covering international users and east coast users), and then finally offer 24×7 support as our product becomes mission critical in an organization. - Professional Services
Syncplicity’s current user base is very much an early adopter group – which means they like figuring things out themselves, or turning to a community-based support model for assistance. As we acquire more customers, we may offer training, customization, and engagement services to serve the entire breadth of technological knowledge and ability of our users. - Documentation
One of my first tasks at Syncplicity will be to centralize our FAQs and documentation into one user manual that is easy to search, update, and collaborate on with the Syncplicity team.
So, stay tuned as I start my new role at Syncplicity. Watch the company grow along with the support and services team!