Apps that Sync

Over the past week or so I’ve been playing with adding real-time sync to One to Watch using TouchDB, syncing via Cloudant (where I now work). The idea is two fold:

	- All your devices will be in sync. The same list of films, the same ratings, the whole caboodle. At the moment this is within a few seconds, but I'm wondering how much this drains the battery (TouchDB holds open a connection to Cloudant to receive notification of changes). It might be best to do periodic sync rather than always-on.
	- I'd like to add a web app once the sync works well, talking to the Cloudant database.

Cloudant’s service is based on CouchDB. CouchDB is a document database built from the ground up with disconnected operation in mind. It has a robust protocol for transferring data between databases, including getting two or more databases into an identical state. It’s obvious this should be a great fit for keeping mobile devices in sync with each other. Even better, the protocol is designed around the assumption that all devices are peers, and so doesn’t require a central point for a “master” copy of the data. Think git rather than svn.

For mobile devices, however, a central mediator often makes sense due to the constantly changing network conditions, addresses and so on a device experiences. Part of my work at Cloudant revolves around making it as easy as possible for developers to make use of this power within the constraints of a mobile gadget.

This is where TouchDB comes in. TouchDB is an Apache-licensed project that was started with the goal of creating a mobile-device native datastore able to speak the CouchDB replication protocol, and so perform as a peer in synchronising data between a mobile device and a remote CouchDB database. There are two main projects, one for Android and one for iOS. The iOS project is more advanced, fortunately for my needs.

I’ve only just started my work with TouchDB. My first baby steps are learning the ins and outs of the library through the challenge of porting my own application over to using it. Fortunately One to Watch uses a home-brewed key-value store so the data model was already document-based and easy to bring into TouchDB land.

As I wanted to become more than cursorily familiar with TouchDB’s innards, I decided to work with the low-level database classes rather than the better documented CouchCocoa interface, which mimics CouchDB’s. It turns out there’s quite a bit of assumed knowledge — understandably — down in the guts of the implementation. Still, after a weeks teasing apart of the necessary from the unneeded, I’ve got to a stage where I’ve two devices and a simulator exchanging data with a master database sat on Cloudant. There’s something magical about watching changes leap from one device to the another, seemingly through the air.

While the databases seem pretty robust at keeping in sync with each other, my UI still needs a lot of rework to be able to respond to arbitrary changes appearing in the database that are not a result of a user’s actions. There’s still along way to go:

	- Make sure all the relevant views watch for changes.
	- Figure out what to do when the user makes changes to the same film on two devices --- CouchDB helps with this, but it's still up to the app developer to merge changes in this scenario.
	- Build out a server side mediator to give devices access to a user's film database.
	- Think about how and if to build a web app to manage films.
	- Wring my hands about how to pay to keep such a service online.

It’s all pretty exciting, and hopefully I can find the time to work through the whole thing from start to finish. In general, all this experience will feed directly into my work for Cloudant, making the finished system pretty awesome. Well, that’s what I hope. And then I just have to get on the road to tell everyone about it.

I’ve been fascinated with the idea of all devices feeling like views onto the same datastore for over a decade — since my first Psion mobile computer — so being able to hopefully make it much easier through putting work into both Cloudant’s service and device libraries should be pretty fun.