Syncing is Hard

In episode 23 of Build and Analyze, a 5by5 podcast with Dan Benjamin and Marco Arment. Marco answers a question about syncing and hosting for a mobile app with a web/hosted back-end.

Around minute 23 Marco reads a question from Ryan and starts talking about syncing issues.

“The reason why most apps that could use sync don’t have it, isn’t because hosting is expensive, it’s because syncing is hard. Syncing is really hard, to do syncing properly…”

Marco knows what he’s talking about, his experience creating Instapaper adds great insight.

Syncing is the feature I struggled with building Jottpad. Jottpad is a simple iPhone app, only 5 screens - Login/Register, Lists, Items, Settings, and Sharing. Without syncing this app would have taken around a month, with syncing, about 6 months. I was building the API on the web side & consuming the API for the iPhone app. This is also my first foray into iPhone development & the first time I’ve built this level of syncing logic.

Now would be a good time to point out that while composing this post I discovered writing about why Syncing is Hard… is hard, so grab a beer & keep reading.

Why is Syncing so Hard?

1. 3 Step process
2. Smallest amount of data possible
3. Multiple devices
4. Delete is not as simple as you think
5. Conflicts

3 Step Process
Any sync action requires at least 3 steps. I’ll illustrate with a simple Add List action from the iPhone to Jottpad.com

1. Add List “payload” sent from iPhone to Jottpad.com. Jottpad.com creates List record to match what was added on iPhone. Also creates a DeviceSync record to store the current status of the List for all Device/User combinations.
2. Response from Jottpad.com to iPhone including additional data from Jottpad.com such as Id of List record created on Jottpad.com. iPhone updates with ListId from Jottpad.com & marks a SyncDateTime timestamp on the List.
3. Acknowledge the response (send a SyncAck) to Jottpad.com. This SyncAck closes the loop and assures Jottpad.com is in sync with the iPhone. Jottpad.com marks a SyncDateTime timestamp on the DeviceSync record.

The SyncAck step might seem like overkill but I found it necessary to ensure knowing Jottpad.com & the iPhone were in sync. This step helps facilitate the next step by keeping a snapshot of the current Sync state on the iPhone & Jottpad.com.

Smallest Amount of Data
Speed is a feature. Sync the smallest amount of data & only when needed. This is common sense, but in practice requires a lot more work than syncing everything, every time. Only sending the smallest subset of data on each sync requires tracking the status of all data elements that could potentially need synced. In the case of Jottpad this means each List & Item have individual Sync status per device/user combination the DeviceSync record.

My wife walks into the grocery store and opens Jottpad on her iPhone. This initial syncing when you launch Jottpad I’ll call “Munging” - basically figuring out everything that has changed since the last time you launched Jottpad on the iPhone. The idea here is to only send data from the iPhone to Jottpad.com which has not been synced & only send data from Jottpad.com to the iPhone which has not been synced. This initial Munge, each time you launch Jottpad, is by far the most logic (lines of code) & data intensive (amount of data elements to compare) operation so it needs to be fast, reliable, & most importantly just work.

• In this case everything on the iPhone has been synced to Jottpad.com so the initial Munge is asking for anything that might have changed since the last sync on Jottpad.com

• Jottpad.com gets the Munge request.

  • Check the payload from the iPhone for any changes - in this case nothing.
  • Determine if anything has changed on Jottpad.com that needs synced to this iPhone. Yes - we have a new List - this needs sent back down.

• iPhone gets response from Jottpad.com with info about new List. Do the add and send the SyncAck back to Jottpad.com.

• Jottpad.com receives SyncAck & marks the new List as synced.

This is a simple Munging scenario but the underlying logic is the same regardless - only send the data elements which have been modified, added, or deleted since the last sync.

Multiple Devices
Syncing with one device is easy - you can hack in a couple extra columns on the List & Item records for LastSyncDate and call it a day. When you start syncing with multiple devices the complexity grows quickly. Along with that complexity comes a bunch of potential bugs & edge cases. Again, this seems like common sense, but it bit me in the ass when my first design for syncing fell apart.

Delete
Delete, it seems so easy. Swipe right on the iPhone, tap Delete. I found Delete Sync logic to be the most convoluted of the three Sync actions - Add, Modify, Delete.

Syncing a Delete action is backwards. The problem being once you Delete an Item, it’s gone. How can you sync something that is gone? What about all the other devices that have already synced the Item you just deleted?

Let’s look at another example to try and clarify…

I share a List with my wife - Things We Need. I go to the store, check the list & see we need a new coffee grinder. I buy the coffee grinder and instead of just checking it off the list, I delete. Why? Well I’m pretty sure we won’t be buying another coffee grinder anytime soon so there is no reason to keep this on the list. My iPhone Syncs with Jottpad.com, the coffee grinder is deleted, gone on my iPhone and gone on Jottpad.com.

Meanwhile my wife is also on her way home from work and stops at the store. She opens up Jottpad, sees we need a coffee grinder, buys the coffee grinder and also deletes it. Uh-oh, she gets an error trying to Sync - something something “object reference not set to an instance of an object”. She makes a mental note to tell me about Jottpad having issues, gets in the car and goes home.

Not only do we now have 2 coffee grinders but Jottpad has a serious issue - deleting from my iPhone is not getting synced to her iPhone - what happened & how do we fix this?

Well I started with not actually deleting the Item that was deleted - you put it into a MarkedForDelete status. Think of MarkedForDelete as putting something into the Trash on your computer. It’s still there but not really. Jottpad.com needs to be the master record - I can’t delete from Jottpad.com until all devices that have synced the Item coffee grinder have been synced. At this point the MarkedForDelete Item can be safely deleted.

Conflicts
Conflicts happen when two people Modify the same item before the other person has synced the changes. In other words, syncing a modified item from the iPhone with a modified item on Jottpad.com.

The simple fix is create a ModifiedDateTime timestamp on each record, do a compare on ModifiedDateTime whenever you munge and the newest, most recently modified item wins. Essentially this works, although it can be a little weird if you modify an item, sync, and the item comes back as a different value then what you just modified and synced. This is definitely an edge case and in an ideal world I would have built some sort of merge dialog which would ask the user which value they want to use - but I copped out for Version 1 cause at some point you have to ship the product.

Wrap it up
Syncing is hard, writing about it is hard, and if you’re still reading this you are either:

a. Writing syncing logic and looking for somebody else to share in your pain
b. My wife graciously proof reading this Opus

Jottpad. What is it?

Let’s start with what it’s not - task management. Jottpad is not a task manager. Well at least not yet. If you want a task manager get Omnifocus or Things.

jot
verb (used with object) jot¬ted jot¬ting
1. to write down quickly or briefly

Jottpad is for quickly recording any thought & organizing those thoughts into lists. Or, another way to think about it, creating lists of things that you need to remember or don’t want to forget.

For Example…

Need to Remember

• garlic for the garlic mashed potatoes
• mundane weekly shopping list items such as dog food or garbage bags

Don’t Want to Forget

• that new restaurant which is opening next weekend
• planning a trip to Portland & you just discovered an amazing hotel
• blog idea about “syncing is hard”
• the name of the sweet new app you just read about

How I Use Jottpad

My thoughts are organized into 4 lists…

1. Things We Need - a list I share with my wife for groceries and misc shopping related items
2. Portland - a list I share with another couple, we are planning a week long vacation
3. Blog Ideas - random topics that pop into my head for this blog
4. Remember Me - misc stuff I don’t want to forget and will act on later

Basically, Jottpad is simple, you jott down thoughts & organize however you want. It’s not a full-featured task manager. It’s not a full-featured shopping list. It’s a Jottpad. I’ve kept it simple on purpose. Sure, I’ll be adding more features - but always with the idea to keep Jottpad easy to use & fast.

Started Something

It’s been about 3 months since I started something, www.jottpad.com. Exactly ZERO people signed up aside from myself and the 10 beta testers for my soon to be released iPhone app.

What are we to learn from this…

1. Jottpad is not very compelling in the current form - we live in a mobile app world where all the potential users want this optimized for their mobile device.
2. If the only reason I’m using it on a daily/weekly basis is because I wrote it - we need to bump up the useful quotient a little more.

When the point of Jottpad is to jott down anything you don’t want to forget - the obvious solution to all these problems is make a native, mobile iPhone app. iPhone users always have their phone - it’s the one thing I won’t leave home without. I rarely even walk around the house without the iPhone in my pocket or in my hands.

Alright, let’s make an easy to use, great looking, fast iPhone app that supports all the functionality and syncs with www.jottpad.com. So I did.. I bought my first Mac since college on January 14, 2011 & I got to work. I’m happy to say that with a week or so more of beta testing Jottpad will be ready for submission to the app store.

Few screenshots of Jottpad…

Start Something

I’m a software guy. I like to build stuff and solve problems.

I am lucky to work for a company that gives me the ability to solve hard, interesting problems on a daily basis. However, I regularly wish that the “I work for a company” part was “I created something”. I took an idea and built it from scratch. No I’m not quitting my real job - that would be absurd, but I wanted to scratch an itch.

So… I did. I created jottpad. Yes, it’s another list app. They are everywhere, search the app store on an iOS device and you’ll have a plethora of choices. But this list app is different - it is my list app - I built it. I know how it works, why it works the way it works, and I can morph it into whatever I want. This list app is for me, well me & my wife - it was her idea.

It’s 5:47pm on Thursday. You’re still hard at work in front of your computer and you get a text.

Wife - “I’m on my way home from work & I’m stopping at the store. Do you need anything?”

Me - “Nope, I’m good. See ya soon.”

It’s 7:34am on Friday, next morning. I walk into the bathroom to brush my teeth - no toothpaste.

Guess I needed something.

I go into the kitchen to make breakfast - out of bread & bananas - now how am I going to make a peanut butter & banana sandwich.

Guess I needed something else.

This is not a difficult problem - the day before I knew I was out of toothpaste, bread & bananas but I didn’t record it anywhere. Sure I made a mental note - but my recollection failed me when my wife called and asked if I needed anything.

simplelyst was born. I’ve got an iPhone, when I run out of toothpaste I open simplelyst and add it to the “Stuff I ran out of and need if you go to the store on the way home on Thursday list” - or something like that. You get the idea. You run out of stuff, you immediately add it to a list - no forgetting the next day.

Alright, that is all well and good. Now when my wife calls I can open up my list and read off all the stuff I need her to pickup from the store. Next problem. What if my list is largish - 10+ items. No way she is going to remember all that. Plus I’ve already got this nice list app with checkboxes and everything - but I’ve reverted to reading her a list over the phone. Awwww ha, I’ll email her the list. Well this is better but still no way to cross off/check off - somehow record the action of getting the items on the list.

Wow, this was so much easier when we just wrote stuff down on a piece of paper. Went to the store with our shopping list, and crossed things off as they went in the cart.

Hmm, that’s the behavior we wanted. A physical representation of the list available to either of us whenever we go to the store. Well, we both have iPhones with us at all times & simplelyst.com works on the iPhone. I know, I’m smart, I’ll just share the list I created.

Voila, now my wife can login to simplelyst.com, see all her lists & the list I created. Problem solved. Physical representation of the list.

Umm, so I know what you’re thinking. Rick, you built a list app of which there are hundreds & you added sharing - which isn’t revolutionary. And you wrote a post about it thinking somebody would care.

Well yeah, like I said, this list app is different - it is my list app - I built it. I know how it works, why it works the way it works, and I can morph it into whatever I want. This list app is for me, well me & my wife - it was her idea. Use it if you want. I’ll keep making it better & the 1st step in that direction is creating a native iOS app…