Design Issue: Changing Cloud Storage Accounts With The SMSyncServer

4/2/16

Introduction
Working on the SMSyncServer, I just came across a senario I hadn’t anticipated. So, I’m going to write about it to see if I can work it out. Therapy for software design!

The design of SMSyncServer hinges on end-users signing in to their cloud storage system. And that cloud storage system is used to redundantly store the mobile app’s data files, and used to sync those data files across devices. From the app user’s perspective, they sign into their cloud storage system (e.g., their Google account, storing files in Google Drive), and then the app that uses the client portion of SMSyncServer uploads the app’s files to the sync server (and downloads files that have been changed by other instances of the app running on other devices, signed in with the same cloud storage credentials).

When the app using the SMSyncServer client uploads a file it provides an identifier, a UUID, for the file. The UUID is what enables the app and the SMSyncServer to communicate about particular files. The SMSyncServer client stores that UUID and a file version number internally to the client. The version number is important because SMSyncServer needs a way to determine if files on the server have changed. I.e., if another app has uploaded an updated version of the same file to the server.

Now consider the following situation. Suppose that an app has 10 files, each identified by UUID’s and, internally to the client, version numbers for the files. The app user signs in to their cloud storage account, and the SMSyncServer client uploads their files to the cloud storage system, and stores meta data for those files in the sync server. Next, suppose that the app user decides that they want to sign out of their cloud storage account. This is the start of the scenario I’d not thought about before.

At the moment the user signs out of their cloud storage account, the link between the SMSyncServer client and the cloud repository of their files is broken. In some ways, this is not a big deal. But presumably the SMSyncServer client (and app using that client) should deal with this gracefully. The app shouldn’t just fail. The user should still have access to their local files on the device, but of course, the files will not be synced across devices any more. There is a question here about how or if the app should continue to interact with the SMSyncServer client. For example, suppose that the user changes a file. Should that file update be recorded with the SMSyncServer client despite the fact that it cannot upload the changes to the server? Keep this issue, about whether file updates should continue to be recorded, in mind. We’ll refer to it as (*) and come back to it momentarily. Note that unless we do something differently, the client itself still knows about the UUID’s of prior uploaded files and their versions when the user was signed into their cloud storage account.

There are two main cases we need to be concerned about after the user signs out of their cloud storage account:

1) What should happen if they now sign into the same cloud storage account?

2) What should happen if they now sign into a different cloud storage account?

Case 1: User signs into same cloud storage account
Referencing back to (*) above, suppose that the app had indeed continued to register updates to files with the SMSyncServer client. This would then be similar to the already known situation where the device just didn’t have a network connection for a period of time. Not exactly the same, but similar. It’s not exactly the same because we wouldn’t have information on the client about the server account being used by the app. i.e., the user wouldn’t be signed in to a cloud storage account. Once the user signs back into the same cloud storage account, the changes that have thus been recorded can be uploaded. And normal operation of the SMSyncServer can resume. Of course, we’d have to keep a record of the prior cloud storage account used, for comparison purposes—so we could know that the user has signed back into the same account.

Case 2: User signs into different cloud storage account
If the user now signs into a different cloud storage system, what should happen? This is more complicated. Remember those 10 files we were talking about before, that the app user had created? The SMSyncServer client had a record of those 10 files in its internal meta data. Suppose for the moment that the app user hadn’t made any changes to those files in the interval after signing out of their original cloud storage account, and before signing into their other cloud storage account. Well, the file upload design of the SMSyncServer client works by only uploading files that have changed. Since no files have changed, the SMSyncServer client would not upload any files to the server. This, however, is a problem—it results in files being out of sync from the device to the server—we have no reason to believe that the server, with respect to this new cloud storage account, would have any of these 10 files stored. Files in the server meta data (the “FileIndex”) are identified with respect to a user by a userId. The userId directly corresponds to a cloud storage account. The purpose of the FileIndex meta data is to reference file contents stored in cloud storage.

The problem seems to come down to: How do we cause the SMSyncServer client to upload the relevant files to the server? Of, course this is going to cause some work to be done. We’re talking about an example with 10 files, but those could be large files. Or it could be 100 files or 1000 files. This can take time. But, still, the principle is syncing files from the device to the server. This raises a point that when signing into (or out of) a cloud storage account: The user needs to be carefully informed or warned about the consequences of that action—that the action can result in lots of network activity and time consumed. And network activity, especially when taking place over a cellular network, can have real monetary costs.

How do we do this? How do we cause the SMSyncServer client to upload the relevant files to the server? This is more complicated than it might seem. The client itself, while maintaining UUID references to files, doesn’t maintain file system references to app files on the local device. This can be clearly seen in the SMSyncServer API methods that deal with NSData “files” (uploadData) and temporary files (uploadTemporaryFile). The data for neither of these persist in the local file system after the upload is done. And even the “Immutable” file upload method (uploadImmutableFile) only assumes that the file doesn’t change until after the upload is done.

Traversal and reupload of files
So, in terms of the SMSyncServer client, this situation needs to be handled by the app using the SMSyncServer client. That is, the app needs to do a traversal of all of its files and re-upload those again to this new cloud storage account using the SMSyncServer client. This traversal wouldn’t necessarily be a new aspect to an app though. If an app started using SMSyncServer in a new version after the user had already had some data present, then such traversal functionality would already be needed. Relatedly, what should happen to the SMSyncServer client meta data along with this traversal and re-upload? Should the client meta data be flushed? Should it be retained?

Key to this last question is versioning of files. The client meta data keeps track of the versions of files by UUID. If there are multiple devices synced with the same data, and the client meta data is flushed, then in order to keep file versions in sync, the meta data will need to be flushed on those other devices, and the file and meta data will need to be re-synced across the network with those other devices. A potentially time consuming and generally expensive operation! Another way to handle this would be, for this traversal and re-upload occurring due to changing the user cloud storage account, to upload using the same file versions as previously present in the client meta data. This would require some changes to the SMSyncServer client interface to change its operation depending on whether the upload operation was occurring due to a regular upload or to a re-upload because of the user changing their cloud storage account. If the operation was a regular upload, e.g., the 2nd upload of an existing UUID, then the file version would be incremented. If the operation was a re-upload, then the file version from the current client meta data would be used.

There are two more tricky parts to be concerned about. First, remember that previous cloud storage account that the user signed out of? Well, the 10 files that we were talking about are still referenced in the FileIndex in the server for that previous cloud storage account. The FileIndex with regard to those 10 files and that previous cloud storage account needs to first be flushed before uploading occurs—those are effectively stale references at this point. This would not, of course, change the actual file contents in that previous cloud storage— by the fact of the app user signing out of that cloud storage account the SMSyncServer no longer has permission by the user to deal with those files.

Second, the user of the app could be signed in to the original cloud storage account from multiple devices. Indeed, that is a main intent of the SMSyncServer—to sync data across user devices. The appropriate administrative behavior for a user would be to sign-out of all of those devices, and sign back in on all of the devices with the new cloud storage account. But how can we be assured that does happen? Will clearing all of the file meta data from the server index be sufficient?

Conclusions & Discussion
It seems like we need an identifier for a user beyond that of an individual cloud storage account. Call this new identifier a User App Identifier (UAI). This UAI would persist across the user signing into and out of a cloud storage account, and would refer to one collection of files for a specific app that is possibly shared (and synced) across devices. For example, this could be the collection of pet health info data for the pets of a family stored by an app such as Petunia. The UAI would be created the first time a user signs in with a particular cloud storage account, when they had not previously been assigned a UAI. When a user signs out of a specific cloud storage account, and then signs into a new cloud storage account, this is taken as a request to flush their old FileIndex information on the server, and to resync their data from the device to the cloud, but their UAI would remain the same. Requests to the server from other devices to use the old cloud storage account for that UAI would be refused. On those other devices, the user would need to sign in with the new cloud storage account. Thus, on the server, UAI’s would be related to the currently active cloud storage account being used by the UAI.

It should be noted that in Case 2, where the user signs into a different cloud storage account, I’ve avoided consideration of copying data, on the server, from one cloud storage system to the other storage system. Two factors led to this avoidance: 1) There might be a very good reason that user is signing out. For example, perhaps the user doesn’t have access to that data any more. 2) This style of potentially long-running server operation is somewhat at odds with the design of the SMSyncServer so far. With a couple of exceptions in the SMSyncServer, server operations are relatively short running. If implemented as a server copy operation, this would certainly be the longest running server-side operation possible on the SMSyncServer.

Overall, this cloud storage account changing was far more complicated than I thought it was going to be. All that comes from pulling a thread on an unanticipated scenario! I’m going to delay implementation of this ability to change cloud storage accounts until after the SMSyncServer is feature complete. This functionality is not a small change!

About the author: Christopher G. Prince has his B.Sc. in computer science (University of Victoria, B.C., Canada), an M.A. in animal psychology (University of Hawaii, Manoa), an M.S. in computer science (University of Louisiana, Lafayette, USA), and a Ph.D. in computer science (University of Louisiana, Lafayette, USA). His M.S. and Ph.D., while officially in computer science, were unofficially in cognitive science, split between animal psychology and computer science. Chris is a dedicated animal person, and has also developed: Catsy Caty Toy, a customizable and shareable iPhone and iPad app for your cats (http://GetCatsy.com)Petunia, an app for recording and sharing pet health information (http://GetPetunia.com), and WhatDidILike, an iPhone app to keep track of restaurants and food that you like (http://WhatDidILike.com).

Creative Commons License
“Design Issue: Changing Cloud Storage Accounts With The SMSyncServer” by Christopher G. Prince is licensed under a Creative Commons Attribution 4.0 International License. Permissions beyond the scope of this license may be available at chris@SpasticMuffin.biz.