Option for performing local sync first

[tute]

I'm finding this project really useful, thank you very much for it!

I'm building a mobile app for places with low connectivity. One issue I find is that it's better to get completely offline than on slow EDGE/3G connections, cause when it knows it's online it will wait even for the slowest queries. I'd like to be able to tell dualStorage to infer offline, and perform sync in the background, or manually when I tell it to.

Is there any way we can do so right now? If not, I'm happy to receive general guidance and try it in a fork!

Possibly related: #4

Thank you very much in advance.

[nilbus]

I think what you're doing is a good idea. You should be able to pull this off by setting your collection/model to be local, except when you call syncDirtyAndDestroyed. With 1.1.0, you should be able to do something like:

SomeCollection = Backbone.Collection.extend({
  backgroundSync: function() {
    this.isSyncing = true;
    this.syncDirtyAndDestroyed();
    this.isSyncing = false;
  },
  local: function() { return !this.isSyncing; }
});

It seems like syncDirtyAndDestroyed ought to always ignore the local option when saving, since it's intended to be an online operation. I added a commit just now to master that implements this, so you should be able to do this and call syncDirtyAndDestroyed on your collection:

SomeCollection = Backbone.Collection.extend({
  local: true
});

Could you test it out and make sure it works for you?

Since you brought this up, I also just updated the README to document that you can use a function for the local/remote options. Thanks!

[tute]

Thanks for your quick response!

I'm now playing with the offline collections, and still learning how it works.

One idea: dualStorage was working for me always online, trying connections all the time, but enquing for later the failed connections. That's a killer approach. What I imagine could even be better (I rephrase to make sure we are on the same page) is that dualStorage continues to think it's always online, but every connection is async. So, if I add an item to a collection, it instantly adds it, updates localStorage, and tries a connection or enqueues.

Fetch the index page? No problem! Grab the collection from localStorage, and check if the API is available to grab fresh data. Much like HTML5 manifest: where it loads everything as if offline, and any change to be seen takes another request.

Is your approach of taking the models offline (local: true) aligned with these ideas?

Thank you very much for your time.

[nilbus]

Yes, exactly. local: true will make everything happen locally (offline); you just need to call syncDirtyAndDestroyed to try to sync in the background. There is no callback from syncDirtyAndDestroyed to let you know how it went, but you can monitor the events emitted by the collection and model model from the save and destroy calls that syncDirty and syncDestroyed make.

Probably the reason it was implemented this way (default to onlineSync when online) in the first place was the thought that a delayed success response with an up-to-date response seemed preferable to two success callbacks, one for the local/offline sync and one for the online sync (if successful). @lucian1900, does that sound right, or was it thought out that far?

It may be worth thinking about making this the default behavior in the future for the next major release (2.0). Specifically:

• Sync locally first and give an immediate success callback with a status that allows the callback to differentiate whether this response is up to date or cached
• Attempt to sync with the remote, and on success/error, call the appropriate callback again, this time with the server response
• Have Backbone.dualStorage manage detecting the online/offline state, sync retries, etc. This would resolve Automatically sync when connectivity resumes #4 and Prevent duplicate create requests when attempting to sync new dirty records again #35, and allow for the deprecation of syncDirtyAndDestroyed.

If you're interested in having Backbone.dualStorage manage that, I think that would be great. Feel free to discuss with me how you think it should behave, and maybe you can start a new branch to begin work on it. Let me know what you think.

Also please let me know if my change in master allows syncDirtyAndDestroyed to work properly for models/collections that use local: true. Thanks!

[lucian1900]

In the app this was built for initially, the local source was the authoritative one, so a single callback was preferable. There could be a callback passed to syncDirtyAndDestroyed or attached somewhere else if it's automatic, to find out whether the sync failed.

[nilbus]

@tute I'm sorry, it seems that I remembered wrong about setting local: true. Apparently in local mode, it will not mark models dirty, so syncDirtyAndDestroyed will have no effect. See https://github.com/nilbus/Backbone.dualStorage/blob/master/backbone.dualstorage.coffee#L235

I could change that line to allow you to pass the dirty: true option to all your save and destroy calls.

options.dirty = true if options.remote is false and not local

That's really not ideal, but it would work as a workaround for now.

Alternatively, you could set local dynamically only when fetching, but not when saving/destroying. That would get you instant loads from cache and slow/online write operations. Something like:

SomeCollection = Backbone.Collection.extend({
  localFirstFetch: function() {
    this.localFetching = true;
    this.fetch();
    this.localFetching = false;
    this.fetch();
  },
  local: function() { return this.localFetching; }
});

What do you think?

[tute]

Hi, thank you both for your great feedback. Following two ideas sound so good to me:

• Sync locally first and give an immediate success callback with a status that allows the callback to differentiate whether this response is up to date or cached
• Attempt to sync with the remote, and on success/error, call the appropriate callback again, this time with the server response

I was actually wanting so much to have callbacks for both offline and online updates, right now. Having the feedback (for the developer or end users as well) that something is only on your device until you get a better connection or manually sync seems like very healthy to me, right now writing app code that works around it inspecting localStorage["/api/#{collection}_dirty"] with a setTimeout after render (as nasty as it sounds! :)).

That's another topic, regarding the local-first behavior I'm still not testing (have to install some software to throttle my connection, will get back to you on it). The idea for fetch() seems reasonable, but I'll then have the problem for create/update, so I yet wouldn't do it that wat. The use case: slow 3Gs are worse than offline, and turning your phone into airplane mode and not being able to receive calls: not cool. :)

Till next, thank you again very much!

[tute]

Here's a different flow for a similar problem (I'm sure you already know it, in my to-do list to test and compare): https://github.com/ggozad/Backbone.cachingSync#behavior

[nilbus]

I actually was not aware of Backbone.cachingSync. Thank you for bringing it to my attention. I'll have to take a look and see what they're doing too.

I will definitely consider your desired sync behavior for a future release. You have a very compelling use case. Feel free to start on something before I do, if you decide to stick with dualStorage.

[tute]

Just a heads up: localFirstFetch working really well, will continue to play with it.

I just changed local: function() { return !this.localFetching; } to local: function() { return this.localFetching; } (no negation).

Keep in touch!

[tute]

I don't understand why do we need local() to be the negation of localFetching. Am I understanding the flag backwards?

[nilbus]

Awesome, I'm glad that worked for you.

You didn't misunderstand - I got it backward. Whoops! Edited my original comment. :-)

[tute]

Cool. And my slow action was fetch, all the others are correctly handled (syncing the views instantly, performing the API call for as long as it takes), so I'm really happy with the solution. For the record, it ended up as:

  localFirstFetch: (options = {}) ->
    @fetch(options)
    setTimeout (=>
      @localFetching = true
      @fetch(options)
      @localFetching = false
    ), 1

  local: ->
    @localFetching

As I write this I see that I could just change local as an attribute?

[nilbus]

Yes, that would work too and perhaps be a little simpler.

[tute]

Indeed this is all, getting elegant! :)

  localFirstFetch: (options = {}) ->
    @fetch(options)
    setTimeout (=>
      @local = true
      @fetch(options)
      @local = false
    ), 1

Checking it with a connection throttler and looking good. Will now try with no timeout.

[tute]

So with no timeout somehow it doesn't render, that's the best I could come up with. Should we integrate it into the library?

[nilbus]

Are you saying the page doesn't render because of a javascript error?

[tute]

Oh no, it's like if an event is not fired, or something. There has to be a race condition or something, I could spend more time figuring it out why do we need that delay (kind of like a defer).

[nilbus]

ah, I see. I'll have to look into this more later.

[tute]

I do have an add listener on a collection, that renders the views, and they don't fire without that setTimeout. We can work it out together, meanwhile I don't find it too ugly.

[maxfi]

I think I might have this sorted. See pull request #86

[maxfi]

@tute Do does this solution work the same as #86? Also, what software did you use to throttle your connection?

[tute]

I don't know if it works exactly the same, we should compare each other's solution. :)

To throttle the connection in my Mac I use http://slowyapp.com/.

[maxfi]

I have slowy app too. Not sure if it throttles local host though...