iOS Background Transfer – what about uploads?

Downloading files in the background is easy with iOS since the introduction of the NSUrlSession background transfer APIs. There are plenty of examples and instructions around for ObjectiveC but also for Xamarin.iOS.

Documentation about the APIs is pretty exhaustive over at Apple and covers everything you have to know to get started.

And then, you start asking yourself: what about uploading data? And it turns out there aren’t so many working examples and in fact it can be quite tricky to get things up and running. I was in contact with Apple support and they helped me to find answers to many of my questions. This blog post is trying to conserve to essence of my mail communication as it can help others who are challenged with implementing background upload.

Simple question: does NSUrlSession support uploading data in the
background at all?

Yes. And in my experience this support works just fine. The only specific gotcha that I’m aware of relates to the style of upload: background sessions only allow you to upload from a file, not from memory or from a stream. But you mentioned that you’re uploading from a file, so that’s clearly not the problem here.

It so happens that I have a test program that does exactly that. I just re-ran it on a device (iPod touch, 5th generation, running iOS 8.3) here my office and the upload worked as expected.

I’ve attached a copy of my test program. Here’s how you can test it:

  1. On your Mac, run the “ImageReceiveServer.py” server that’s included in the archive.
  2. Open the “BackgroundUpload.xcodeproj” project in Xcode.
  3. Change “biff.local.” in “UploadsViewController.m” to be the .local name of your Mac
  4. (you can find this in the Sharing preferences panel).
  5. Build and run the app on a device that’s on the same Wi-Fi as your Mac.
  6. If the device has WWAN (cellular data), temporarily disable that so that you know the request is going over Wi-Fi.
  7. In the app, tap the ‘add’ button.
  8. Tap Extra Large to get a relatively large upload.
  9. Tap the Start button.
  10. Move the app to the background.
  11. After a short while you’ll see the server log a successful upload.

For an even more authenticate test, stop the app in Xcode and re-run it from the home screen, then repeat steps 6 through 10. The upload should still work, even though the app was suspended in the background (if you run the app from Xcode it doesn’t get suspended when it goes into the background).

[I make the test app available for download right here – please note that this is an ObjectiveC version; I converted the app over to Xamarin.iOS but never completed it, anyway here’s a link to it]

The app I’m creating has to upload an arbitrary number of assets. I cannot grab all of them and write them to my temp folder for upload, it would just consume too much disk space. So my idea was to schedule the first upload task and once it has finished, schedule a new one from the session delegate after requesting the next asset. This would happen while the app is in the background or has been terminated. Is it possible to dynamically spawn upload tasks in the background?

Yes, but you’re going to run into problems doing the assets one at a time. That’s because NSURLSession limits how frequently it will resume (or relaunch) your app in the background, and the more often you resume the longer the delay. If you upload dozens of items sequentially like this you’ll eventually resume so infrequently that it’ll look like uploads have stopped.

I discussed this in more detail in the following DevForums post.

[Since the link above points to Apple’s dev forums, you’ll need an account to access it. Here’s the compressed version:

It’s definitely true that the best practice is to upload one large file. This will result in less load on the device and the best network behaviour.

If that’s not possible (due to server-side constraints that you can’t fix, for example), you should definitely dump as many tasks on the background session as you can. That’s because the the NSURLSession background infrastructure has limits on how frequently it will resume (or relaunch) your app. So, let’s say you have 5 requests to run. If you run them like this:

  1. start request 1
  2. suspend
  3. request 1 completes on the wire
  4. resume to handle the completion of request 1
  5. start request 2
  6. suspend
  7. request 2 completes on the wire
  8. resume to handle the completion of request 2
  9. and so on

you’ll find that your resumes start getting more and more delayed. That is, you’ll see increasing delays between step 3 and 4, step 7 and 8, and so on.

OTOH, if you dump all the requests on NSURLSession at once, you’ll get something like this:

  1. start requests 1, 2, 3, 4 and 5
  2. suspend
  3. request1 completes on the wire
  4. request 3 completes on the wire
  5. request 2 completes on the wire
  6. request 4 completes on the wire
  7. request 5 completes on the wire
  8. resume to handle the completion of requests 1, 2, 3, 4 and 5

Because you’re not being resumed often, NSURLSession won’t impose an excessive delay before the resume (between steps 7 and 8).

This is especially important if you’re handling authentication challenges, because NSURLSession must resume your app to handle the challenge and that starts increasing your resume delay.

One more thing I’ve learnt about NSURLSession background sessions today: while NSURLSessions are thread safe in general, /background/ sessions are not thread safe on iOS 7.0.x. This is a bug. The plan is for it to be fixed in iOS 7.1; I don’t know if the fix is in the current iOS 7.1 beta seed.

On iOS 7.0.x you can minimise the risk by serialising your calls to the various task creation functions (-downloadTaskWithRequest:, -uploadTaskWithRequest:fromFile:, and so on). Alas, this isn’t guaranteed to prevent problems.

This issue is reported as fixed in iOS 7.1. If you encounter other thread safety problems on 7.1 or later, please do let us know by filing a new bug report.]

In situations like this I recommend that you batch up your requests. For example, each time you resume you might start the upload of 100 items. Or, if disk size is your primary concern, you could copy over assets until you’ve used a certain amount of disk space and then upload those.

The web API I have to support doesn’t simply allow me to upload. I have to fire a request first which returns an upload ID. Then I have to upload and when I’m done, I have to call the web service once more to finalize the upload, again using the ID I got in the beginning. The upload task itself can’t perform these calls. So the way I solved it is that the first call is made while the app is still in the foreground, the upload task gets created and when the task is done, I finalize the upload and then schedule a new upload task (as stated above). I think this can be a problem since there is no guarantee the scheduled upload task will run immediately and that means, if it finally starts, the requested upload ID might have become invalid again, what’s your opinion?

This is not going to be easy. The key question is, how long do these IDs last? If they remain valid for days, your current approach should be fine. OTOH, if they expire in hours (or less!) you are likely to run into problems. NSURLSession background transfers can take days to complete for a variety of reasons. For example:

  • the device may be low on power
  • the device may not go on to Wi-Fi
  • the device may have exceeded its daily power or networking budget

See https://devforums.apple.com/message/1034335#1034335

[Again, here’s the content of that post over at the Apple forums

That reminds me of an NSURLSession background session gotcha that I recently uncovered: iOS has a system-wide resource budget mechanism which prevents it from using too much CPU, battery, networking (WWAN and Wi-Fi, independently), toothpaste, and so on in a given time period.

The mechanics of the network usage budget are complex. I can’t explain all the details here–they are deliberately undocumented because they are expected to evolve over time. However, there are some things that you can reasonably expect:

  • The budget always applies to discretionary tasks, that is, tasks created from a session whose configuration has the “discretionary” flag set.
  • The system may infer that a task is discretionary based on various criteria. For example, currently the system infers that a task started when the app is in the background is discretionary.
  • That inference may change during the lifetime of the task. For example, in WWDC 2014 Session 707 “What’s New in Foundation Networking” [1] we mentioned that, on iOS 8, the inference that a started-in-the-background task is discretionary is dropped when the app moves to the front.

If you use an NSURLSession background session to transfer /large/ amounts of data, you may encounter the Wi-Fi daily network budget. Last I checked that’s around 5 GB per day although, as I mentioned earlier, these things are most definitely subject to change.

The take-home point here is that, if you start a set of large discretionary transfers and see a lack of progress after some fixed amount of data has been transferred, and then things pick up again roughly 24 hours later, you’re probably exceeding the Wi-Fi daily network usage budget. There is no API or user-level control over this budget; you just have to wait.

One thing you /should/ be aware of is the task “priority” property we added in iOS 8. This is useful if you’re transferring different types of resources and you want one to be given priority over the other. For example, let’s say you’re uploading a huge photo gallery. You would give the gallery index a high priority and the photos themselves a lower priority. That way, if the user asks to upload a second gallery, the index of the second gallery won’t get stuck behind all the photos in the first gallery, and the user will be able to see the outline of the gallery even if they can’t yet see all the photos.]

So, if the upload IDs expire quickly, it’s definitely possible that an upload ID might expire before the upload completes. There are a variety of options you have available to improve this (I’ll go into them below) but the best possible solution would be to change the server side of this equation.

As far as improving your upload experience, here’s some suggestions:

  • Batch the uploads, as I described above.
  • Obviously you want your upload IDs to remain valid as long as possible. If you have any input on the server side of things, that might be an easy change for them to make.
  • You don’t have to do all your background networking using NSURLSession background sessions. If you have a request that you expect to complete quickly, it’s perfectly fine to use a standard NSURLSession for that. In your specific case it might make sense to use a standard session to allocate and commit your upload IDs, and a background session to do the actual data transfer.

The main gotcha here is that you’ll have to ensure that your app doesn’t get suspended while these standard request is running. You can use a UIApplication background task to do that, although it puts strict limits on how long you can run. Specifically:

  1. When you move from the foreground to the background, the limit is currently 3 minutes.
  2. When you are resumed in the background, the limit is 30 seconds.

(1) should be plenty but (2) is much more of a challenge.

IMPORTANT: These limits have changed in the past and will change again in the future. I’m quoting them here just so you have some idea of what to expect. You should manage this process with the UIApplication background task API, specifically:

  • monitor the time remaining using UIApplication’s backgroundTimeRemaining property
  • make sure you implement your background task expiry handler correctly

NSURLSession tasks have an overall timeout which you can configure via NSURLSessionConfiguration’s timeoutIntervalForResource property. That timeout defaults to a very long value (7 days, IIRC). If you know that your upload IDs expire after a certain amount of time, you could shorten the timeout so that you fail sooner. That won’t necessarily help you make progress–remember that if all your requests time out and the system has to resume your app, that’ll count against your resume rate limit–but it’s probably a good idea not to leave pointless work in the background session’s queue of stuff to do.

In my case, the request header contains a login token. If it is expired, the user has to login again and retry. Since I do not know when my tasks will run, it is possible that the token has expired meanwhile. It is kind of the same discussion as with the upload IDs. The difference is that the token is part of the request header and not of the URL. There is a callback in the session delegate that allows me to authenticate. But what happens then and when is it called?

This depends on how the authentication failure manifests itself. For standard HTTP authentication–server responds with 401 Unauthorized with authentication details in the “WWW-Authenticate” header–NSURLSession will resume (or relaunch) your app and call the -URLSession:task:didReceiveChallenge:completionHandler: delegate callback. Your app can handle that in the same way it would in a foreground session.

IMPORTANT: Last I checked these resumes /do/ count against the resume rate limiter. ISTR some discussion about changing that. I will get an update from the NSURLSession engineering team and get back to you on this point.

Now, in your case it seems like your app can’t take advantage of this because to fix the authentication you need to change a header, and you can’t change a header on an existing task. However, I suspect that you’ll still be able to make things work. AFAICT there are two things that might happen:

  • the task might fail outright — In this case you will eventually be resumed and you can create a new task with the new authentication header.
  • the request might fail with a 401 Unauthorized status, which will trigger an authentication challenge for the task — You can handle this by simply cancelling the task and starting a new one with the new authentication header.

For downloads these approaches might cause problems because the act of cancelling the task would prevent it from being resumed. However, there is no automatic support for resuming uploads, so that won’t get in your way.

I’m glad I checked because this behaviour /has/ changed. iOS 8 and later have a resume rate limiting mechanism but it’s implemented in a different way. A lot of the details are the same:

  • nsurlsessiond still tracks how often it has resumed (or relaunched) your app in the background to handle NSURLSession-related events.
  • IIt still maintains a minimum period for background work.
  • It still doubles that work period every time it resumes (or relaunches) your app.
  • It still resets that work period when the user brings your app to the front.

The key difference is how the work period is applied. In iOS 7 the work period was applied after the task completes. In iOS 8 and later the work period is applied /before/ a new task starts. That is, if a new task is created while the app was resumed by nsurlsessiond, that task gets delayed by that work period.

Notably, this delay does not apply to existing tasks that have been stalled while they handle an authentication challenge.

To bring this back to the concrete, imagine an app that’s been running (and resuming) in the background for a while, and thus has a long work period. Further, imagine that the app has a task, let’s call it task A, that’s finally started executing in the background session and has hit an authentication challenge.

In iOS 7 nsurlsessiond would get the challenge and schedule the app to be resumed. However, with the long resume delay the app wouldn’t see the challenge for a while. Eventually the app would resume, get the challenge, handle it, and then task A would continue executing.

But what happens if the app starts a second task, task B, while resumed to handle the authentication challenge for A. This would start executing immediately. Which is nice, I guess.

This approach does implement a resume rate limiter but at the cost of delaying the handling of authentication challenges. This can be a problem in some cases (most notably NSURLAuthenticationMethodServerTrust) where the challenge might time out.

In iOS 8 you get different behaviour. Once task A is executing, its authentication challenge causes the app to resume immediately. The app responds to the challenge and task A continues immediately. This is good because there’s little chance that the challenge will time out.

However, if the app schedules task B while it’s resumed because of task A’s authentication challenge, task B doesn’t /start/ until the app’s work period has expired. So, you still get a resume rate limiter, just implemented in a different way.

This will definitely impact on your app because you’re not using standard authentication challenges. If you get an authentication challenge on task A and respond by cancelling task A and starting a replacement task B, nsurlessiond doesn’t know that these are really the same task and thus task B is delayed for the work period.

Testing Background Session Code

This paragraph is copied from Apple’s Dev Forums and deals with testing background transfer.

When writing an app that uses NSURLSession’s background session support, it’s easy to get confused by three non-obvious artefacts of the development process:

  • When you run your app from Xcode, Xcode installs the app in a new container, meaning that the path to your app changes. This can confuse NSURLSession’s background session support.
    Note This problem was fixed in iOS 9; if you encounter a problem with NSURLSession not handling a container path change in iOS 9 or later, please file a bug.
  • Xcode’s debugging prevents the system from suspending your app. So, if you run your app from Xcode, or you attach to the process some time after launch, and then move your app into the background, your app will continue executing in situations where the system would otherwise have suspended it.
  • Similarly, the iOS Simulator does not accurately simulate app suspend and resume; this has worked in the past but it does not work in the iOS 8 or iOS 9 simulators (r. 16532261).

When doing in-depth testing of NSURLSession background sessions I recommend that you test on a real device, running your app from the Home screen rather than running it from Xcode. This avoids all of the issues described above, resulting in a run-time environment that’s much closer to what your users will see.

If you encounter problems that you need to debug, you have two options:
use logging — It’s important that your app have good logging support anyway, because otherwise it’ll be impossible to debug problems that only crop up in the field. Once you’ve taken the time to create this logging, you can use it to debug problems during development.
attach — If you have a specific problem that you must investigate with the debugger, you can run your app from the Home screen and then attach to the running process (via Xcode’s Debug > Attach to Process command) or use Wait for executable to be launched in the Info tab of the scheme editor.
IMPORTANT As mentioned above, the debugger prevents your app from suspending. If that’s a problem, you can always detach and then reattach later on.
Finally, while bringing up a feature it’s often useful to start from a clean slate on launch; this prevents tasks from a previous debugging session from confusing the current debugging session.

You have a couple of options here:

  • If you’re developing on the simulator, iOS Simulator > Reset Content and Settings is the quickest way to wipe the slate completely clean.
  • On both the simulator and a real device, you can delete your app. This will not only remove the app and everything in the app’s container, but will also remove any background sessions created by the app.
  • You can also take advantage of NSURLSession’s -invalidateAndCancel method, which will cancel any running tasks and then invalidate the session. There’s a few ways to use this:
    During active development it might make sense for your app to call this on launch, guaranteeing that you start with a clean slate.
    Alternatively, you could keep track of your app’s install path and call it on launch if the install path has changed.
    You could have a hidden UI that invalidates the session. This is helpful when you encounter a problem and want to know whether it’s caused by some problem with the NSURLSession persistent state.

Other notes

Here are some more findings I made which also relate to background upload.

The iOS Simulator is no good for testing background transfer and any other background API! If you try to transfer a file in the background, you’ll notice that the delegate’s “DidFinishEventsForBackgroundSession()” will never be called. This one is supposed to trigger if all downloads completed while the app is in the background. It will never be called if downloads complete while the app is in the foreground. In the Simulator it won’t be called in either case. The reason is that the Simulator does not correctly simulate the background mode behavior. You don’t have to use any of the background APIs. Just try to execute a worker thread while backgrounded: it will happily continue, while it gets suspended on the physical device. Conclusion: if you want to test background transfer, do it on the device.

To correctly configure the upload session in Xamarin.iOS, use code like this:

///
<summary>
/// Creates the upload session. This configures an NSUrlSession that support background transfers.
/// </summary>

void CreateUploadSession ()
{
// Initialize our session config. We use a background session to enabled out of process uploads/downloads.
using (var sessionConfig = string.IsNullOrWhiteSpace (BACKGROUND_SESSION_ID) ? NSUrlSessionConfiguration.DefaultSessionConfiguration : NSUrlSessionConfiguration.CreateBackgroundSessionConfiguration (BACKGROUND_SESSION_ID))
{
// Allow downloads over cellular network too.
sessionConfig.AllowsCellularAccess = this.GetAllowCellular();

// We want our app to be launched if required.
sessionConfig.SessionSendsLaunchEvents = true;

// Give the OS a hint about what we are downloading. This helps iOS to prioritize. For example "Background" is used to download data that was not requested by the user and
// should be ready if the app gets activated.
sessionConfig.NetworkServiceType = NSUrlRequestNetworkServiceType.Background;

// Configure how many concurrent uploads we allow at the same time.
sessionConfig.HttpMaximumConnectionsPerHost = 1;

// Let iOS optimize when the transfer starts.
sessionConfig.Discretionary = true;

// Defines how long a task should wait for new date to arrive. iOS default is 60 seconds. Set to 10 minutes.
sessionConfig.TimeoutIntervalForRequest = 60 * 10;

// Defines how long to complete an entires task in a session. iOS default is 7 days. Set to 1 day.
sessionConfig.TimeoutIntervalForResource = 60 * 60 * 24;

// In our case we don't want any (NSURLCache-level) caching to get in the way
// of our tests, so we always disable the cache.
sessionConfig.RequestCachePolicy = NSUrlRequestCachePolicy.ReloadIgnoringCacheData;

// Create a session delegate and the session itself
// Initialize the session itself with the configuration and a session delegate.
var sessionDelegate = new BackgroundTransferSessionDelegate ();
this.uploadSession = NSUrlSession.FromConfiguration (sessionConfig, sessionDelegate, null);
}
}

You will have to use a separate session delegate. You cannot use the callbacks exposed on the session itself!

In order to create an HTTP request for an upload, you’re code will look like this:

///
<summary>
/// Creates a request that can be used to upload a file
/// The request is returned from the method. The data that will have to be uploaded is stored
/// into the passed in localTmpStream object.
/// </summary>

/// <param name="serverUrl">Server API URL</param>
/// <param name = "localFilename">
/// filename of the asset in the tmp folder
/// </param>
static NSMutableUrlRequest CreateUploadRequest (
string serverUrl,
string localFilename)
{
Debug.Assert (!string.IsNullOrWhiteSpace (serverUrl), "Server URL must be sepcified!");

serverUrl = serverUrl.Trim ();

if (!serverUrl.EndsWith ("/", System.StringComparison.Ordinal))
{
serverUrl += "/";
}

var request = new NSMutableUrlRequest (NSUrl.FromString (Path.Combine (serverUrl, "nodes/files/uploads/", uploadId))) {
HttpMethod = "POST"
};

// Use a new boundary to prevent problems with multiple concurrent uploads.
request ["Content-Type"] = "multipart/form-data; boundary=" + BackgroundTransfer.Boundary;

// We can add additional header information, like the filename.
var keys = new object[] {
BackgroundTransfer.HEADER_VALUE_LOCAL_FILENAME
};
var objects = new object[] { localFilename ?? "" };
var dictionary = NSDictionary.FromObjectsAndKeys (objects, keys);

request.Headers = dictionary;

return request;
}
Advertisements