It’s June 25th 2015, Dan’s in his Speedos, and we’re about to try out what is probably the world’s first swimming app on an Apple Watch. We’re at the London Aquatics Centre, host of the 2012 Olympics. It’s the first time we’ve tried it in a 50m pool and we’d be lying if we said we weren’t a little nervous.
What is this spaceship?
In this article we’re going to show you a video of what happened next, talk a little bit about the timeline of how we got to this point and then our experiences developing a swimming app on the Apple Watch. We’ll also give a bit of background on how we created our swimming algorithm and finish with our hopes for the next Apple Watch hardware and the Apple ecosystem.
What’s so special about the potential of the Apple Watch for swimming?
At the time of writing there are no available consumer level swim tracking devices which are also capable of monitoring heart rate in the swimming pool. Heart rate is an incredibly useful metric for calorie burn estimation, tracking fitness improvements and pacing during workouts.
edit: thanks to people commenting re: the Suunto Ambit3 which can also capture heart rate data in the pool. It uses a chest strap which buffers up data whilst swimming. When out of the water (for example resting at the side), the strap is able to burst transmit the stored data to the watch. Unfortunately most men don't get on well with chest straps as they tend to slip down during tumble turns - women fare a bit better due to swimming costumes holding it up. A wrist based sensor would still be a superior solution for this reason.
Swimming workout technology at it's best!
Beyond that, the Apple Watch has the storage and capabilities to guide you through a workout in the pool, whilst showing instructional videos explaining specific drills and terms. Haptic feedback has the potential to subtly communicate things to you during your swim — when to speed up or slow down if aiming for a specific heart rate for instance.
Meanwhile with its standalone wifi connectivity it could download training plans whilst resting at the edge of the pool or even communicate with friends or a coach. Maybe you could even be taking turns playing a swimming game with someone whilst in different pools.
Smart phones rocket fuelled the popularity of running & cycling apps, will the smart watch do the same for swimming?
The Apple Watch screen is fantastically visible underwater.... even with fogged up goggles.
The Apple Watch announcement
On the 9th September 2014 we were anxiously watching the live Apple event having heard the rumours that the Apple Watch was being announced. Perhaps a little differently from most people, there were two headline features we were looking for:
- Waterproof (enough for surface swimming at least)
- Standalone (native) apps on the device
We came away with a promise of native apps sometime in 2015 and no definitive answers about waterproofing. So why was this important to us?
A bit of our History
We’ve been working on swimming technology for the past six years with our very first app called ‘Splashpath’ (now in partnership with Speedo and branded as ‘Speedo Fit’) Our aim has always been to put swimmers on equal terms with runners and cyclists in the world of wearables and software.
We developed the first version of our accelerometer based swim tracking algorithm on the Pebble when it was first released on Kickstarter. We’re currently refining the second version, based on over 20,000 crowd-sourced lengths from real swimmers from across the world. It currently boasts 98% accuracy for freestyle and we’ve been eager to test it out on different devices.
We’ll go into more details of our algorithm, for those who are interested, towards the end of this article.
The Apple Watch is released
Fast forward to April 2015 and the Apple Watch was released along with it’s final specifications. Unfortunately this didn’t include a waterproof rating good enough for swimming. This left us with the uncertainty of when the Apple Watch 2 might be released and if it might be waterproof or not.
Fortunately, a month later, the awesome DCRainmaker got hold of an Apple Watch and did some very extensive waterproof testing. His tests gave us the hope that the watch would be waterproof enough for us to develop our algorithm on the watch when the native apps SDK became available.
WWDC 2015 & WatchOS 2 Beta 1
At WWDC this year the wait for us was over with the announcement of watchOS 2. With the addition of native apps, as well as the APIs needed for our swimming algorithm, we began a skunk works project on our app that Friday — before the conference had even finished.
Two of us had been using the Apple Watch for a few weeks at that point and had grown rather attached to them. The idea of giving up one of them to install the beta and the prospect that it might get destroyed by swimming was pretty difficult. After talk of tossing coins, Dan nobly volunteered to sacrifice his Apple Watch to the cause.
How difficult was it?
Once we had ourselves setup to develop on beta 1, things went surprisingly smoothly. Our algorithm is developed in pure ANSI C which meant that it compiled first time on the Apple Watch with no code changes at all. It also uses fixed math to ensure known mathematical precision.
We’re also used to developing on the Pebble. The Pebble has an incredible battery life of about seven days, but this is partly achieved by having very limited, almost monastic, resources available to apps. The resources available on the Apple Watch, in comparison, are quite amazing.
We were also helped greatly by the similarities in data coming from the accelerometer (the sensor which measures movement) on both the Pebble and Apple Watch. They both give 25 samples per second, are measured in gravities (Gs) and are relative to the same orientation.
Architectural challenges
There are of course some quirks to developing this on the Apple Watch.
On the Pebble, once your app is running it stays active until the user quits. That means you have a continuous stream of data which you can interpret in realtime and alert the swimmer.
With the Apple Watch it constantly switches off the screen to save battery. Not only that, but when the screen is off, your app is suspended — no realtime processing of data.
Fortunately Apple have provided APIs which get around this in an interesting way. The various sensors allow you to say in advance which data is needed and will then buffer this up for you whilst your app is suspended. As soon as the screen turns on again, your app wakes up and can then request all the data which was produced whilst it was sleeping.
You can see this happening in our video above — after the four laps, Dan looks at his watch which initially displays 0 laps and then after a short delay it receives and processes the accelerometer data and updates the display.
It’s a clever solution to the problem of battery life vs. data, and we’re confident with some creative UX solutions we can counteract this slight delay and ensure a graceful experience for the swimmer.
On a similar theme, the water has some positive and also problematic interactions with the screen. The screen is incredibly bright and clear — possibly the best we’ve ever used whilst swimming. On the downside we initially had some buttons on the in-swim screen and found that the water was activating them. Our solution there was to hide all functions behind a force touch menu.
Submersion has a similar effect to 'palming' the device. The screen turns off. Pressing the digital crown underwater will wake up the screen.
HealthKit
In iOS 8 HealthKit really pushed running, walking and cycling to the forefront. You can see this in the inclusion of running/walking distance graphs and cycling graphs which you can use to populate your Health app dashboard. Although it’s possible to enter swim workouts (we do so with manual workouts in Speedo Fit), the distances are absent from any summary graphs and make a swimmer look like they’re lying in bed every time they go for a swim.
It would be really great if we saw swimming become better represented in iOS 9. Here in the UK, for instance, more people swim every week than go for a run.
There are, however, a lot of improvements in HealthKit which have really helped make this swimming app possible.
First of all there’s now the ability to start a workout via HealthKit. This puts the watch in the same mode that Apple’s own Workout app does — when waking the screen after sleeping your app is the first thing the swimmer sees - rather than having to navigate back via the apps menu.
For a swimmer glancing at their wrist in the pool and not being able to interact with a wet or submerged screen easily, this is in invaluable.
Next there’s the way HealthKit on the watch and on the iPhone interact with each other. HealthKit on the watch is almost like a little satellite. It keeps a record of recent health data and allows you to write new information to the HealthKit Store which will then be synced back to the iPhone the next time they’re in range.
For our app this means that when the swim ends, we instantly create a HealthKit workout. That workout contains information indicating it’s a swim, the exact times of the laps performed in the pool as well as the calories burned and heart rate data captured from the strap. We also store our own extended information in the workout metadata identifying the stroke types and which pool the swim was done at.
Our swim is added to HealthKit and now contributes towards our daily activity.
The swim then appears within the Apple Health app and Activities app, contributing towards your daily scores. It’s also available for other HealthKit supported apps such as MyFitnessPal.
When can you get the app?
For the moment this unfortunately has to remain an interesting technology demo. Although iOS 9 and watchOS 2 should be making an appearance later this year in September, the Apple Watch itself is not officially waterproof.
“13.1 Apps that encourage users to use an Apple Device in a way that may cause damage to the device will be rejected”
So, we’ll just have to hold out until the Apple Watch 2 and hope that it has the required waterproof ratings.
The SwimIO Algorithm
As promised earlier, here are a few details about our swim tracking algorithm.
Version 1
This was released as an open beta on the first Pebble and had a companion iOS app (SwimIO Motion). We initially developed the algorithm only with test data recorded on ourselves and a small group of volunteers from our local pool.
The companion app invited people to participate in the beta test and swim with the algorithm. Swimmers marked up their swims with the correct turns whilst deleting incorrect turn markers. They also added metadata information about the stroke they were doing, the pool length and any extra information.
We sent out weekly ‘Swim Missions’ inviting them to do specific workouts to help refine our accuracy in one area or improve our coverage. For example, “This week everyone we need 10 laps backstroke followed by 10 laps fast freestyle” (our beta testers were incredibly responsive to these missions and this proved an immense asset to us. Thanks guys! )
Version 2
Thanks to version 1 and our audience of swimmers, we now have what is probably one of the largest data sets of marked up real world pool swimming. It covers everything from casual swimmers right through to training athletes. All in a variety of strokes, pool lengths and swim conditions.
It’s this real world aspect of the data which has massively benefited our work on version 2 of the algorithm.
Our first piece of work was to create a testing framework in order to objectively evaluate each new candidate version. We built this as a Rails app which is capable of running an algorithm against any subset of our swimming database. We have all of our swims tagged according to various factors such as stroke type and swim conditions (e.g. did they stop and start a lot, knock a lane rope). We also defined a known set of inputs for the algorithm and outputs along with the ability to generate custom graphs from any given run and see all of them for a set of swims on a single page.
When evaluating a given candidate run there are generally always a few problematic swims. Our setup enables us to run that algo locally on the selected swim and then tweak the algorithm whilst live viewing the results in our IDE to see how we can better fit the swim. What has really helped us here is initially prototyping new algo ideas using the R statistical computing language. Tuning parameters to work well for one swim can be quite distorting — being able to then run that tuned algorithm back across the whole data set again keeps things honest and objective.
We’ve built our algorithm up in layers in this way, whilst keeping each aspect parameterised. Once we’ve got a set of promising mechanisms, we’re able to combine machine learning with our test framework in order to find the best possible configuration to maximise accuracy on the highest percentage of swims.
Bravas
It’s at this point that we had to take a step back and really look at what the accelerometer data on a wrist is measuring (at least in terms of turn detection): unusual disruptions in signals. Although we optimised as much as possible to look for disruptions matching the profile of a turn, unusual things are always going to happen at other times. The person swimming in front might suddenly stand up or slow down, someone behind might overtake, or you stop to adjust your goggles.
We’ve developed a technology called Bravas which builds up a picture of the swim as it happens and then constantly re-analyses the swim finding turns where a normal signal based algorithm might have missed them and removing misclassified interruptions. I like to visualise it as a loosely fitting wetsuit which gradually adapts to the swimmer until by the end of their swim it’s a perfectly fitting skin-suit anticipating their every move.
Putting it all together
With Bravas layered on top of our base algorithm and the two of them tuned via machine learning applied using our test framework, we’ve now got an incredibly good algorithm for real world conditions.
The last stage in our development was creating a C implementation which is fast enough to run on a device such as the Pebble. Fortunately we’ve got a good depth of knowledge in our team with a background ranging from telecommunications engineering to console graphics and physics engines. All through our development we tried to make sure we only used methods which would quickly and easily translate to small processors and memory footprints.
Our test framework allowed us to run the C implementation on the same set of swimming data as the R reference and quickly find any differences in results which we could then isolate and fix.
The Apple Watch 2 and the future
We really hope that Apple see the demand for an official, fully waterproofed Apple Watch. Once that is available, we’ll be able to ship a fully featured swimming app.
In the mean time, it would be great to see swimming become a first class citizen in the Apple Health and Activities apps. Running and cycling both have their own distance graphs you can view in the dashboard whilst for swimming you have to drill down to individual workouts. Although you can’t swim with the Apple Watch, many people swim with other devices as well as logging their swim workouts manually on other apps (including our own manual swim tracker app ‘Speedo Fit’)
If there is a future ambition for Apple Health/HealthKit, Apple Watch, the new Activity app and ResearchKit to all tie up and tell the narrative that "Apple user's live longer healthier lives", then recording all fitness activity, including swimming, is a must.
We hope you’ve enjoyed reading what’s turned out to be quite a long post.
It’s been a real team effort with the entire team working in the evening and at weekends on the different component parts to make it a reality.
We’d also really like to thank London Aquatics Centre for their kind permission to film our test in their amazing pool and of course DCRainmaker for his extensive Apple Watch waterproofing tests which made us brave enough to try this.
@earltedly
If you're interested in receiving more news about our swimming algorithm, potential beta testing and being notified of releases, please sign up below.