Accessing FitBit Intraday Data
For Christmas my wife and I brought each other a new FitBit One device (Amazon affiliate link included). These are small fitness tracking devices that monitor the number of steps you take, how high you climb and how well you sleep. They’re great for providing motivation to walk that extra bit further, or to take the stairs rather than the lift.
I’ve only had the device for less than a week, but already I’m feeling the benefit of the gamification on FitBit.com. As well as monitoring your fitness it also provides you with goals, achievements and competitions against your friends. The big advantage of the FitBit One over the previous models is that it syncs to recent iPhones, iPads, as well as some Android phones. This means that your computer doesn’t need to be on, and often it will sync without you having to do anything. In the worst case you just have to open the FitBit app to update your stats on the website. Battery life seems good, at about a week.
The FitBit apps sync your data directly to FitBit.com, which is great for seeing your progress quickly. They also provide an API for developers to provide interesting ways to process the data captured by the FitBit device. One glaring omission from the API is any way to get access to the minute by minute data. For a fee of $50 per year you can become a Premium member which allows you do to a CSV export of the raw data. Holding the data, collected by a user hostage is deeply suspect and FitBit should be ashamed of themselves for making this a paid for feature. I have no problem with the rest of the features in the Premium subscription being paid for, but your own raw data should be freely available.
The FitBit API does have the ability to give you the intraday data, but this is not part of the open API and instead is part of the ‘Partner API’. This does not require payment, but you do need to explain to FitBit why you need access to this API call and what you intend to do with it. I do not believe that they would give you access if your goal was to provide a free alternative to the Premium export function.
So, has the free software community provided a solution? A quick search revealed that the GitHub user Wadey had created a library that uses the urls used by the graphs on the FitBit website to extract the intraday data. Unfortunately the library hadn’t been updated in the last three years and a change to the FitBit website had broken it.
Fortunately the changes required to make it work are relatively straightforward, so a fixed version of the library is now available as andrewjw/python-fitbit. The old version of the library relied on you logging into to FitBit.com and extracting some values from the cookies. Instead I take your email address and password and fake a request to the log in page. This captures all of the cookies that are set, and will only break if the log in form elements change.
Another change I made was to extend the example dump.py script. The previous version just dumped the
previous day’s values, which is not useful if you want to extract your entire history. In my new version it
exports data for every day that you’ve been using your FitBit. It also incrementally updates your data dump if
you run it irregularly.
If you’re using Windows you’ll need both Python and Git
installed. Once you’ve done that check out my repository at
github.com/andrewjw/python-fitbit. Lastly, in the newly checked
out directory run python examples/dump.py <email> <password> <dump directory>.
Comments
Thanks so much for writing this (and for updating it...) Very helpful to look at my raw data.
Byron Ferguson (@ByronLFerguson)
29 Mar 2013
I am hoping to use Andrew's library. Would anyone please help me skip steps in learning how to use Git and Python by telling me why I get a syntax error when I run this code in Python Client?:
python /examples/dump.py
rickpack
21 Apr 2013
I am giving up now, but at least I learned a fair amount about Python and its programming community. Anyone who wishes to use Andrew's module and comes from an inexperienced position like me might experiment with the latest version of Python 2 instead of Python 3. If you decide to go with Python 3 like I did (latest final version is the greatest?), this site can help with installing modules that may be necessary to use Andrew's module: . Also, errors encountered on commas appear to be fixed in Python 3 by replacing the comma with an open parenthesis and then closing it appropriately. I hope this saves someone the few hours I spent bumbling around.
rickpack
21 Apr 2013
Hi,
Sorry to hear you were having trouble. I haven't tried it on python 3. I'll try to find some time this week to make it work on both 2 and 3.
Andrew
Andrew Wilkinson
21 Apr 2013
I had the Problem that running it as described above yielded the following error:
Traceback (most recent call last):
File "examples/dump.py", line 14, in
import fitbit
ImportError: No module named fitbit
I moved the dump.py from the example folder to the parent folder, so it lies beside the fitbit folder.
I now run: python dump.py
works. Python is 2.7 on OS X (from macports)
Heine
22 Apr 2013
Thanks for your work.
I had a problem due to the time format. As I'm European I use 24hour Time Format.
That leads to a crash: ValueError: time data '23:16' does not match format '%I:%M%p'
I had to change line 104 in client.py to the following:
timestamps = [datetime.datetime.strptime(e.attrib['description'].split(' ')[-1], "%H:%M") for e in elements]
now it works like a charme.
Thanks again!
Heine
22 Apr 2013
Hi Heine,
I've made a couple of fixes so you should be able to run the dump script as I wrote in my blog post, and it will support the european time format.
Thanks for letting me know about the issues!
Andrew
Andrew Wilkinson
25 Apr 2013
Hi,
I've updated the script so it works with both Python 2 and 3. If you do a git pull it should start working for you. Let me know if you have any trouble!
Thanks,
Andrew
Andrew Wilkinson
25 Apr 2013
Thanks Andrew, your script is a great relief for a python n00b like me. I got it to export my intraday fitbit values for this year back until feb 7, 2013. I also have older data (feb 6 and before that). How can I get the older data as well? The script ended without error message. Thanks!
Holger
09 May 2013
Is there a day around the 6th Feb where you don't have any step data recorded? It assumes a day with no steps is the beginning of your history.
Andrew
Andrew Wilkinson
09 May 2013
I have days without recording, but in Feb I have recordings every day: https://www.evernote.com/shard/s1/sh/c0a5c4c6-f60c-4141-890c-d84ee2d299a2/2d03dc640d0d757bb9196c78071cb046 . I would just like to export the last 2 years and fill all the missing days with zeros...
Holger
09 May 2013
Love the github code, Andrew! Thanks for sharing this! I was wondering if you are plotting the data or using it in any way? I'm thinking of using scipy/matplotlib to work up some of the data. Wondering if you have any tips?
Big cheers!
-Allen
AllenH
19 Jun 2013
Hi,
No, I'm just saving the csv files for backup purposes. The fitbit website had enough graphs for my needs!
Andrew
Andrew Wilkinson
19 Jun 2013
Hi,
I have been trying this out today and it looks like Fitbit may have changed things again. I get data for sleep, calories etc. but not for steps. This is using the dump example.
Is it still working for others?
Thanks
Al
Al
27 Aug 2013
Hi Al,
I checked my data last night and the steps data is downloading correct. Can you confirm that the other csv files have data in them, but steps.csv is just full of zeroes?
Thanks,
Andrew
Andrew Wilkinson
29 Aug 2013
Hey Andrew, I'm having the same issue where the steps data is an empty file and then the other files are downloaded but only contain 0 values any suggestions?
Paul
16 Oct 2013
Hi there, just to confirm that the steps csv is full of zeroes also (it's the only one, all others have data in them).
Thanks for your time putting the script together though.
R
Richard
10 Nov 2013
HI again,
Just to say that I fiddled a little with the code, and commented out (see below) the condition in the dump_day method that checked for zero steps to continue on, this then populated the steps.csv file for me correctly (when compared with the same fitbit data on the dashboard).
# Assume that if no steps were recorded then there is no data
#if sum([s[1] for s in steps]) == 0:
# return False
Thanks again,
R
Richard
12 Nov 2013
Not sure if anyone is still reading this, but there's no issue tracker open on Github....
Was initially getting "NameError: global name 'HTTPError' is not defined" -- which may be caused by differences between Python versions of versions of urllib; adding "from urllib2 import HTTPError" to client.py helps.
Now getting "NameError: global name 'HTTPError' is not defined" -- which seems to be because Fitbit is now returning the HTML ellipsis character in its results, which is not defined in XML and thus not understood by xml.etree.ElementTree.... Not sure what the fix is there.
Whatever the cause, it's successfully grabbing data for the first day (ie, today), then dying. Looks promising otherwise; thanks for keeping this alive!
TD
10 Jan 2014
Hi TD,
Thanks for pointing this out, I have just pushed a fix for this to GitHub.
Thanks,
Andrew
Andrew Wilkinson
14 Jan 2014
Hi Andrew,
thanks for your great script to get the fitbit data.
Additionaly i want to use the dump2sqlite.py script. But i have problems in indentifying the proper credentials from the cockies (Firefox 27.0 both on Win and OSX 10.9.1). It would be very helpful to get assistance for gaining the information.
Thanks.
Uwe
15 Feb 2014
Andrew, thank you for this great post. I was successfully capturing intraday data, however, when I went to run the script today, I received an invalid username or password error. I haven't changed my credentials or the process by which I run the script. I was wondering if you had any thoughts on this. Any guidance would be greatly appreciated.
Thanks,
Jonathan
Jonathan
03 Mar 2014
Hi Jonathan,
Is this still failing for you? I've just checked my download process and it's working fine. Perhaps there was a temporary issue at fitbit? How often are you running the script? If you run it too often them you might get blocked.
Andrew
Andrew Wilkinson
03 Mar 2014
Hi Uwe,
I'm afraid I don't use the dump2sqlite script so I've not updated it. If you look through the commits I've made to the other script it should be relatively straight forward to apply them to it as well.
I'm happy accept pull requests!
Andrew
Andrew Wilkinson
03 Mar 2014
Andrew,
Thank you for the guidance. I hadn't ran it in a few days; however, I tried an alternative account and the script ran fine. Is there any simple way to get my account "unfrozen"? Or, will it eventually unfreeze?
Thanks again,
Jonathan
Jonathan
04 Mar 2014
I'm now seeing the same issue. I suspect FitBit have changed something on the site. I'll try to fix it this week.
Andrew Wilkinson
04 Mar 2014
Thanks Andrew. Much appreciated.
Sincerely,
Jonathan
Jonathan
05 Mar 2014
Hi,
I've just committed a fix so if you git pull you should be good to go. Fitbit have started redirecting users to an https site, which broke the script.
Andrew
Andrew Wilkinson
08 Mar 2014
Andrew,
Again, thank you so much. The fix worked perfectly.
Jonathan
Jonathan
10 Mar 2014
Hi there. This is amazing. It is just what I have been looking for. Everything else has been a huge nightmare but I got this one working within minutes. Thank you!
katystreet
25 Mar 2014
Thanks for writing the library, I used the dump.py example to get intraday step data for visualizing in Processing. You can see a screenshot here: https://www.flickr.com/photos/watz/13938381021/
If you want my Processing code I'll be happy to send it to you, I've linked to the GitHub repo wherever I've posted it.
mariuswatz
21 Apr 2014
[…] struggled for a while to figure out how to download my data. Finally I found this post which outlines how Andrew Wilkinson does it in python and includes a link to his code. His code […]
Fitbit Data Analysis | Like Math, But Louder
05 Jun 2014
Hi Andrew, I was using your (fixed) script to obtain intraday data from fitbit. Unfortunately the script stopped working. It looks like opener = build_opener(HTTPCookieProcessor(cj)) in the login method returns a http 500 error.
Chris
20 Jun 2014
Hi Chris, yes, FitBit seem to have changed something that has broken the script. I'll try to fix it in the next few days.
Andrew Wilkinson
21 Jun 2014
This problem has been solved. Just git pull and you should be good to go.
Andrew Wilkinson
23 Jun 2014
Hello Andrew,
In the dump_day function, steps = c.intraday_steps(date) keeps returning zero steps, even though in the fitbit.com dashboard I can clearly see activity at the respective date. This results in no data being dumped, of course.
Could you please look into it? Am I doing something wrong?
grig (@growlerpig)
18 Mar 2015
Hi Grig,
I've just checked my downloads and they appear to be downloading the steps correctly. I'm afraid I don't know why you're getting 0s.
Thanks,
Andrew
Andrew Wilkinson
19 Mar 2015
Right, I logged some more data for today and it works now. Might be because yesterday I had just done my first sync and the API wasn't yet seeing it.
Thanks for checking!
Grig
grig (@growlerpig)
19 Mar 2015
Andrew, I found your script the other day and was finally able to download my data today. Thanks for providing the dump.py script. I am mostly interested in seeing my sleep data, but I'm sure I'll make use of the rest of it too.
You indicated that you have a Fitbit One (or had). Did you figure out how to download the information for floors? I'm curious what the values for that are, or how I would find that intraday information.
Thank you (and wadey),
Inez
belphebe
06 Sep 2015
Great piece of code - does just what I want. I'm definitely not a Python developer but know enough about coding in general to get this to work. I'm running Python 3.5 on Windows and found a couple of minor issues in the client.py which I got round by making the following changes (which may or may not be the best solution):
Line 9: HTTPError = urllib.error.HTTPError
Line 91: return ET.fromstring(data.strip().replace(b'"…"', b'"..."'))
Also, whenever the day changes in the Sleep data there was a carriage return missing - which I couldn't resolve in the code but fixed easily enough by editing the files being produced
Nick White
07 Dec 2015