In my last article I covered the steps of upgrading a Django 1.5 site to 1.8. For my next trick I will port the site to Python 3. But why should we use Python 3 I hear you ask. Python 2.7 is good enough. For me there are two main reasons:
- Python 3 is a better language. In particular it's easier for newcomers to learn.
- Continuing to use Python 2 causes technical debt. Package maintainers need to support both versions of the language, this time and effort could be better spent on new features or bug fixes.
For more discussion on the topic listen to this from python import podcast Python 3 espisode.
An overview of my approach for upgrading the site.
- Upgrade Django version from 1.5 to 1.8. Done.
- Upgrade other libraries to latest Python 2.7 versions. Done.
- Run 2 to 3 migrations tools
- Run site on Python 3
Can I use Python 3?
Install the caniusepython3 package then run it to check your requirements file:
Quel désastre! Four packages aren't supported. Let's dig into if we can still port.
Fabric isn't an issue, it's run on the laptop deploying not the server. I have Python 2 and 3 installed on my laptop. Alternatively you could try this gist to get Fabric running on Python 3.
According to the README on the django-admin-sortable Github "django-admin-sortable 1.7.1 and higher are compatible with Python 3.". Strangely caniusepython3 doesn't think it is. Let's move on.
I can see on the pywurfl Pypi page "pywurfl is a Python 3 language package" OK great. Let's move on to the last package.
A quick google for xhtml2pdf python 3 shows some discussion on Github which says even though Python 3 isn't supported it might work.
- Don't believe everything caniusepython3 says.
- Man-up. Install Python 3 and see what breaks.
Installing Python 3 Virtual Environments
If you're running a Mac then the easiest way to install Python 3 is to go to https://www.python.org/downloads/ download and run the installer. This will install Python 3 but leave the system Python 2.7 as the default. At the time of writing the latest stable version of Python is 3.4.3.
Optional step listen to Jean Michel Jarre to add to the futuristic feeling.
Next create a virtual environment with Python 3. In the example I'm using virtualenvwrapper:
Now comes the scary step. Let's try and install our requirements:
Noooo! There's an error with wsgiref and what's worse, it's to do with the bloody print statement needing brackets.
It's been a couple of years since I worked on this site, and to be honest I don't remember what wsgiref is used for. I search in the project and can't find a reference to it, so I remove it. I'm sure we'll find out if we need it later, if not we've dropped a redundant requirement.
Python 3 code changes
Now is the time when you're advised to run the 2to3 Automated code translation tool. I opt for the "run the site and see what breaks" method instead, as I'm interested in digging into the differences between Python 2 and 3:
I can see this code is related to downloading CSV files, when I dig into this I find out that the project used to support downloading CSVs for the client. It turned out the client preferred PDFs, so this is in fact obsolete code. I delete it. I am the eradicator of technical debt!
I run the site again and hit a syntax error. It appears to be coming from the xhtml2pdf package.
From reseaching this further is appears that the xhtml2pdf code on Github supports Python 3 but the version on Pypi doesn't:
I update my requirements to install from Github with this magical syntax:
Great success! The homepage of the site has loaded. Now let's check the rest of it. When I submit one of my forms and it saves I now have a new error:
The culprit appears to be this line:
The fix is straightforward, we just need to specify the string encoding:
I hit another error later in the multi-step form:
This error can be tracked to this line in
Looking on Stack Overflow the top rated answer is to change the code to this:
I try submitting my form again. A new error!
Looking into the code further I realise it originates from https://djangosnippets.org/snippets/1688/. The perils of copy and pasting code!
No-one on the internet has solved this problem so I'm going to have to roll up my sleeves and solve it myself. The fix is to change:
The error occurred because
MONTHS.items() is a list in Python 2, but a view in Python 3. The Python docs explain this much better than I could.
On the final step of the form I notice my form drop-down for University are all displaying as "University Object".
This is fixed by changing:
On my University model to:
From the Django docs: "On Python 3, as all strings are natively considered Unicode, only use the __str__() method (the __unicode__() method is obsolete). If you’d like compatibility with Python 2, you can decorate your model class with python2unicode_compatible()." - https://docs.djangoproject.com/en/1.8/ref/models/instances/
In the Django docs I find their Porting to Python 3 guide. I should have probably read before I started, but who likes to read the instructions!
One more thing...
I think I'm finished but then I go to test downloading applicants as PDF functionality and see:
The problem is because StringIO has changed. I change the old code:
This fixes the issue but I'm getting another error with the applicant PDF download code:
This is a straight forward fix
iter.next() needs to be changed to
next(iter). Am I finished? Nope I hit another error:
After a lot of head scratching and a post on Stack Overflow I learn in Python3 you need to use
ByteIO insead of
StringIO because strings are natively considered Unicode not bytes:
I've been lucky with this site that I've not had to manually port any Python 3 packages. Also a number of the hurdles I hit were caused by redundant requirements or functionality so could be removed rather than fixed.
Porting older codebases to Python 3 can be a bit of a pain. However for new projects there's no reason to not use Python 3. Due to the ease of running two versions of Python, you can always start with Python 3, then drop back to Python 2.7 if you find a particular package doesn't support Python 3 yet.
When you encounter a dependency that doesn't seam to support Python 3, have a decent search on Google, Stack Overflow and Github. There are often already Python 3 fixes or forks available, especially for popular packages.
The most troublesome code to fix was code that was copied from somewhere on the internet. In both cases the code was complicated to debug as it was an abstracted reusable solution. If you can write simpler code to do the job then I recommend considering doing this over copy and pasting complex abstract solutions.
I recently spoke about upgrading Django sites to Python 3 at the London Django Meet-up. Here are the slides and video.
If you'd lke to contact me about this article then use twitter @petexgraham or email firstname.lastname@example.org.