August 03, 2019 · 10 mins read

Lesson 2 Part 1 - ML in Production

fastai deploying

After my blog post on the first lesson of fast.ai yesterday (which got recognised by the founder of fast.ai!), today is time for the second lesson. Because this lesson consisted of three distinct parts, I decided to write 3 separate blog posts. The first one, the one you’re reading now is on deploying machine learning models in production. The second one is on gradient descent from scratch in Pytorch. Last but not least, the third post is on things that may go wrong while training a neural network.

In this post, I will show you how to export a machine learning model from Google Cloud Platform, download it to your computer, build a small web app, and deploy everything to Render. My project does cat vs dog classification but it can be anything you want.

Exporting the model

The first thing you need to do is get your model out of google cloud, or whatever cloud computing platform you use. This can be done using just one line of code:

learn.export()

This will generate export.pkl in the directory where your training script, presumably a notebook, was running. If you are using a notebook you are lucky because you can just download this file by selecting it in the file browser and clicking . If you are using ssh’ed into a server you get download the file as explained here.

Creating a simple web app

Before we start creating the web app, start by creating a virtual environment and install the flask and fastai even if you have them installed on your system already. (installation)

# Create virtual env
virtualenv env

# Activate virtual env
source env/bin/activate

# Install dependencies
pip install flask
pip install fastai

The framework we will use is Flask. Although there might be better options out there, Flask can do everything we want and it is super simple to use. Install it using pip:

pip3 install flask

You need just a few line of codes to get your app running. First, initialise an app object. Then you create your first route, a location of the website.

from flask import Flask

# Get a new app object
app = Flask(__name__)

@app.route('/')
def index():
	return 'Hello world!'

if __name__ == '__main__':
    app.run(host='0.0.0.0')

By setting the host to 0.0.0.0 the app is accessible to all computers that your computer accepts public traffic from. In your LAN this could be other computers. In our case, when we are running it on a server, it means that all computers on the internet have access to this page.

If you run this code locally and visit localhost:5000 you should be able to see “Hello World!” If you don’t see this output, check if Flask caught an error or outputted another port for your app.

Sending HTML responses

While sending text to the user is interesting, we would rather use HTML. If you have never used HTML before, don’t worry: you don’t need to understand how it works - that’s not the point of this tutorial.

To send HTML documents to the user, you need to change the initial program a little. More specifically, update the following lines

from flask import Flask

# Get a new app object
app = Flask(__name__)

to

from flask import Flask, render_template

# Get a new app object
app = Flask(__name__, template_folder='templates')

Create a directory in your application’s directory called templates. Add the following to the index.html file (inside templates):

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8">
        <title>Dog / Cat Classifier</title>
    </head>
    <body>
        <h1>Dog / Cat classifier</h1>

        <form action='/' method='POST' enctype='multipart/form-data'>
            <input type='file' name='file' accept="image/png, image/jpeg"><br>
            <input type='submit' name='upload_btn' value='upload'>
        </form>
    </body>
</html>

Then update the index route to:

return render_template('index.html')

Now reload the web page and you’ll see a simple html form.

It’s not my goal to get this page to look pretty. It’s just for demonstration purposes. Feel free to add some styling yourself.

Accepting image uploads

The page allows us to upload images to the web server. If you tried to do this already, you probably got a 405 Method not allowed error. That’s because the route currently only accepts so-called GET requests (asking). Update the route declaration so the route accepts POST requests (telling) as well:

@app.route('/', methods=['GET', 'POST'])

In order for us to learn more about the request (POST requests are requests too. The naming is a little unconventional), you should import request from flask like so (note: we also import make_response that we’ll need in a second):

from flask import Flask, render_template, request, make_response

The request object will store information about the request. To learn what kind of request the user made, we access the method (GET, POST, etc.) property. Change the index() function accordingly:

global learn

if request.method == 'GET':
    return render_template('index.html')
elif request.method == 'POST':
    # todo: process image
	  return 'you made a POST request!'

Usually it’s bad practise to have global variables. In this case however, it’s the most convenient way to share the learn object, which can may take some time to load, with all the users.

Finally, we can access the images sent with the request like so:

request.images

In the index() function, replace the line that says todo with:

try:
		data = request.files['file']
		img = open_image(data) # convert to an image fastai can understand
except KeyError:
		return make_response('Bad request', 400)

This makes sure we have an image to work with, and otherwise notifies the user he made an invalid request.

Processing the images

Now to the most fun part: integrating machine learning.

Start by importing fastai.vision like so:

from fastai.vision import *

Start by setting the device to a CPU:

vision.defaults.device = vision.torch.device('cpu')

The reason for this (source):

“You probably want to use CPU for inference, except at massive scale (and you almost certainly don’t need to train in real-time). If you don’t have a GPU that happens automatically.”

Start by defining the path and loading the model you downloaded (you might need to move it to your application folder) using load_learner

path = vision.Path('.')
learn = vision.load_learner(path)

You can now use the learn object in the POST part of the index route:

pred_class,pred_idx,outputs = learn.predict(img)

Finally, return the prediction along with the accuracy in JSON format:

return jsonify({
    'result': str(pred_class),
    'accuracy': float(outputs[pred_idx])
})

This uses another function of flask: jsonify that you need to import as well.

This completes the web app! You can now use it on your computer.

Deploying to Render

Although running code on your computer is fun, exposing something the world is even better. To do so, we’re using Render. Render is a great service for deploying code. It is super easy to use, even if you have never deployed before and it is quite cheap as well: only $5 a month for 1 cpu and 1gb of ram. The best thing, however, is that you can pay per second so you can follow the tutorial and then shut down you server if you want to do so. You can see more information on the pricing page.

We’ll need a requirements.txt that contains all there requirements for this project. You can create that like so (in an active virtual environment):

pip freeze > requirements.txt

Assuming you’re familiar with git and GitHub, initialize a git repository and push the code to GitHub.

If you don’t know what the above means, check out a tutorial like this one. It’ll give you superpowers!

Then create an account at render.com.

Go to this page to create a new Render project from your GitHub repository. Just select the project, give it a name and enter something like this for the following fields:

  • Build Command: pip install -r requirements.txt
  • Run Command: python3 main.py

Click create web service et voila, there’s your production server. Easy as that.

Conclusion

As you’ve seen in this tutorial, it’s easy to export a fast.ai model and load it in another app. You’ve also learned how to write a basic web application using flask and deploy it for the world to see!

You can view my code on GitHub and you should also take a look at the deployment.

This was the first of a three part series on fast.ai lesson 2. You should also read part 2, part 3 and more posts about my fast.ai journey.

A huge thanks to Sam Miserendino for proofreading this post!