Streaming algorithmically-generated images with Flask

I’m working on building a web application with Flask that will algorithmically generate a neural artwork image based on user-uploaded input images and allow the user to see the images generated at intermediate stages of the computation. For the purpose of this post, all you need to know about my project is that I have a large computation that runs over many iterations and generates an intermediate image in each iteration, and I would like the user to be able to see those intermediate images in their browser as they are generated. Instead of my actual computation, I’ll use a silly example: on each iteration, generate a random noise image:

for i in range(10):
    noise_image = np.random.uniform(0, 256,(224, 224, 3)).astype('float32')
    # send noise image to browser???

To accomplish this goal, some Googling suggested that content streaming might be what I’m looking for. Streaming allows you to send a large amount of data to the client one chunk at a time. This sounds perfect for me: as soon as one image is ready to go, I can stream it to the client and then carry on with my computation. The way this works is by turning our computation into a generator that yields the current image state after each iteration. Then we can call the stream_template() function defined in the Flask documentation and pass it our generator:

def stream_template(template_name, **context):
    app.update_template_context(context)
    t = app.jinja_env.get_template(template_name)
    rv = t.stream(context)
    return rv

# view function
@app.route('/new_image')
def generate_image():
    def generate():
	for i in range(10):
	    noise_image = np.random.uniform(0, 256,(224, 224, 3)).astype('float32')
	    # yield something
    return Response(stream_template('image.html', data=generate()))

The problem then becomes: what do we yield? How do we get the image data to the client? At first, I simply wanted to stream the binary image data directly to the template. I ran into a number of issues with this approach, most notably the following:

  • If the image to stream is too large, Flask puts a giant cookie into the header, causing the browser to produce a ERR_RESPONSE_HEADERS_TOO_BIG
  • Even if the image is sufficiently small that the previous issue doesn’t arise, the numpy array needs to be converted into an image (using spcipy.misc.imsave). Saving the image to a temporary IO stream and then sending that binary stream to the browser caused me extreme difficulty with encoding and decoding the stream as well as issues with Flask complaining about working outside of “application context”

As a workaround, I decided to save the image to a file instead of trying to stream the binary image and then yielding the file name to the template. But then, we still have the issue of serving up those saved files. I discovered that if we want to serve those files using Flask by assigning them to a route whose view calls send_from_directory(), Flask queues those file requests AFTER each call to the generator. That is, if our template looked like:

<img id="my_image">

{% for im_name in data: %}
<script>
    document.getElementById("my_image").src = 'out/{{ im }}';
</script>
{% endfor %}

with a route @app.route(out/) , all of the calls to the generator in the new_image/route would be queued up before any of the calls to the out/ route. This doesn’t work for us, since we want the images to be displayed as they are generated.

Finding no good answer to this issue, my ultimate solution was to use a separate server in the out/directory to serve up the images. That way, when we request images in the template, these requests are made to an independent server and will get executed as soon as they are received. Here’s the final working code:

@app.route('/new_image')
def generate_image():
    def generate():
	for i in range(10):
	    noise_image = np.random.uniform(0, 256,(224, 224, 3)).astype('float32')
	    scipy.misc.imsave('out/' + str(i) + '.jpg', noise_image)
	    time.sleep(1)
	    yield(str(i) + '.jpg')
    return Response(stream_template('image.html', data=generate()))
<img id="my_image">

{% for im_name in data: %}
<script>
    document.getElementById("my_image").src = 'http://localhost:8000/{{ im_name }}';
</script>
{% endfor %}

The result is the produces the following animated image in your web browser, where each frame was created dynamically in real time! (To show you, I converted the result to a gif and set it to repeat its 10 frames in a loop.)

output_hngm5q

Just wait until we swap out the white noise function for our own algorithmically-generated neural images!

UPDATE (10/12/16): Since my original post, I discovered that while the generator is creating the data to stream, it does not have access to the current Flask context (see my later post on  Flask contexts). So, for instance, if we wanted to include a link in our streamed template:

<img id="my_image">
<a href="{{ url_for('index') }}">Home</a> 

{% for im_name in data: %}
<script>
    document.getElementById("my_image").src = 'out/{{ im }}';
</script>
{% endfor %}

we would get a run time error in the call to url_for:

RuntimeError: Attempted to generate a URL without the application context being pushed.

The error arises because url_for requires an active application context. In order to keep the context around during the data generation process, we need to use Flask’s stream_with_context() method to wrap the generator. The last line of our view function should then be:

return Response(
    stream_template(
        'image.html', 
        data=stream_with_context(generate())
    )
)

 

 

One thought on “Streaming algorithmically-generated images with Flask

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s