uWSGI Transformations

Starting from uWSGI 1.9.7, a “transformations” API has been added to uWSGI internal routing.

A transformation is like a filter applied to the response generated by your application.

Transformations can be chained (the output of a transformation will be the input of the following one) and can completely overwrite response headers.

The most common example of transformation is gzip encoding. The output of your application is passed to a function compressing it with gzip and setting the Content-Encoding header. This feature rely on 2 external packages: libpcre3-dev, libz-dev on Ubuntu.

plugin = python,transformation_gzip
http-socket = :9090
; load the werkzeug test app
module = werkzeug.testapp:test_app
; if the client supports gzip encoding goto to the gzipper
route-if = contains:${HTTP_ACCEPT_ENCODING};gzip goto:mygzipper
route-run = last:

route-label = mygzipper
; pass the response to the gzip transformation
route = ^/$ gzip:

The cachestore routing instruction is a transformation too, so you can cache various states of the response.

plugin = python,transformation_gzip
http-socket = :9090
; load the werkezeug test app
module = werkzeug.testapp:test_app
; create a cache of 100 items
cache = 100
; if the client support gzip encoding goto to the gzipper
route-if = contains:${HTTP_ACCEPT_ENCODING};gzip goto:mygzipper
route = ^/$ cache:key=werkzeug_homepage
route = ^/$ cachestore:key=werkzeug_homepage
route-run = last:

route-label = mygzipper
route = ^/$ cache:key=werkzeug_homepage.gz
; first cache the 'clean' response (for client not supporting gzip)
route = ^/$ cachestore:key=werkzeug_homepage
; then pass the response to the gzip transformation
route = ^/$ gzip:
; and cache it again in another item (gzipped)
route = ^/$ cachestore:key=werkzeug_homepage.gz

Another common transformation is applying stylesheets to XML files. (see The XSLT plugin)

The toxslt transformation is exposed by the xslt plugin:

uwsgi --plugin xslt --http-socket :9090 -w mycd --route-run "toxslt:stylesheet=t/xslt/cd.xml.xslt,params=foobar=test&agent=\${HTTP_USER_AGENT}"

The mycd module here is a simple XML generator. Its output is then passed to the XSLT transformation.

Streaming vs. buffering

Each transformation announces itself as a “streaming” one or a “buffering” one.

Streaming ones are transformations that can be applied to response chunks (parts). An example of a streaming transformation is gzip (you do not need the whole body to begin compressing it). Buffering transformations are those requiring the full body before applying something to it. XSLT is an example of buffering transformation. Another example of buffering transformations are those used for storing response in some kind of cache.

If your whole pipeline is composed by only “streaming” transformations, your client will receive the output chunk by chunk. On the other hand a single buffering transformation will make the whole pipeline buffered, so your client will get the output only at the end.

An often using streaming functionality is gzip + chunked:

plugins = transformation_gzip,transformation_chunked
route-run = gzip:
route-run = chunked:

The whole transformation pipeline is composed by streaming plugins, so you will get each HTTP chunk in realtime.

Flushing magic

The “flush” transformation is a special one. It allows you to send the current contents of the transformation buffer to the client (without clearing the buffer).

You can use it for implementing streaming mode when buffering will be applied. A common example is having streaming + caching:

plugins = transformation_toupper,transform_tofile
; convert each char to uppercase
route-run = toupper:
; after each chunk converted to upper case, flush to the client
route-run = flush:
; buffer the whole response in memory for finally storing it in a file
route-run = tofile:filename=/tmp/mycache

You can call flush multiple times and in various parts of the chain. Experiment a bit with it…

Available transformations (last update 20130504)

  • gzip, exposed by the transformation_gzip plugin (encode the response buffer to gzip)
  • toupper, exposed by the transformation_toupper plugin (example plugin transforming each character in uppercase)
  • tofile, exposed by the transformation_tofile plugin (used for caching to response buffer to a static file)
  • toxslt, exposed by the xslt plugin (apply xslt stylesheet to an XML response buffer)
  • cachestore, exposed by the router_cache plugin (cache the response buffer in the uWSGI cache)
  • chunked, encode the output in HTTP chunked
  • flush, flush the current buffer to the client
  • memcachedstore, store the response buffer in a memcached object
  • redisstore, store the response buffer in a redis object
  • template, apply routing translations to each chunk

Working on

  • rpc, allows applying rpc functions to a response buffer (limit 64k size)
  • lua, apply a lua function to a response buffer (no limit in size)