Happstack

new tutorial on acid-state, ixset, and lenses

2012-02-01T13:16:00.000-06:00

Hello,

I just uploaded a new tutorial which covers three distinct, but related topics:

acid-state - a native Haskell, noSQL, RAM-cloud database
IxSet - a multi-indexed collection type (similar to Data.Map but with support for multiple keys)
data-lens - a library which provides syntax that makes it easier to update nested records and other data types

The tutorial is available here:

http://happstack.com/docs/crashcourse/AcidState.html

The section on lenses is a great applied introduction to the concept which requires no understanding of Happstack, acid-state, or IxSet. So, even if you don't care about acid-state, you may find it useful if you have ever wondered what the heck lenses are.

ANN: happstack-server 6.5.1

2012-01-22T19:00:00.000-06:00

I am pleased to announce the release of happstack-server 6.5.1. Changes include:

support for https:// (aka, SSL/TLS) https://groups.google.com/group/happs/browse_thread/thread/d92c24a7a81adfe8
directory listings now sorted in alphabetical order
added Happstack class instances for ReaderT, WriterT, StateT, and RWST
happstack-server now compiles without producing any warnings
various code cleanup and formatting

Discussion here:

http://www.reddit.com/r/haskell/comments/os5kb/ann_happstackserver_651_https_aka_ssltls_support/

happstack.com: now less evil

2012-01-14T11:35:00.002-06:00

If you experienced some outage today trying to reach happstack.com, it is because we very ungracefully transfered our domain from the evil pro-SOPA godaddy to the less-evil anti-SOPA namecheap. Yay!

You are probably already familiar with SOPA/PIPA, but if not, here is some more information:

http://lifehacker.com/5860205/all-about-sopa-the-bill-thats-going-to-cripple-your-internet

The DNS changes have probably propagated to you by now, but if not, they should in the next 24 hours.

ANN: happstack-lite - a friendly introduction to Happstack

2011-11-14T20:23:00.001-06:00

Happstack is a powerful Haskell web framework with a rich API that has
evolved over the last 7 years to meet the needs of real world web
development.

Unfortunately a rich and flexible API can also be daunting when all
you need is something simple. What many people don't realize is that
inside Happstack lives a very simple and easy to use web framework.

I am pleased to announce a new member of the Happstack family:

happstack-lite.

happstack-lite is minimal version of Happstack that is designed give
you everything you need to get started with web develop in a much
cleaner, easier package. It is an ideal starting point for novice Haskell programmers or developers who are new to the Happstack family.

We have done that by:

1. collecting the most essential types and functions into a single, well-documented module

2. giving all the functions simple type signatures. No crazy monad
transformers, far fewer type classes, etc.

3. creating a new, small tutorial (less than 2000 words) that shows
you how to use all the features of happstack-lite.

But, the best part is that happstack-lite is 100% compatible with regular Happstack. If you find you do need some of the extra features that Happstack offers, you can simply import the extra functions and use them. Migrating from happstack-lite to happstack is trivial -- mostly just changing `import Happstack.Lite` to `import Happstack.Server`. So, you are not locked into staying with happstack-lite if you decide you need more. And you can still use the many addon packages such as happstack-hsp or happstack-jmacro if you want.

happstack-lite is not crippled or underpowered. In fact, it offers features and performance on par with other Haskell frameworks while being as simple as possible. If you have ever looked at Happstack in the past and felt it was too complicated -- I would highly encourage to look again. Especially if the last time you looked was prior to Happstack 6.

Check out the tutorial here: happstack-lite.

We would love to hear your feedback and criticisms. Please see the related discussion at:

http://www.reddit.com/r/haskell.

type-safe urls via web-routes, part 2: Boomerang

2011-07-21T13:18:00.000-05:00

web-routes is a system for implementing type-safe web-routes in Haskell web applications. In part 1, we saw the basics of how to use web-routes.

In part 1, we used template haskell to automatically derive the mapping between the url string and the url type. This makes development very easy, but the automatically derived urls are not all that visually appealing.

In this part, we introduce a way to manually specify the mapping between the url type and the url string. This gives the developer 100% control over the mapping. Up to now, the problem with creating a manual mapping has been that you had to implement the specification twice. You had to write a show function to turn a url type into a url string, and a read function to turn a url string into a url type. These functions are basically inverses of each other, both following the same spec.

What we really want to do is write the spec once and automatically extract the printer and parser from the spec. That way we can be sure they are inverses of each other, and we do not have to repeat ourselves. And we do not have to repeat ourselves. Because that is annoying.

Fortunately, Sjoerd Visscher and Martijn van Steenbergen figured out exactly how to do that and published a proof of concept library know as Zwaluw. With permission, I have refactored their original library into two separate libraries: boomerang and web-routes-boomerang.

boomerang is a general purpose, standalone invertible parser library. web-routes-boomerang adds glue code around boomerang so that it can be used to with web-routes.

Using boomerang we can take a route like this one:

> data Sitemap
>     = Home
>     | Article ArticleId
>     | UserOverview
>     | UserDetail Int String
>       deriving (Eq, Ord, Read, Show, Data, Typeable)
>

a specify a route map like this:

> sitemap :: Router Sitemap
> sitemap =
>     (  rHome
>     <> rArticle . (lit "article" </> articleId)
>     <> lit "users" . users
>     )
>     where
>       users =  rUserOverview
>             <> rUserDetail </> int . lit "-" . anyString

which defines a mapping like this:

url		type
/	<=>	Home
/article/int	<=>	Article int
/users	<=>	UserOverview
/users/int-string	<=>	UserDetail int string

The sitemap function looks like a ordinary parser. But, what makes it is exciting is that it also defines the pretty-printer at the same time.

For more in-depth details, see the relevant section of the Happstack Crash Course. If you just want to play with boomerang then start here.

type-safe urls via web-routes, part 1

2011-07-11T15:14:00.000-05:00

web-routes is a suite of libraries which provide "type-safe" web routing. Instead of working directly with url strings in your web application, you create types to represent the routes.

The web-routes library has been around for several years. The idea itself is fairly obvious to many Haskell programmers, and so the idea has been rediscovered many times.

web-routes is designed to be exceedingly flexible. It is not explicitly tied to any haskell web framework, templating system, etc. It forms the basis of type-safe urls in yesod. There is also support for Happstack and HSP.

web-routes does not impose any particular method of mapping the url types to strings. We provide a variety of ways to defining mappings via addons including quasi-quotation, template haskell, generics, parsec, and more. If none of these systems suit your fancy, you can easily provide your own system. You basically need to provide two functions:

urlToString :: url -> [(String, String)] -> [String]
stringToUrl :: [String] -> Either String url

If you have an idea for how to map urls to types, I would encourage you to consider basing it on web-routes. A significant amount of effort has gone into handling issues like unicode and url escaping (among other things). If you use web-routes you do not need to know *any* of those rules. web-routes takes care of all that nastiness for you. If you find that web-routes is not suitable for your needs, I would love to hear why and see if we can not fix it.

The biggest issue with web-routes has been the lack of good documentation. So, I am pleased to announce that the first in a series of tutorials on web-routes is now online at: http://happstack.com/docs/crashcourse/WebRoutes.html.

Comments, questions, and corrections appreciated!

Happstack + JMacro + HSX

2011-05-23T15:22:00.000-05:00

Happstack (a Haskell web framework) has long had support for using HSX so that you can use XML syntax in your Haskell page templates. Happstack now has support for using JMacro to add javascript to your templates.

JMacro was written by Gershom Bazerman, and was first released in July of 2009. Our contribution is the creation of two glue libraries:

hsx-jmacro: a thin wrapper which makes it easy to embed JMacro into HSX (independent of Happstack).
happstack-jmacro: another thin wrapper that makes it easy to generate and serve JMacro .js scripts using Happstack

The use of Happstack+HSX+JMacro has been documented in detail in this section of the Happstack Crash Course.

Here is a little example of what Happstack+HSX+JMacro looks like:

> helloJMacro :: JMacroPart Response
> helloJMacro =
>     toResponse <$> defaultTemplate "Hello JMacro" ()
>       <div>
>        <% [$jmacro| 
>            var helloNode = document.createElement('h1');
>            helloNode.appendChild(document.createTextNode("Hello, JMacro!"));
>            document.body.appendChild(helloNode);
>            |] %>
>       </div>
>

As you can see, the syntax used by JMacro is almost identical to JavaScript. So, you do not have to learn some special DSL to use it. In fact, JMacro can work with most JavaScript you find in the wild. Using JMacro has a number of advantages over just using plain-old javascript:

syntax checking ensures that your JavaScript is syntactically valid at compile time. That eliminates many common JavaScript errors and reduces development time.
hygienic names and scoping automatically and transparently ensure that blocks of JavaScript code do not accidentally create variables and functions with conflicting names.
Antiquotation, marshalling, and shared scope make it easy to splice Haskell values into the JavaScript code. It also makes it easy to programmatically generate JavaScript code.

The last point means that instead of trying to concat strings to call JavaScript functions with Haskell values like this:

hello str = <button onclick=("alert(" ++ str ++ ");")>click me</button>

You can write:

hello str = <button onclick=[$jmacro| alert(`(str)`); |]>click me</button>

The first version has a major bug -- it does not properly escape the str which can lead to javascript injection attacks. The JMacro version, on the other hand, automatically escapes the string.

You are not limited to just splicing strings and other primitive types into the JavaScript code. By creating an instance of ToJExpr you can easily pass any Haskell value to JavaScript.

While JMacro can easily be used for simple templating, it is actually a very powerful tool for generating JavaScript. It is essentially Template Haskell for JavaScript. It parses the JavaScript into an abstract syntax tree, and supports quasiquotation and antiquotation.

To learn more about using JMacro with Happstack and HSX, start reading here.

And a big thanks for Gershom Bazerman for creating JMacro. It's not part of Happstack or HSX, so you should have no problems using it with other web frameworks or templating libraries.

MACID Haskell persistent data store overhauled

2011-05-02T13:18:00.001-05:00

Over the past few weeks Lemmih has been working on a major overhaul of MACID. MACID is a persistent data store explicitly designed for use with Haskell. It can store arbitrary Haskell data structures and queries are written in straight-forward Haskell (no funky DSLs). It is thread-safe, and supports the classic ACID properties. Unplug your machine with out losing your data.

The homepage for the new MACID library can be found here.

There are two new libraries acid-state and safecopy, which will replace happstack-data and happstack-state in Happstack 7.

safecopy provides versioned binary serialization of datatypes with version migration. It adds this functionality on top of the cereal library.

acid-state provides ACID transactions on top of safecopy.

What's New

The new MACID is like the old MACID, but better. The fundamental concepts are the same, but it is a lot cleaner, more robust and less magical.

The rewrite of MACID addresses many long standing wish-list items, including:

MACID now a separate project

acid-state and safecopy are now free standing libraries, with no references or dependencies on anything Happstack. They have their own maintainer, homepage, source repository, etc. So, now you can used MACID, even if you don't use Happstack. (This has actually been true for a while, but now it is a lot more obvious).

If all you want is versioned binary serialization and migration, then you can use the safecopy library on its own.

update / query take an explicit state handle now

In happstack-state, query and update accessed the state via a global IORef. This meant that you could only have one instance of MACID per application. It also weakened type-checking resulting runtime errors that should have been avoidable.

It also meant that you could only have one ACID store per application.

Now update and query take an explicit handle.

No more Component class

happstack-state has a Component class, which is mostly worthless. It gives the promise of power, but doesn't really deliver anything. It is just extra boilerplate for the most part. This class is gone entirely in acid-state.

Less boilerplate for serialization instances

In happstack-data, creating a Serialization instance for a type required you to call $(deriveSerialize ''Foo) and you have to create an instance Version Foo. With safecopy, you only need one line, '$(deriveSafeCopy 0 'base ''Foo)'

In happstack-data there were three type classes, Serialize, Version, and Migrate. This has been simplified down to just two in safecopy: SafeCopy and Migrate.

Better Safety

happstack-state has a number of corner cases that it does not handle gracefully. For example, bad things happen if update events call fail or error. acid-state handles these correctly and includes a test suite specifically for testing these types of failures.

Additionally, acid-state is a lot better at flushing data to disk.

Performance

Initial performance testing shows that acid-state is fast. A simple data store that holds an integer which is incremented by an update event was able to achieve 13,300 updates per second.

Still Coming

There is still a bunch of work to come.

acid-state currently does not support Windows. This is because the code to safely flush data to disk requires the posix library. Windows support can be added via some #ifdefs. If you are familiar with Windows and file IO and want to help, your contribution will be greatly appreciated.

Migrating data from happstack-state to acid-state is certainly feasible, but the process is not yet documented. But it will be.

There are also major improvements to IxSet planned.

And, of course, replication and sharding. Fortunately, it will be easier to implement these on top of the new acid-state code base.

Get Started Today

acid-state is usable today! You can install it from hackage. There are code examples here and here. If you are starting a new MACID based project, you are encouraged to consider acid-state.

To use happstack-ixset you will currently need to add this extra instance to your code:

import Data.SafeCopy
import Happstack.Data.IxSet            (IxSet)
import qualified Happstack.Data.IxSet  as IxSet

instance (SafeCopy a, Ord a, Typeable a, IxSet.Indexable a) => SafeCopy (IxSet a) where
    putCopy ixSet = contain $ safePut (IxSet.toList ixSet)
    getCopy = contain $ IxSet.fromList <$> safeGet

Once IxSet is factored out of happstack into a separate library that instance will be provided automatically.

It is safe to mix happstack-state and acid-state in the same application if you want to use acid-state in an existing application.

More updates to come as things develop!

ANN: happstack-heist now available

2010-11-15T15:48:00.001-06:00

Thanks to cdsmith's blog post, and Happstack's recent update to mtl-2, I am pleased to announce happstack-heist.

Detailed documentation on using Heist with Happstack is available in the Happstack Crash Course.

Happstack is a flexible Haskell Web Framework with many supporting optional components.

Heist is an XML templating engine. The static portions of your templates are written in XML files which are loaded by the server at runtime. This makes it easy to modify the templates without having to recompile and restart the server. It also makes it easy to work with template designers who do not know Haskell.

The dynamic portions of the templates are generated in Haskell and are spliced into the templates. This means you have the full expressive power of Haskell at your disposal for the generated portions of the templates. That is a lot nicer than trying to using something like XSLT (though Happstack does support that as well).

Happstack offers a wide variety of templating solutions including BlazeHtml, HSP, Hamlet, and HStringTemplate. But Heist fills a nice hole in the spectrum, and we are pleased to be able to offer it now.

It is available in darcs and on hackage. It has been tested against Happstack from darcs, but should work against Happstack stable as well.

If there are any bugs or improvements you would like to see, let us know!

- jeremy

Recompile your Haskell-based templates faster than you can hit F5.

2010-10-20T20:55:00.000-05:00

There are two main classes of templating solutions in Happstack:

1. DSLs/libraries such as BlazeHtml, HSP, Hamlet, etc, where your templates are written in Haskell and compiled at compile time.

2. Libraries like heist, HStringTemplate, etc, where your templates are written in some external template file and read at runtime by the server.

Each method has strengths and weaknesses -- and so each project needs to pick the solution that works best for them.

For my projects I love using HSP. I like having the full expressive power of Haskell in my templates, and the added safety that the type checker provides. But I hate having to recompile, relink, and restart my app server dozens and dozens of times when I am developing my templates. And, so it is with great pleasure that I present the triumphant return of happstack-plugins!

happstack-plugins

happstack-plugins leverages the recently revived plugins package so that individual page templates can be automatically recompiled and reloaded into a running happstack application. happstack-plugins uses hinotify to watch the haskell source files containing your page templates. Whenever you save changes, the page is automatically recompiled and reloaded into the running server. Typically this happens fast enough that by the time you switch to the browser and hit reload, the updated page is already available.

You can see a demo of happstack-plugins in action here:

How to use happstack-plugins

Using happstack-plugins is very straight-forward. First you need to install the happstack-plugins library which is currently only available in the happstack darcs repository:


darcs get http://patch-tag.com/r/mae/happstack

For best performance you should put each page template in its own module so that it can be recompiled and reloaded faster.

The templates themselves require no special modifications. Here is a simple helloPage template:

> module HelloPage where
>
> import Happstack.Server
>
> helloPage :: String -> ServerPart Response
> helloPage noun = ok (toResponse $ "hello, " ++ noun)

This template takes a single String argument and returns a text/plain page which says, "hello, <string>". We could just as well use BlazeHTML, HSP, etc, but using String keeps this example short and simple.

As I mentioned, there is nothing new going on here, it just a normal happstack ServerPart.

The interesting changes are in the Main module. There are only 3 simple changes required to support templates. But first, some boring stuff at the top of the module:

> {-# LANGUAGE CPP, TemplateHaskell #-}
> module Main where
> 
> import Control.Monad     (msum)
> import Happstack.Server

1. Here we #ifdef some module imports. These two modules provide the same interface. The Dynamic version actually does page recompilation and reloading. The Static version just links things in the normal way. This makes it easy to use dynamic loading during development but static linking for the live server by simply defining or undefining PLUGINS.

> #ifdef PLUGINS
> import Happstack.Server.Plugins.Dynamic
> #else
> import Happstack.Server.Plugins.Static
> #endif
> import HelloPage

2. In main we call initPlugins which starts the recompiler/reloader and hinotify. If you import Happstack.Server.Plugins.Static, initPlugins is a 'noop', so we do not have to add any extra #ifdefs.

> main :: IO ()
> main =
>    do ph <- initPlugins
>       simpleHTTP nullConf $ pages ph

3. Here is where we actually specify a template to load dynamically:

> pages :: PluginHandle -> ServerPart Response
> pages ph =
>    msum [ $(withServerPart 'helloPage) ph $ \helloPage ->
>               (helloPage "hello")
>         ]

Normally we would just have:

> pages :: PluginHandle -> ServerPart Response
> pages ph =
>    msum [ helloPage "world"
>         ]

So the new part is the template haskell function withServerPart which effectively takes three arguments:

1. the name of the symbol to dynamically load
2. the PluginHandle which initPlugins returned
3. a function which will use the loaded symbol

so, withServerPart effectively has the type:

> withServerPart :: (MonadIO m, ServerMonad m) => Name -> PluginHandle -> (a -> m b) -> m b

Even though we are dynamically reloaded the page at runtime, the compiler will still check that the types are correct when will compile the main application.

If we change helloPage "hello" to helloPage 1 and try to build Main.lhs we will get the error.


Main.lhs:50:28:
    No instance for (Num String)
      arising from the literal `1' at Main.lhs:50:28
    Possible fix: add an instance declaration for (Num String)
    In the first argument of `helloPage', namely `1'
    In the expression: (helloPage 1)
    In the second argument of `($)', namely
        `\ helloPage -> (helloPage 1)'
Failed, modules loaded: HelloPage.

What's left to do?

There are two big features on the TODO list. If you think happstack-plugins is cool, I encourage you to work on them!

1. The underlying plugins library is broken when it comes to hierarchical modules. Ideally I would put all the pages in Pages.*. For example Pages.HelloPage. But, that does not work. As a hack, you can modify System.Plugins.Make.build and comment out output in the let flags = ... declaration. This fixes hierachical modules, but requires you to run your app with its working directory set to the root directory of your project. That is fine for happstack app development, but not an ideal solution for all users of the plugins library. If someone could fix hierarchical module support in plugins, that would be great for everyone.

2. hinotify is only supported under Linux. However, it should not be that hard to make hinotify support optional (via a compile time flag). With out hinotify, we would just do a quick stat() everytime the template is invoked and see if a recompilation is needed. When a compilation is needed, you will have to wait for that page to recompile and reload -- but it will still be much faster than rebuilding and restarting the whole server.

Is the RqData monad still needed?

2010-10-13T19:39:00.000-05:00

cdsmitch recently asked if RqData is really needed in Happstack. The answer is, "no, but it is still useful sometimes."

I can say "no" with certainty because in the darcs version of Happstack, it is already optional.

The new and improved RqData

Functions like look now work in any monad which is an instance of HasRqData:

> look :: (Functor m, Monad m, HasRqData m) => String -> m String

Since there is a HasRqData instance for ServerPart, we effectively have the function:

> look :: String -> ServerPart String

Here is an example of using look with out having to jump through any hoops:

> module Main where
>
> import Happstack.Server (ServerPart, look, nullConf, simpleHTTP, ok)
>
> helloPart :: ServerPart String
> helloPart =
>     do greeting <- look "greeting"
>        noun     <- look "noun"
>        ok $ greeting ++ ", " ++ noun
>
> main :: IO ()
> main = simpleHTTP nullConf $ helloPart

Now if we visit http://localhost:8000/?greeting=hello&noun=rqdata, we will get the message hello, rqdata

Sweet!

But why keep RqData around?

Using look in the ServerPart monad is simple. But when it fails, it just calls mzero. That can be very frustrating if you are debugging your forms or debugging calls to your web service API. Instead of an error telling you what parameter was missing, you simply get a generic 404 error.

Using the RqData monad/applicative functor gives you the option to provide detailed error messages when something goes wrong:

> module Main where
>
> import Control.Applicative ((<$>), (<*>))
> import Happstack.Server (ServerPart, badRequest, nullConf, ok, simpleHTTP)
> import Happstack.Server.RqData (RqData, look, getDataFn)
>
> helloRq :: RqData (String, String)
> helloRq =
>     (,) <$> look "greeting" <*> look "noun"
>
> helloPart :: ServerPart String
> helloPart =
>     do r <- getDataFn helloRq
>        case r of
>          (Left e) -> 
>              badRequest $ unlines e
>          (Right (greet, noun)) ->
>              ok $ greet ++ ", " ++ noun
>
> main :: IO ()
> main = simpleHTTP nullConf $ helloPart

If you visit http://localhost:8000/?greeting=hello&noun=world, you will get the familiar greeting hello, world.

But if you leave off the query parameters http://localhost:8000/, you will get a list of errors:


Parameter not found: greeting
Parameter not found: noun

This is really nice when you are debugging your code.

Now with more composability!

Since RqData and ServerPart are instances of Applicative and Alternative you can now reuse many functions from those libraries. For example, if a query parameter is optional, you can simply write:

>     do greet <- optional $ look "greeting"

There is also a new combinator checkRq which can be used to validate query parameters, or to convert a query parameter to another type:

> checkRq :: (Monad m, HasRqData m) => m a -> (a -> Either String b) -> m b

If you are curious be sure to check out the Happstack Crash Course where the new RqData module is documented in detail with many working examples.

I would love to hear feedback on the new and improved RqData module, and any suggestions for improvement!

Also, be on the look out for a future blog post about the RqData Arrow. :)

Changes to request body and RqData in head

2010-07-19T12:08:00.000-05:00

I have just pushed some patches which affect the way the Request body and RqData are handled in happstack 0.6. This contains user visible changes which will affect you if you:

Use RqData

Directly use the rqBody field in Request

Directly use the rqInput field in Request

Directly work with the Input type

Allow file uploads

Some of the changes fix bugs (design flaws), and others are for new features and functionality. The non-compatible API changes are pretty small, so it should be easy to port your code. It basically comes down to:

getDataFn, withDataFn, etc take an extra argument of the type BodyPolicy

getDataFn, withDataFn, etc return Either [String] a instead of Maybe a

the inputValue field of the Input type is now Either FilePath L.ByteString instead of L.ByteString

you have to explicitly import the module Happstack.Server.RqData

In this post I will describe what motivated these changes. I am
hoping to also get feedback and these changes before we release 0.6 since it will be less painful to make further changes now.

the Request body and space usage

In the old code the Request type stores the request body as a simple lazy ByteString:

> newtype RqBody = Body { unBody :: L.ByteString } deriving (Read,Show,Typeable)
> 
> data Request = Request { ...
>                        , rqBody        :: RqBody
>                        }

This feels nice, because it is a simple, pure value. Unfortunately, it is really not a great idea in practice. The request body does not initially require any space, because it is an unevaluated lazy ByteString. But the ServerPart holds the Request in its environment, and that means the garbage collection can not free the RqBody as you evaluate it. If the request body contained gigabytes of data, that could be disastrous.

The solution in Happstack 0.6 is to use an MVar to hold the request body:

> 
> data Request = Request { ...
>                        , rqBody        :: MVar RqBody
>                        }

Instead of using rqBody directly, it is better to use takeRequestBody, so that your code will not break if we switch to IORef or something else.

> takeRequestBody :: Request -> IO (Maybe RqBody)
> takeRequestBody rq = tryTakeMVar (rqBody rq)

Now, when you process the RqBody the Request will not be holding onto it, so the garbage collection can free it (assuming your code to not hold onto it and introduce a new space leak).

This does have a drawback however. A ServerPart can call mzero at anytime, and processing will move onto the next ServerPart. However, if you have already taken the RqBody then the next ServerPart may be missing critical data it needs. But, if we left the RqBody intact, that would result in the space leak. I think that in practice, if a ServerPart made enough progress that it started consuming the RqBody and then failed, it is unlikely that another ServerPart would succeed and need the RqBody. If another ServerPart succeeds, it is probably just a 404 Not Found handler or something similar, which does not need the request body. So it seems like it is better to have the default behavior be the more space friendly solution.

We will also provide peekRequestBody and/or putRequestBody functions so that you can opt to leave the request body intact. It is up to you to be sensible about using them.

`BodyInput` and space usage

In RqData, the cookies, QUERY_STRING, and request body (when appropriate) are parsed into a [(String, Input)], where String is the name of the key, and Input is the value.

In Happstack 0.6, Input will be the type:

> data Input = Input
>     { inputValue       :: Either FilePath L.ByteString
>     , inputFilename    :: Maybe FilePath
>     , inputContentType :: ContentType
>     } deriving (Show,Read,Typeable)

In Happstack 0.5 the inputValue is simply a L.ByteString. Once again, this seems fine at first. After all, the inputValues are lazy ByteString, so we can process them lazily, right? Well, not quite. In the unprocessed request body, the key/value pairs are laid out like this:


key1
value1
key2
value2
key3
value3
key4
value4
...

If we were to consume the key/value pairs in a sequential manner, then we would be ok. But, generally we want to use functions which can lookup a specific key. Imagine we want to look up key4. In order to do that we have to first read in all the preceding key/value pairs. If we knew we only cared about key4 then we could just toss the rest. But with the monadic RqData code we don't know that. (A future post will talk about an arrow based alternative where we do know that). So, we have to store all the key/value pairs in case we want to lookup key1 after key4.

In Happstack 0.5, we store all those values in RAM. But, some of those values might be (huge) files. That clearly isn't going to work. So we once again trade off a bit a simplicity/elegance for the practical matter of not having unlimited amounts of RAM. Instead we store some values in RAM and some values on the disk. How do we decide what goes where? That brings us to BodyPolicy.

`BodyPolicy`

When parsing the request body, we need some way to decide what values should be stored in RAM and what values should be saved to disk. Additionally, we want to impose limits on how much data can be stored in either location. If a user decides to post the contents of /dev/random you are likely to want to cut them off at some point. However, the specific values for the quotas are application specific. In fact, they may be specific to the particular form that is being processed. For example, an admin user might have higher quotas than a regular user.

The answers to these questions are provided by the BodyPolicy, which looks like:

> data BodyPolicy 
>    = BodyPolicy { inputWorker :: Int64 -> Int64 -> Int64 -> InputWorker
>                 , maxDisk     :: Int64 -- ^ maximum bytes to save to disk (files)
>                 , maxRAM      :: Int64 -- ^ maximum bytes to hold in RAM 
>                 , maxHeader   :: Int64 -- ^ maximum header size (this only affects headers in the multipart/form-data)
>                 }

The inputWorker is the function that actually decides where values should be saved, and implements the quotas. Its Int64 arguments are the quotas for the disk, ram, and other headers which don't really get saved, but which can temporarily take up space. The next three fields are the values to pass to the inputWorker.

In most cases, you do not need to write you own inputWorker. It is sufficient to use the defaultBodyPolicy:

> defaultBodyPolicy :: FilePath -> Int64 -> Int64 -> Int64 -> BodyPolicy

The first argument is the directory to store temporary files in, and the next three arguments are the quota values. I am not going to cover defaultBodyPolicy in detail in this post. But it is well documented in the Happstack Crash Course.

Improvements to `RqData`

The new RqData module also includes a number of new features.

There is now an Applicative functor instance for RqData. The applicative functor instance accumulates errors. This means if you try look up multiple invalid keys, the error message will report all the missing values, not just the first one. This is nice when you are debugging your code, and is also nice if you provide a web service (REST API, etc) and want to provide your API users with detailed error messages instead of "Invalid Request".

We now provide two filters (body and queryString) which limit the scope of the look* functions to either the request body or the QUERY_STRING.

A new function lookFile is provided to assist with handling file uploads.

A new function checkRq is provided to help you convert
request parameters to Haskell types, or to check that a value meets some conditions.

Summary

This post gives some of the background on the changes to how we handle the request body and form data. To actually see what the changes look like in practice, you should check out the RqData section in the Happstack Crash Course. It gives detailed examples of all the features and changes I talked about in this post. I have also updated the haddock documentation in darcs.

I would love to hear your opinions. Do you love the changes? Hate the changes? Have better ideas about how to solve the problems? In terms of handling the raw request body, I believe both Yesod and Snap use the same basic approach -- the first handler to try to use the request body gets the whole thing, and everyone else gets nothing. (And they provide ways to put the request body back if you want to..).

sendfile 0.7.1

2010-07-11T16:12:00.000-05:00

I have just uploaded sendfile 0.7.1 to hackage.

The sendfile library exposes zero-copy sendfile functionality in a portable way. If a platform does not support sendfile, a fallback implementation in Haskell is provided. It currently has zero-copy support for Linux, Darwin, FreeBSD, and Windows.

The sendfile functionality typically reduces CPU-load and (possibly) increases IO throughput.

The new release of sendfile adds the ability to hook into the send loop. This is useful if you want to tickle timeouts or update a progress bar while the file is being sent.

This turned out to be rather tricky because each platform implements sendfile a little differently. But, the point of the sendfile library is to provide a unified interface so that other developers do not have to know any of the platform specific details.

The solution in 0.7.1 is to use a simple, specialized iteratee. Each pass of the sendfile loop can end in one of three states:

(1) the requested number of bytes for that iteration was sent
successfully, there are more bytes left to send.

(2) some (possibly 0) bytes were sent, but the file descriptor
would now block if more bytes were written. There are more bytes
left to send.

(2) All the bytes were sent, and there is nothing left to send.

We handle these three cases by using a type with three
constructors:


 data Iter
     = Sent       Int64    (IO Iter)
     | WouldBlock Int64 Fd (IO Iter)
     | Done       Int64

All three constructors provide an Int64 which represents the
number of bytes sent for that particular iteration. (Not the total
byte count).

The Sent and WouldBlock constructors provide IO Iter as their
final argument. Running this IO action will send the next block of
data.

The WouldBlock constructor also provides the Fd for the output
socket. You should not send anymore data until the Fd would not
block. The easiest way to do that is to use threadWaitWrite to
suspend the thread until the Fd is available.

A very simple function to drive the Iter might look like:


runIter :: IO Iter -> IO ()
runIter iter =
   do r <- iter
      case r of
        (Done _n)      -> return ()
        (Sent _n cont) -> runIter cont
        (WouldBlock _n fd cont) -> 
            do threadWaitWrite fd
               runIter cont

You would use it as the first argument to a *IterWith function, e.g.


 sendFileIterWith runIter outputSocket "/path/to/file" 2^16

If we want to do something fancier, such as update timeouts or a progress bar, we can do it in a custom runIter function. If we are using a non-standard I/O manager, we might be able to suspend the thread via a call other than threadWaitWrite.

What Next?

The new version of sendfile will be used to improve the timeout handling in the Haskell web framework, Happstack.

It would be nice if the sendfile library could export a low-level function like:


sendfile :: Fd -> Fd -> Int64 -> Int64 -> IO (Bool, Int64)

It would take the output socket, and input file descriptor, an offset, and length, and return the number of bytes written, and whether the output socket blocked.

Unfortunately, it is not possible to provide a portable implementation of this sendfile function. That would require functions which can operate directly on the Fds. But those functions live in the unix package, which is not portable.

Another non-solution is to have a module like, Network.Socket.SendFile.LowLevel which is only exported on the platforms which provide a low-level sendfile implementation. However, it is my understanding that this is not really allowed by the cabal policy because there would be no way to specify that you require a version of the sendfile library that exports .LowLevel.

So, I believe a more correct solution is to create a *new* package, sendfile-lowlevel, which exports Network.Socket.SendFile.LowLevel. This assumes that there is some way to mark that a package is only available on certain platforms. However, I am not sure if that can be done.

Hopefully the new API provides enough flexibility that there is no need for an even lower-level API to be exposed. If you think you need something lower-level, let me know, and let's see if we can work something out.