Following up to the previous post, a couple of people pointed out to me that the cjson library is faster than simplejson, and that the latest simplejson has a small C extension.
Re running the tests with simplejson 2.0.9 and cjson yields the following results:
5000 total records (0.730s) get_thrift (0.040s) get_pb (0.620s) ser_thrift (0.550s) 555125 bytes ser_pb (2.980s) 415125 bytes ser_json (0.030s) 718455 bytes ser_cjson (0.040s) 718455 bytes ser_yaml (12.770s) 623455 bytes ser_thrift_compressed (0.630s) 287621 bytes ser_pb_compressed (3.020s) 284441 bytes ser_json_compressed (0.090s) 293073 bytes ser_cjson_compressed (0.080s) 293073 bytes ser_yaml_compressed (13.260s) 291106 bytes serde_thrift (1.460s) serde_pb (5.250s) serde_json (0.070s) serde_cjson (0.060s) serde_yaml (44.110s)
There doesn't seem to be much doubt about it, if you need to serialize basic python data structures and don't need the extra features of thrift or protocol buffers, it is hard to beat JSON.
The test code at sertest.tgz has also been updated to use time.clock instead of time.time.