I have been play­ing around with dupli­cat­ing anoth­er program’s web ser­vice API. I want to be able to act as a sub­sti­tute but need to rewrite it in a new lan­guage and change a bunch of its fea­tures. To do this I have been run­ning the orig­i­nal program’s func­tion­al tests against my own code.

Unfor­tu­nate­ly it is some­times dif­fi­cult to see exact­ly what is going on. The tests are writ­ten in Python, which I am not very famil­iar with, and are wrapped through var­i­ous lay­ers adding dif­fer­ent head­ers.

Hap­pi­ly Apache sup­ports log­ging with mod_dumpio so I can run the tests until they fail and then sim­ply look at the last request. That was the plan at least, but a sin­gle call gen­er­ates six­ty lines of log. Much of it look­ing like the sam­ple below, obvi­ous­ly the mid­dle line is con­tent the oth­er two are noise.

[Thu Jan 15 10:06:08.118940 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(63): [client ::1:53956] mod_dumpio: dumpio_out (data-HEAP): 1 bytes
[Thu Jan 15 10:06:08.118945 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(103): [client ::1:53956] mod_dumpio: dumpio_out (data-HEAP): 0
[Thu Jan 15 10:06:08.118952 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(63): [client ::1:53956] mod_dumpio: dumpio_out (metadata-EOS): 0 bytes

A quick search online threw up dumpio_parser.pl which extracts the requests from log files. How­ev­er it works for Apache 2.2, not the slight­ly mod­i­fied for­mat in Apache 2.4.

So I rewrote the input por­tion of dumpio_parser.pl.

Then the out­put was in a mar­gin­al­ly use­ful for­mat, it essen­tial­ly pro­vid­ed the same con­tent lines in the log. I want­ed a for­mat I could throw at curl to retry the request and see if my bug fix had worked.

Fix­ing that I was still not sat­is­fied. I didn’t like the out­put struc­ture, it cre­at­ed a col­lec­tion of tiny files, one for each request. And it ignored the out­put from the serv­er. AND the struc­ture was weird, first pro­gram I have seen using perl -n.

So I rewrote it.

Pre­sent­ing dumpio2curl.

> tail -n 100 apache.log | perl ../dumpio2curl/dumpio2curl.pl

# Tue Jan 13 10:23:22.729122 2015 — pid 15952 — client 50304
curl -v \
–head­er “User-agent:” –head­er “Accept:” \
–request “DELETE” \
–head­er “Accept-Encod­ing: iden­ti­ty” \
–head­er “Con­tent-Length: 0” \
–head­er “X-For­ward­ed-For:” \
–head­er “X-For­ward­ed-Script-Name: ” \
–head­er “X-For­ward­ed-Scheme: http” \
–head­er “X-Wsgiproxy-Ver­sion: 0.1” \
–head­er “X-For­ward­ed-Serv­er: localhost:88” \
–head­er “Autho­riza­tion: Hawk …” \

# Tue Jan 13 10:23:22.752411 2015 — pid 15952 — client 50304
# HTTP/1.1 200 OK
# Date: Mon, 12 Jan 2015 23:23:22 GMT
# Serv­er: Apache/2.4.10 (Debian)
# X-Pow­ered-By: PHP/5.6.4–1
# X-Weave-Time­stamp: 1421105002.7521
# X-Last-Mod­i­fied: 1421105002.73
# Con­tent-Length: 2
# Con­tent-Type: application/json; charset=utf8
# []