I have been play­ing around with duplic­at­ing another program’s web ser­vice API. I want to be able to act as a sub­sti­tute but need to rewrite it in a new lan­guage and change a bunch of its fea­tures. To do this I have been run­ning the ori­ginal program’s func­tional tests against my own code.

Unfor­tu­nately it is some­times dif­fi­cult to see exactly what is going on. The tests are writ­ten in Python, which I am not very famil­iar with, and are wrapped through vari­ous lay­ers adding dif­fer­ent headers.

Hap­pily Apache sup­ports log­ging with mod_dumpio so I can run the tests until they fail and then simply look at the last request. That was the plan at least, but a single call gen­er­ates sixty lines of log. Much of it look­ing like the sample below, obvi­ously the middle line is con­tent the other two are noise.

[Thu Jan 15 10:06:08.118940 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(63): [client ::1:53956] mod_dumpio: dumpio_out (data-HEAP): 1 bytes
[Thu Jan 15 10:06:08.118945 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(103): [client ::1:53956] mod_dumpio: dumpio_out (data-HEAP): 0
[Thu Jan 15 10:06:08.118952 2015] [dumpio:trace7] [pid 1840] mod_dumpio.c(63): [client ::1:53956] mod_dumpio: dumpio_out (metadata-EOS): 0 bytes

A quick search online threw up dumpio_parser.pl which extracts the requests from log files. How­ever it works for Apache 2.2, not the slightly mod­i­fied format in Apache 2.4.

So I rewrote the input por­tion of dumpio_parser.pl.

Then the out­put was in a mar­gin­ally use­ful format, it essen­tially provided the same con­tent lines in the log. I wanted a format I could throw at curl to retry the request and see if my bug fix had worked.

Fix­ing that I was still not sat­is­fied. I didn’t like the out­put struc­ture, it cre­ated a col­lec­tion of tiny files, one for each request. And it ignored the out­put from the server. AND the struc­ture was weird, first pro­gram I have seen using perl –n.

So I rewrote it.

Present­ing dumpio2curl.

> tail -n 100 apache.log | perl ../dumpio2curl/dumpio2curl.pl

# Tue Jan 13 10:23:22.729122 2015 — pid 15952 — cli­ent 50304
curl –v \
–header “User-agent:” –header “Accept:” \
–request “DELETE” \
–header “Accept-Encoding: iden­tity” \
–header “Content-Length: 0″ \
–header “X-Forwarded-For:″ \
–header “X-Forwarded-Script-Name: ” \
–header “X-Forwarded-Scheme: http” \
–header “X-Wsgiproxy-Version: 0.1″ \
–header “X-Forwarded-Server: localhost:88″ \
–header “Author­iz­a­tion: Hawk …” \


# Tue Jan 13 10:23:22.752411 2015 — pid 15952 — cli­ent 50304
# HTTP/1.1 200 OK
# Date: Mon, 12 Jan 2015 23:23:22 GMT
# Server: Apache/2.4.10 (Debian)
# X-Powered-By: PHP/5.6.4 – 1
# X-Weave-Timestamp: 1421105002.7521
# X-Last-Modified: 1421105002.73
# Content-Length: 2
# Content-Type: application/json; charset=utf8
# []