wget
wgetrc の読み込みを抑制するには--no-config 、netrc の読み込みを抑制するには --execute=netrc=off。
$ wget --quiet --debug --no-config --execute=netrc=off --output-document /dev/null 'http://www.example.com/' Setting --no-config (noconfig) to 1 Setting netrc (netrc) to off Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.16 on linux-gnu. URI encoding = ‘UTF-8’ Caching www.example.com => 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 Created socket 5. Releasing 0x0000000000838840 (new refcount 1). ---request begin--- GET / HTTP/1.1 User-Agent: Wget/1.16 (linux-gnu) Accept: */* Host: www.example.com Connection: Keep-Alive ---request end--- ---response begin--- HTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: max-age=604800 Content-Type: text/html Date: ***, ** *** **** **:**:** GMT Etag: "**************" Expires: ***, ** *** **** **:**:** GMT Last-Modified: ***, ** *** **** **:**:** GMT Server: ECS (********) Vary: Accept-Encoding X-Cache: HIT x-ec-custom-error: 1 Content-Length: 1270 ---response end--- Registered socket 5 for persistent reuse.
curl
curl の後の -q は curlrc の読み込みを抑制する。
$ curl -q --silent --verbose --output /dev/null 'http://www.example.com/'
* Hostname was NOT found in DNS cache
* Trying 93.184.216.34...
* Connected to www.example.com (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: www.example.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: max-age=604800
< Content-Type: text/html
< Date: ***, ** *** **** **:**:** GMT
< Etag: "*********"
< Expires: ***, ** *** **** **:**:** GMT
< Last-Modified: ***, ** *** **** **:**:** GMT
* Server ECS (********) is not blacklisted
< Server: ECS (********)
< Vary: Accept-Encoding
< X-Cache: HIT
< x-ec-custom-error: 1
< Content-Length: 1270
<
{ [data not shown]
* Connection #0 to host www.example.com left intact
Mozilla Firefox
$ export NSPR_LOG_MODULES=nsHttp:5 $ export NSPR_LOG_FILE=/tmp/log.txt $ firefox -private http://www.example.com/
(snip) 4628[819140]: http request [ 4628[819140]: GET / HTTP/1.1 4628[819140]: Host: www.example.com 4628[819140]: User-Agent: Mozilla/5.0 (******************************) Gecko/******** Firefox/**** 4628[819140]: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 4628[819140]: Accept-Language: ja,en-US;q=0.7,en;q=0.3 4628[819140]: Accept-Encoding: gzip, deflate 4628[819140]: Connection: keep-alive 4628[819140]: ] (snip) 5056[8197d0]: http response [ 5056[8197d0]: HTTP/1.1 200 OK 5056[8197d0]: Content-Encoding: gzip 5056[8197d0]: Cache-Control: max-age=604800 5056[8197d0]: Content-Type: text/html 5056[8197d0]: Date: ***, ** *** **** **:**:** GMT 5056[8197d0]: Etag: "**************" 5056[8197d0]: Expires: ***, ** *** **** **:**:** GMT 5056[8197d0]: Last-Modified: ***, ** *** **** **:**:** GMT 5056[8197d0]: Server: ECS (********) 5056[8197d0]: Vary: Accept-Encoding 5056[8197d0]: X-Cache: HIT 5056[8197d0]: x-ec-custom-error: 1 5056[8197d0]: Content-Length: 606 5056[8197d0]: ] (snip)
perl
HTTP::Request や HTTP::Response は HTTP::Message のメソッドを継承しているため、これらのオブジェクトに HTTP::Message の dump メソッドが使える。dump は適当にうまいことやってくれるので、ちょっとした確認には便利。
$ cat ./lwp-test.pl
#!/usr/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
use URI;
my $uri = URI->new('http://www.example.org');
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET' => $uri);
my $res = $ua->request($req);
print $res->request->dump;
print $res->dump;
exit;
$ perl ./lwp-test.pl
GET http://www.example.org
User-Agent: libwww-perl/****
(no content)
HTTP/1.1 200 OK
Cache-Control: max-age=604800
Connection: close
Date: ***, ** *** **** **:**:** GMT
Accept-Ranges: bytes
ETag: "*********"
Server: ECS (********)
Vary: Accept-Encoding
Content-Length: 1270
Content-Type: text/html
Expires: ***, ** *** **** **:**:** GMT
Last-Modified: ***, ** *** **** **:**:** GMT
Client-Date: ***, ** *** **** **:**:** GMT
Client-Peer: **.***.***.**:**
Client-Response-Num: 1
Title: Example Domain
X-Cache: HIT
X-Ec-Custom-Error: 1
X-Meta-Charset: utf-8
X-Meta-Viewport: width=device-width, initial-scale=1
<!doctype html>
<html>
<head>
<title>Example Domain</title>\n
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
\40\40\40\40\40\40\40\40
}
div {
width: 600px;
margin: 5em auto;
...
(+ 758 more bytes not shown)
本当にヘッダだけが欲しい場合は HTTP::Headers オブジェクトを使う。HTTP::Request や HTTP::Response オブジェクトから HTTP::Headers オブジェクトを得るには headers メソッド、HTTP::Headers からヘッダを表示するには as_string メソッドを使う。
$ cat ./lwp-test.pl
#!/usr/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
use URI;
my $uri = URI->new('http://www.example.org');
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET' => $uri);
my $res = $ua->request($req);
print $res->request->headers->as_string, "\n";
print $res->headers->as_string, "\n";
exit;
$ perl ./lwp-test.pl
User-Agent: libwww-perl/****
Cache-Control: max-age=604800
Connection: close
Date: ***, ** *** **** **:**:** GMT
Accept-Ranges: bytes
ETag: "*********"
Server: ECS (********)
Vary: Accept-Encoding
Content-Length: 1270
Content-Type: text/html
Expires: ***, ** *** **** **:**:** GMT
Last-Modified: ***, ** *** **** **:**:** GMT
Client-Date: ***, ** *** **** **:**:** GMT
Client-Peer: **.***.***.**:**
Client-Response-Num: 1
Title: Example Domain
X-Cache: HIT
X-Ec-Custom-Error: 1
X-Meta-Charset: utf-8
X-Meta-Viewport: width=device-width, initial-scale=1
他のクライアントと違って Client-Date などの Client-XXX が付いている点は注目すべき。これはlwp が自動的につけるものであって、実際に送られてきたものではない。この特殊ヘッダの処理はlwp-dumpのソースが参考になる。
$ lwp-dump --request http://www.example.org/
GET http://www.example.org/
User-Agent: lwp-dump/**** libwww-perl/****
(no content)
HTTP/1.1 200 OK
Cache-Control: max-age=604800
Date: ***, ** *** **** **:**:** GMT
Accept-Ranges: bytes
ETag: "*********"
Server: ECS (********)
Vary: Accept-Encoding
Content-Length: 1270
Content-Type: text/html
Expires: ***, ** *** **** **:**:** GMT
Last-Modified: ***, ** *** **** **:**:** GMT
X-Cache: HIT
X-Ec-Custom-Error: 1
<!doctype html>
<html>
<head>
<title>Example Domain</title>\n
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
\40\40\40\40\40\40\40\40
}
div {
width: 600px;
margin: 5em auto;
...
(+ 758 more bytes not shown)
また、ヘッダの順番とフィールド名も保存されない点にも注意が必要。
一般
$ LANG=C strace wget ...
自前のサーバの出力を確認する方針
netcat の出力からリクエストヘッダを確認する方針も取れる。
$ nc.openbsd -vv -k -l 127.0.0.1 8080; Listening on [127.0.0.1] (family 0, port 8080) Connection from [127.0.0.1] port 8080 [tcp/http-alt] accepted (family 2, sport 51210) GET / HTTP/1.1 User-Agent: Wget/**** (linux-gnu) Accept: */* Host: 127.0.0.1:8080 Connection: Keep-Alive Connection closed, listening again. Connection from [127.0.0.1] port 8080 [tcp/http-alt] accepted (family 2, sport 51211) GET / HTTP/1.1 User-Agent: Wget/**** (linux-gnu) Accept: */* Host: 127.0.0.1:8080 Connection: Keep-Alive Connection closed, listening again.