nfsでファイルを読み書きしているときに、たまにものすごくレスポンスが遅くなる場合がある。原因はntpサーバマシンとntpクライアントマシンの間で時刻が大きく食い違っていたことが原因と考えられる。時刻同期に失敗していたnfsサーバマシンの時刻あわせを行い、様子見運用。ntpレスポンスが遅くなったときにそのときに/var/log/messageに流れたエラーらしきものが下。。
Apr 5 07:58:34 debian kernel: [2368986.028400] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:35 debian kernel: [2368987.028164] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:36 debian kernel: [2368988.028188] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:37 debian kernel: [2368989.028175] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:38 debian kernel: [2368990.028150] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:39 debian kernel: [2368991.028182] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:40 debian kernel: [2368992.028190] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:41 debian kernel: [2368993.028162] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:42 debian kernel: [2368994.028160] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:43 debian kernel: [2368995.028301] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:44 debian kernel: [2368996.028338] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:45 debian kernel: [2368997.028446] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:46 debian kernel: [2368998.028164] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:47 debian kernel: [2368999.028171] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:48 debian kernel: [2369000.028168] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:49 debian kernel: [2369001.028088] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:50 debian kernel: [2369002.028161] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:51 debian kernel: [2369003.028177] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:52 debian kernel: [2369004.028209] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:53 debian kernel: [2369005.028168] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:54 debian kernel: [2369006.028157] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:55 debian kernel: [2369007.028242] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:56 debian kernel: [2369008.028155] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:57 debian kernel: [2369009.028132] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:58 debian kernel: [2369010.028145] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:58:59 debian kernel: [2369011.028145] set_rtc_mmss: can't update from 0 to 58 Apr 5 07:59:00 debian kernel: [2369012.028069] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:01 debian kernel: [2369013.028138] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:02 debian kernel: [2369014.028132] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:03 debian kernel: [2369015.028123] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:04 debian kernel: [2369016.028063] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:05 debian kernel: [2369017.028067] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:06 debian kernel: [2369018.028059] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:07 debian kernel: [2369019.028068] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:08 debian kernel: [2369020.028058] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:09 debian kernel: [2369021.028057] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:10 debian kernel: [2369022.028059] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:11 debian kernel: [2369023.028067] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:12 debian kernel: [2369024.028063] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:13 debian kernel: [2369025.028067] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:14 debian kernel: [2369026.028066] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:15 debian kernel: [2369027.028073] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:16 debian kernel: [2369028.028057] set_rtc_mmss: can't update from 0 to 59 Apr 5 07:59:17 debian kernel: [2369029.028161] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:18 debian kernel: [2369030.028145] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:19 debian kernel: [2369031.028108] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:20 debian kernel: [2369032.028195] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:21 debian kernel: [2369033.028153] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:22 debian kernel: [2369034.028113] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:23 debian kernel: [2369035.028074] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:24 debian kernel: [2369036.028064] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:25 debian kernel: [2369037.028061] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:26 debian kernel: [2369038.028069] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:27 debian kernel: [2369039.028066] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:28 debian kernel: [2369040.028067] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:29 debian kernel: [2369041.028072] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:30 debian kernel: [2369042.028064] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:31 debian kernel: [2369043.028072] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:32 debian kernel: [2369044.028069] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:33 debian kernel: [2369045.028247] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:34 debian kernel: [2369046.028074] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:35 debian kernel: [2369047.028064] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:36 debian kernel: [2369048.028060] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:37 debian kernel: [2369049.028120] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:38 debian kernel: [2369050.028165] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:39 debian kernel: [2369051.028195] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:40 debian kernel: [2369052.028182] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:41 debian kernel: [2369053.028749] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:42 debian kernel: [2369054.028152] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:43 debian kernel: [2369055.028183] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:44 debian kernel: [2369056.028170] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:45 debian kernel: [2369057.028115] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:46 debian kernel: [2369058.028157] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:47 debian kernel: [2369059.028188] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:48 debian kernel: [2369060.028157] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:49 debian kernel: [2369061.028156] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:50 debian kernel: [2369062.028171] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:51 debian kernel: [2369063.028184] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:52 debian kernel: [2369064.028177] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:53 debian kernel: [2369065.028163] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:54 debian kernel: [2369066.028162] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:55 debian kernel: [2369067.028169] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:56 debian kernel: [2369068.028201] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:57 debian kernel: [2369069.028207] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:58 debian kernel: [2369070.028169] set_rtc_mmss: can't update from 1 to 59 Apr 5 07:59:59 debian kernel: [2369071.028152] set_rtc_mmss: can't update from 1 to 59
まずはnfsサーバマシンとクライアントマシンの時刻を確認すると大きく(3分くらい)違っている。
nfsserver:~$ date Sun Apr 5 08:46:19 JST 2009 nfsclient:~$ date Sun Apr 5 08:39:16 JST 2009
両方のマシンでのntpqの結果からnfsserverでは、時刻がずれ過ぎて時刻同期に失敗していることがわかる。offsetが168700[msec]程度なので、ほぼ2分48秒のずれがある。30分以上ずれると同期に失敗すると言う話を聞いたことがあるが、そうじゃないのかな。
nfsserver:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
doga.jp 133.243.238.243 2 u 61 64 17 39.279 168739. 26565.7
ns.doga.co.jp 133.243.238.243 2 u 23 64 37 48.324 168745. 128116.
suisho.attritio 133.100.9.2 2 u 31 64 37 9.928 168748. 128445.
www.4koma.com 133.243.238.243 2 u 25 64 37 23.192 168751. 128115.
nfsclient:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
+211.19.56.138 192.168.0.5 2 u 177 1024 377 12.837 2.072 0.340
*doga.jp 133.243.238.243 2 u 678 1024 377 17.071 1.699 6.645
+ns2.ktroad.net 210.173.160.57 3 u 204 1024 377 17.039 1.324 0.061
-7c29477b.i-revo 61.114.187.55 2 u 167 1024 377 16.798 0.937 1.085
時刻がずれすぎている場合は手作業で直す必要がある。そのためにntpdateコマンドを使う。コマンドが見つからないのでインストールする。
nfsserver:~# ntpdate doga.jp -su: ntpdate: command not found nfsserver:~# aptitude search ntp p chntpw - NT SAM password recovery utility p cyrus-nntpd-2.2 - Cyrus mail system (NNTP support) v cyrus21-nntpd - v cyrus22-nntpd - p libnet-ntp-perl - Perl module to query NTP servers p libnews-nntpclient-perl - News::NNTPClient, Perl support for accessing NNTP servers p mountpy - script for quick mounting of removable devices p msntp - A very simple and portable SNTP client/server i ntp - Network Time Protocol daemon and utility programs p ntp-doc - Network Time Protocol documentation p ntpdate - client for setting system time from NTP servers p openntpd - OpenBSD NTP daemon nfsserver:~# aptitude -R install ntpdate Reading package lists... Done Building dependency tree Reading state information... Done Reading extended state information Initializing package states... Done Reading task descriptions... Done The following NEW packages will be installed: ntpdate The following packages are RECOMMENDED but will NOT be installed: lockfile-progs 0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 59.8kB of archives. After unpacking 209kB will be used. Writing extended state information... Done Get:1 http://ftp.jp.debian.org lenny/main ntpdate 1:4.2.4p4+dfsg-8 [59.8kB] Fetched 59.8kB in 0s (80.4kB/s) Selecting previously deselected package ntpdate. (Reading database ... 93881 files and directories currently installed.) Unpacking ntpdate (from .../ntpdate_1%3a4.2.4p4+dfsg-8_i386.deb) ... Processing triggers for man-db ... Setting up ntpdate (1:4.2.4p4+dfsg-8) ... Reading package lists... Done Building dependency tree Reading state information... Done Reading extended state information Initializing package states... Done Writing extended state information... Done Reading task descriptions... Done
導入が終わったら時刻あわせ。ntpクライアントデーモンが動いているとntpdateが使うソケットが使用中となって同期できないので、ntpdateする前にntpクライアントデーモンをstopする。同期完了したら再度ntpクライアントデーモンをstartしておく。
nfsserver:~# date Sun Apr 5 08:46:19 JST 2009 nfsserver:~# ntpdate ntp1.jst.mfeed.ad.jp 5 Apr 08:47:04 ntpdate[3568]: the NTP socket is in use, exiting nfsserver:~# date Sun Apr 5 08:47:12 JST 2009 nfsserver:~# /etc/init.d/ntp stop Stopping NTP server: ntpd. nfsserver:~# ntpdate ntp1.jst.mfeed.ad.jp 5 Apr 08:50:49 ntpdate[3577]: step time server 210.173.160.27 offset 173.438520 sec nfsserver:~# date Sun Apr 5 08:50:53 JST 2009 nfsserver:~# /etc/init.d/ntp start Starting NTP server: ntpd.
同期確認のためにntp -qする。すぐには反映されないので30分くらい待って再確認。同期されていることがわかる。
nfsserver:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
ns.doga.co.jp 133.243.238.243 2 u - 64 1 62.996 -10.454 7.343
www.hypercore.c 210.173.160.27 3 u 1 64 1 16.511 1.519 0.507
kanda.likk.net 133.100.9.2 2 u - 64 1 10.775 -15.372 0.526
nfsserver:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
+ns.doga.co.jp 133.243.238.243 2 u 29 64 37 74.702 46.415 23.793
+www.hypercore.c 210.173.160.27 3 u 9 64 37 15.582 64.689 7.138
*kanda.likk.net 133.100.9.2 2 u 42 64 37 9.673 64.877 7.862
- set_rtc_mmss - Google 検索
- set_rtc_mmss: can't update from ** to ** - tomisima::tmp - cucumberグループ
- ntp.conf - Google 検索
- NTPDの設定
- ntpq - Google 検索
- ntpq出力内容の説明
解決したと思ったら同じようなエラーメッセージがまたでる。マザーボードの電池切れという可能性も否定できなくなってきた。と思ったら、同じ人の記事にntpdで更新していれば問題ないとの記述あり。うちはntpdで同期作業を行っているんだけどなぁ。
ntpdを一回終了して、ntpdateで時刻あわせ、hwclockでソフトクロックにハードクロックを同期。この状態でwatch ntpq -pすると、初めにoffset値-15msec程度だったのが数分後にoffsetが4000msecくらいに増加(同じ参照サーバで)。結局再起動してしまったよ。これで解決したとしてもなんだかなぁ。今のところ、再起動前に比べてwatch ntpq -pのoffset値が大きくならない。
ただし、同期が出来ているマシンのntpq -pとは若干異なっている。offsetが比較的大きい。
nfsserver:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
+ntp.kiba.net 197.54.93.103 2 u 21 64 377 19.300 35.570 2.853
*www.4koma.com 133.243.238.163 2 u 17 64 377 21.404 35.848 3.424
-211.19.56.141 211.19.56.138 3 u 27 64 377 12.831 34.355 2.384
+suisho.attritio 133.100.9.2 2 u 17 64 377 8.733 36.825 1.484
nfsclient:~# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*211.19.56.138 192.168.0.5 2 u 652 1024 377 16.197 -1.138 1.616
+doga.jp 133.243.238.243 2 u 136 1024 377 17.160 -1.188 2.200
+ns2.ktroad.net 210.173.160.57 3 u 693 1024 377 17.490 -0.976 1.004
+7c29477b.i-revo 61.114.187.55 2 u 645 1024 377 16.552 0.463 1.151