如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(34) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 3.252.212.4 | ec2-3-252-212-4.eu-west-1.compute.amazonaws.com | IE |
| 54.170.77.139 | ec2-54-170-77-139.eu-west-1.compute.amazonaws.com | ? |
| 3.252.139.237 | ec2-3-252-139-237.eu-west-1.compute.amazonaws.com | IE |
| 3.255.97.138 | ec2-3-255-97-138.eu-west-1.compute.amazonaws.com | IE |
| 54.74.202.187 | ec2-54-74-202-187.eu-west-1.compute.amazonaws.com | IE |
| 3.255.128.227 | ec2-3-255-128-227.eu-west-1.compute.amazonaws.com | IE |
| 54.78.194.111 | ec2-54-78-194-111.eu-west-1.compute.amazonaws.com | IE |
| 3.255.213.22 | ec2-3-255-213-22.eu-west-1.compute.amazonaws.com | IE |
| 3.249.194.12 | ec2-3-249-194-12.eu-west-1.compute.amazonaws.com | IE |
| 54.74.160.2 | ec2-54-74-160-2.eu-west-1.compute.amazonaws.com | IE |
| 52.208.61.166 | ec2-52-208-61-166.eu-west-1.compute.amazonaws.com | IE |
| 3.255.100.219 | ec2-3-255-100-219.eu-west-1.compute.amazonaws.com | IE |
| 52.51.169.199 | ec2-52-51-169-199.eu-west-1.compute.amazonaws.com | IE |
| 52.208.216.4 | ec2-52-208-216-4.eu-west-1.compute.amazonaws.com | ? |
| 3.255.207.199 | ec2-3-255-207-199.eu-west-1.compute.amazonaws.com | IE |
| 54.74.120.38 | ec2-54-74-120-38.eu-west-1.compute.amazonaws.com | IE |
| 34.246.183.249 | ec2-34-246-183-249.eu-west-1.compute.amazonaws.com | IE |
| 3.251.88.208 | ec2-3-251-88-208.eu-west-1.compute.amazonaws.com | IE |
| 52.17.94.132 | ec2-52-17-94-132.eu-west-1.compute.amazonaws.com | ? |
| 34.241.21.147 | ec2-34-241-21-147.eu-west-1.compute.amazonaws.com | IE |
| 34.253.1.153 | ec2-34-253-1-153.eu-west-1.compute.amazonaws.com | IE |
| 54.75.58.44 | ec2-54-75-58-44.eu-west-1.compute.amazonaws.com | IE |
| 34.242.77.198 | ec2-34-242-77-198.eu-west-1.compute.amazonaws.com | IE |
| 54.74.142.1 | ec2-54-74-142-1.eu-west-1.compute.amazonaws.com | IE |
| 54.171.134.58 | ec2-54-171-134-58.eu-west-1.compute.amazonaws.com | IE |
| 54.155.71.44 | ec2-54-155-71-44.eu-west-1.compute.amazonaws.com | IE |
| 34.250.219.68 | ec2-34-250-219-68.eu-west-1.compute.amazonaws.com | IE |
| 63.34.21.170 | ec2-63-34-21-170.eu-west-1.compute.amazonaws.com | IE |
| 52.30.221.134 | ec2-52-30-221-134.eu-west-1.compute.amazonaws.com | IE |
| IP地址(138) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 54.229.192.74 | ec2-54-229-192-74.eu-west-1.compute.amazonaws.com | IE |
| 54.75.4.77 | ec2-54-75-4-77.eu-west-1.compute.amazonaws.com | IE |
| 34.251.157.163 | ec2-34-251-157-163.eu-west-1.compute.amazonaws.com | IE |
| 63.33.190.132 | ec2-63-33-190-132.eu-west-1.compute.amazonaws.com | IE |
| 176.34.204.41 | ec2-176-34-204-41.eu-west-1.compute.amazonaws.com | IE |
| 3.250.66.54 | ec2-3-250-66-54.eu-west-1.compute.amazonaws.com | IE |
| 54.78.117.22 | ec2-54-78-117-22.eu-west-1.compute.amazonaws.com | IE |
| 34.253.40.179 | ec2-34-253-40-179.eu-west-1.compute.amazonaws.com | IE |
| 34.251.210.108 | ec2-34-251-210-108.eu-west-1.compute.amazonaws.com | IE |
| 34.254.202.30 | ec2-34-254-202-30.eu-west-1.compute.amazonaws.com | IE |
| 34.242.222.14 | ec2-34-242-222-14.eu-west-1.compute.amazonaws.com | IE |
| 34.244.92.181 | ec2-34-244-92-181.eu-west-1.compute.amazonaws.com | IE |
| 54.217.141.255 | ec2-54-217-141-255.eu-west-1.compute.amazonaws.com | IE |
| 34.255.181.176 | ec2-34-255-181-176.eu-west-1.compute.amazonaws.com | IE |
| 194.163.169.235 | vmi1131820.contaboserver.net | DE |
| 63.33.211.78 | ec2-63-33-211-78.eu-west-1.compute.amazonaws.com | IE |
| 3.249.253.129 | ec2-3-249-253-129.eu-west-1.compute.amazonaws.com | IE |
| 18.203.111.214 | ec2-18-203-111-214.eu-west-1.compute.amazonaws.com | IE |
| 54.78.8.161 | ec2-54-78-8-161.eu-west-1.compute.amazonaws.com | IE |
| 34.245.229.79 | ec2-34-245-229-79.eu-west-1.compute.amazonaws.com | IE |
| 34.255.4.48 | ec2-34-255-4-48.eu-west-1.compute.amazonaws.com | IE |
| 18.203.236.62 | ec2-18-203-236-62.eu-west-1.compute.amazonaws.com | IE |
| 54.78.201.228 | ec2-54-78-201-228.eu-west-1.compute.amazonaws.com | IE |
| 3.249.157.194 | ec2-3-249-157-194.eu-west-1.compute.amazonaws.com | IE |
| 3.250.114.166 | ec2-3-250-114-166.eu-west-1.compute.amazonaws.com | IE |
| 52.18.139.122 | ec2-52-18-139-122.eu-west-1.compute.amazonaws.com | IE |
| 34.244.51.40 | ec2-34-244-51-40.eu-west-1.compute.amazonaws.com | IE |
| 54.155.163.164 | ec2-54-155-163-164.eu-west-1.compute.amazonaws.com | IE |
| 34.243.86.119 | ec2-34-243-86-119.eu-west-1.compute.amazonaws.com | IE |
| 34.253.239.114 | ec2-34-253-239-114.eu-west-1.compute.amazonaws.com | IE |
| 176.34.198.89 | ec2-176-34-198-89.eu-west-1.compute.amazonaws.com | IE |
| 34.245.192.75 | ec2-34-245-192-75.eu-west-1.compute.amazonaws.com | IE |
| 52.210.35.132 | ec2-52-210-35-132.eu-west-1.compute.amazonaws.com | IE |
| 18.202.239.4 | ec2-18-202-239-4.eu-west-1.compute.amazonaws.com | IE |
| 18.203.235.198 | ec2-18-203-235-198.eu-west-1.compute.amazonaws.com | IE |
| 34.242.6.221 | ec2-34-242-6-221.eu-west-1.compute.amazonaws.com | IE |
| 34.243.115.4 | ec2-34-243-115-4.eu-west-1.compute.amazonaws.com | IE |
| 54.74.221.183 | ec2-54-74-221-183.eu-west-1.compute.amazonaws.com | IE |
| 3.252.139.226 | ec2-3-252-139-226.eu-west-1.compute.amazonaws.com | IE |
| 54.217.177.242 | ec2-54-217-177-242.eu-west-1.compute.amazonaws.com | IE |
| 3.253.54.60 | ec2-3-253-54-60.eu-west-1.compute.amazonaws.com | IE |
| 3.249.169.194 | ec2-3-249-169-194.eu-west-1.compute.amazonaws.com | IE |
| 3.252.97.79 | ec2-3-252-97-79.eu-west-1.compute.amazonaws.com | IE |
| 3.249.218.124 | ec2-3-249-218-124.eu-west-1.compute.amazonaws.com | IE |
| 34.246.160.103 | ec2-34-246-160-103.eu-west-1.compute.amazonaws.com | IE |
| 34.242.158.185 | ec2-34-242-158-185.eu-west-1.compute.amazonaws.com | IE |
| 34.251.19.245 | ec2-34-251-19-245.eu-west-1.compute.amazonaws.com | IE |
| 54.171.138.210 | ec2-54-171-138-210.eu-west-1.compute.amazonaws.com | ? |
| 18.203.233.54 | ec2-18-203-233-54.eu-west-1.compute.amazonaws.com | IE |
| 54.247.205.124 | ec2-54-247-205-124.eu-west-1.compute.amazonaws.com | IE |
| 63.32.95.78 | ec2-63-32-95-78.eu-west-1.compute.amazonaws.com | IE |
| 52.212.161.192 | ec2-52-212-161-192.eu-west-1.compute.amazonaws.com | IE |
| 3.254.86.96 | ec2-3-254-86-96.eu-west-1.compute.amazonaws.com | IE |
| 52.211.151.177 | ec2-52-211-151-177.eu-west-1.compute.amazonaws.com | IE |
| 34.253.220.10 | ec2-34-253-220-10.eu-west-1.compute.amazonaws.com | IE |
| 34.248.12.11 | ec2-34-248-12-11.eu-west-1.compute.amazonaws.com | IE |
| 34.254.113.28 | ec2-34-254-113-28.eu-west-1.compute.amazonaws.com | IE |
| 54.216.101.54 | ec2-54-216-101-54.eu-west-1.compute.amazonaws.com | IE |
| 3.250.43.9 | ec2-3-250-43-9.eu-west-1.compute.amazonaws.com | IE |
| 34.242.104.135 | ec2-34-242-104-135.eu-west-1.compute.amazonaws.com | IE |
| 63.35.182.237 | ec2-63-35-182-237.eu-west-1.compute.amazonaws.com | IE |
| 3.252.93.211 | ec2-3-252-93-211.eu-west-1.compute.amazonaws.com | IE |
| 18.202.248.160 | ec2-18-202-248-160.eu-west-1.compute.amazonaws.com | IE |
| 3.254.192.209 | ec2-3-254-192-209.eu-west-1.compute.amazonaws.com | IE |
| 63.34.12.133 | ec2-63-34-12-133.eu-west-1.compute.amazonaws.com | IE |
| IP地址(3) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 54.242.42.41 | ec2-54-242-42-41.compute-1.amazonaws.com | US |
| 54.81.178.12 | ec2-54-81-178-12.compute-1.amazonaws.com | US |
| 54.242.52.245 | ec2-54-242-52-245.compute-1.amazonaws.com | US |
對於未知蜘蛛或者爬蟲。它的用途對網站來說可能是好的，也可能是壞的，這取決於它是什麼。所以說，這需要站長進一步分析判斷這些尚不明確的爬蟲行為，再作最終決定。 但，根據以往的經驗，未宣告行為目的及未命名的蜘蛛爬蟲，通常都有不可告人的祕密，我們理應對其行為進行控制，比如攔截。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 domainsbot 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: domainsbot Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
