如何做搜索引擎蜘蛛日志分析
搜索引擎蜘蛛日志文件是一种非常强大但未被站长充分利用的文件，分析它可以获取有关每个搜索引擎如何爬取网站内容的相关信息点，及查看搜索引擎蜘蛛在一段时间内的行为。
| IP地址(336) | 服务器名称 | 所属国家 |
|---|---|---|
| 23.178.112.209 | outbound2j.letsencrypt.org | US |
| 54.93.233.4 | ec2-54-93-233-4.eu-central-1.compute.amazonaws.com | DE |
| 35.91.3.63 | ec2-35-91-3-63.us-west-2.compute.amazonaws.com | US |
| 3.143.143.25 | ec2-3-143-143-25.us-east-2.compute.amazonaws.com | US |
| 54.185.13.181 | ec2-54-185-13-181.us-west-2.compute.amazonaws.com | US |
| 23.178.112.208 | outbound2i.letsencrypt.org | US |
| 23.178.112.107 | outbound1h.letsencrypt.org | US |
| 23.178.112.106 | outbound1g.letsencrypt.org | US |
| 54.218.35.76 | ec2-54-218-35-76.us-west-2.compute.amazonaws.com | US |
| 18.117.159.99 | ec2-18-117-159-99.us-east-2.compute.amazonaws.com | US |
| 54.184.182.85 | ec2-54-184-182-85.us-west-2.compute.amazonaws.com | US |
| 52.15.74.146 | ec2-52-15-74-146.us-east-2.compute.amazonaws.com | US |
| 23.178.112.202 | outbound2c.letsencrypt.org | US |
| 34.210.192.69 | ec2-34-210-192-69.us-west-2.compute.amazonaws.com | US |
| 3.145.60.188 | ec2-3-145-60-188.us-east-2.compute.amazonaws.com | US |
| 35.164.115.136 | ec2-35-164-115-136.us-west-2.compute.amazonaws.com | US |
| 3.128.201.87 | ec2-3-128-201-87.us-east-2.compute.amazonaws.com | US |
| 23.178.112.102 | outbound1c.letsencrypt.org | US |
| 35.92.118.9 | ec2-35-92-118-9.us-west-2.compute.amazonaws.com | US |
| 3.145.19.128 | ec2-3-145-19-128.us-east-2.compute.amazonaws.com | US |
| 35.93.12.38 | ec2-35-93-12-38.us-west-2.compute.amazonaws.com | ? |
| 3.142.70.114 | ec2-3-142-70-114.us-east-2.compute.amazonaws.com | ? |
| 3.135.199.109 | ec2-3-135-199-109.us-east-2.compute.amazonaws.com | US |
| 34.221.11.35 | ec2-34-221-11-35.us-west-2.compute.amazonaws.com | US |
| 35.92.59.45 | ec2-35-92-59-45.us-west-2.compute.amazonaws.com | US |
| 18.223.211.26 | ec2-18-223-211-26.us-east-2.compute.amazonaws.com | US |
| 34.222.163.213 | ec2-34-222-163-213.us-west-2.compute.amazonaws.com | US |
| 3.145.5.186 | ec2-3-145-5-186.us-east-2.compute.amazonaws.com | US |
| 54.187.64.166 | ec2-54-187-64-166.us-west-2.compute.amazonaws.com | US |
| 3.16.13.226 | ec2-3-16-13-226.us-east-2.compute.amazonaws.com | US |
| 35.87.122.78 | ec2-35-87-122-78.us-west-2.compute.amazonaws.com | ? |
| 54.186.131.13 | ec2-54-186-131-13.us-west-2.compute.amazonaws.com | US |
| 23.178.112.203 | outbound2d.letsencrypt.org | US |
| 18.221.236.99 | ec2-18-221-236-99.us-east-2.compute.amazonaws.com | US |
| 35.87.14.198 | ec2-35-87-14-198.us-west-2.compute.amazonaws.com | ? |
| 3.19.72.131 | ec2-3-19-72-131.us-east-2.compute.amazonaws.com | US |
| 18.236.170.127 | ec2-18-236-170-127.us-west-2.compute.amazonaws.com | US |
| 3.145.213.3 | ec2-3-145-213-3.us-east-2.compute.amazonaws.com | US |
| 2600:3000:1511:200::30 | 2600:3000:1511:200::30 | US |
| 18.222.123.46 | ec2-18-222-123-46.us-east-2.compute.amazonaws.com | ? |
| 52.41.99.246 | ec2-52-41-99-246.us-west-2.compute.amazonaws.com | US |
| 54.71.56.64 | ec2-54-71-56-64.us-west-2.compute.amazonaws.com | US |
| 54.190.10.33 | ec2-54-190-10-33.us-west-2.compute.amazonaws.com | US |
| 18.224.94.158 | ec2-18-224-94-158.us-east-2.compute.amazonaws.com | US |
| 3.141.165.28 | ec2-3-141-165-28.us-east-2.compute.amazonaws.com | US |
| 35.91.30.56 | ec2-35-91-30-56.us-west-2.compute.amazonaws.com | US |
| 34.215.137.243 | ec2-34-215-137-243.us-west-2.compute.amazonaws.com | US |
| 18.221.7.100 | ec2-18-221-7-100.us-east-2.compute.amazonaws.com | US |
| 35.89.121.147 | ec2-35-89-121-147.us-west-2.compute.amazonaws.com | US |
| 18.217.45.29 | ec2-18-217-45-29.us-east-2.compute.amazonaws.com | US |
| 35.91.136.165 | ec2-35-91-136-165.us-west-2.compute.amazonaws.com | US |
| 3.144.245.163 | ec2-3-144-245-163.us-east-2.compute.amazonaws.com | US |
| 13.59.62.81 | ec2-13-59-62-81.us-east-2.compute.amazonaws.com | US |
| 13.59.29.52 | ec2-13-59-29-52.us-east-2.compute.amazonaws.com | US |
| 34.218.50.164 | ec2-34-218-50-164.us-west-2.compute.amazonaws.com | US |
| 34.220.158.94 | ec2-34-220-158-94.us-west-2.compute.amazonaws.com | US |
| 3.15.191.190 | ec2-3-15-191-190.us-east-2.compute.amazonaws.com | US |
| 2600:3000:1511:200::1d | 2600:3000:1511:200::1d | US |
| 52.14.102.240 | ec2-52-14-102-240.us-east-2.compute.amazonaws.com | US |
| 34.212.49.199 | ec2-34-212-49-199.us-west-2.compute.amazonaws.com | US |
| 52.14.63.163 | ec2-52-14-63-163.us-east-2.compute.amazonaws.com | US |
| 2600:3000:1511:200::1e | 2600:3000:1511:200::1e | US |
| 54.213.139.231 | ec2-54-213-139-231.us-west-2.compute.amazonaws.com | US |
| 23.178.112.105 | outbound1f.letsencrypt.org | US |
| 18.191.141.144 | ec2-18-191-141-144.us-east-2.compute.amazonaws.com | US |
| 3.140.190.204 | ec2-3-140-190-204.us-east-2.compute.amazonaws.com | US |
| 35.167.67.31 | ec2-35-167-67-31.us-west-2.compute.amazonaws.com | US |
| 2600:3000:2710:200::83 | 2600:3000:2710:200::83 | US |
| 23.178.112.103 | outbound1d.letsencrypt.org | US |
| 3.141.28.134 | ec2-3-141-28-134.us-east-2.compute.amazonaws.com | US |
| 35.88.64.53 | ec2-35-88-64-53.us-west-2.compute.amazonaws.com | US |
| 3.12.103.105 | ec2-3-12-103-105.us-east-2.compute.amazonaws.com | US |
| 18.216.222.125 | ec2-18-216-222-125.us-east-2.compute.amazonaws.com | US |
| 54.202.43.248 | ec2-54-202-43-248.us-west-2.compute.amazonaws.com | US |
| 54.189.161.65 | ec2-54-189-161-65.us-west-2.compute.amazonaws.com | US |
| 23.178.112.108 | outbound1i.letsencrypt.org | US |
| 35.88.21.138 | ec2-35-88-21-138.us-west-2.compute.amazonaws.com | US |
| 18.217.140.116 | ec2-18-217-140-116.us-east-2.compute.amazonaws.com | US |
| 18.219.163.48 | ec2-18-219-163-48.us-east-2.compute.amazonaws.com | US |
| 34.211.243.12 | ec2-34-211-243-12.us-west-2.compute.amazonaws.com | US |
| 23.178.112.207 | outbound2h.letsencrypt.org | US |
| 3.142.90.193 | ec2-3-142-90-193.us-east-2.compute.amazonaws.com | US |
| 54.201.84.252 | ec2-54-201-84-252.us-west-2.compute.amazonaws.com | US |
| 18.190.176.117 | ec2-18-190-176-117.us-east-2.compute.amazonaws.com | US |
| 35.160.210.230 | ec2-35-160-210-230.us-west-2.compute.amazonaws.com | US |
| 13.250.98.175 | ec2-13-250-98-175.ap-southeast-1.compute.amazonaws.com | SG |
| 35.161.100.73 | ec2-35-161-100-73.us-west-2.compute.amazonaws.com | US |
| 23.178.112.104 | outbound1e.letsencrypt.org | US |
| 3.139.102.245 | ec2-3-139-102-245.us-east-2.compute.amazonaws.com | US |
| 16.171.69.211 | ec2-16-171-69-211.eu-north-1.compute.amazonaws.com | SE |
| 18.143.108.123 | ec2-18-143-108-123.ap-southeast-1.compute.amazonaws.com | SG |
| 34.217.212.110 | ec2-34-217-212-110.us-west-2.compute.amazonaws.com | US |
| 13.53.135.16 | ec2-13-53-135-16.eu-north-1.compute.amazonaws.com | SE |
| 23.178.112.109 | outbound1j.letsencrypt.org | US |
| 3.144.208.194 | ec2-3-144-208-194.us-east-2.compute.amazonaws.com | US |
| 13.49.80.215 | ec2-13-49-80-215.eu-north-1.compute.amazonaws.com | SE |
| 35.160.153.81 | ec2-35-160-153-81.us-west-2.compute.amazonaws.com | US |
| 3.145.32.18 | ec2-3-145-32-18.us-east-2.compute.amazonaws.com | US |
| 2600:3000:1511:200::89 | 2600:3000:1511:200::89 | US |
| 13.229.203.134 | ec2-13-229-203-134.ap-southeast-1.compute.amazonaws.com | SG |
| 54.186.37.221 | ec2-54-186-37-221.us-west-2.compute.amazonaws.com | US |
| 3.129.249.249 | ec2-3-129-249-249.us-east-2.compute.amazonaws.com | ? |
| 13.50.246.227 | ec2-13-50-246-227.eu-north-1.compute.amazonaws.com | ? |
| 13.214.36.220 | ec2-13-214-36-220.ap-southeast-1.compute.amazonaws.com | SG |
| 16.171.137.159 | ec2-16-171-137-159.eu-north-1.compute.amazonaws.com | SE |
| 13.212.185.12 | ec2-13-212-185-12.ap-southeast-1.compute.amazonaws.com | SG |
| 52.32.228.59 | ec2-52-32-228-59.us-west-2.compute.amazonaws.com | US |
| 3.144.235.68 | ec2-3-144-235-68.us-east-2.compute.amazonaws.com | US |
| 16.170.155.60 | ec2-16-170-155-60.eu-north-1.compute.amazonaws.com | SE |
| 23.178.112.211 | outbound2l.letsencrypt.org | US |
| 23.178.112.214 | outbound2o.letsencrypt.org | US |
| 51.20.104.171 | ec2-51-20-104-171.eu-north-1.compute.amazonaws.com | SE |
| 52.77.222.132 | ec2-52-77-222-132.ap-southeast-1.compute.amazonaws.com | SG |
| 18.237.134.91 | ec2-18-237-134-91.us-west-2.compute.amazonaws.com | US |
| 18.191.26.11 | ec2-18-191-26-11.us-east-2.compute.amazonaws.com | US |
| 16.16.126.218 | ec2-16-16-126-218.eu-north-1.compute.amazonaws.com | SE |
| 23.178.112.217 | outbound2p.letsencrypt.org | US |
| 23.178.112.212 | outbound2m.letsencrypt.org | US |
| 13.212.29.150 | ec2-13-212-29-150.ap-southeast-1.compute.amazonaws.com | SG |
| 2600:3000:1511:200::81 | 2600:3000:1511:200::81 | US |
| 54.189.166.5 | ec2-54-189-166-5.us-west-2.compute.amazonaws.com | US |
| 13.251.124.44 | ec2-13-251-124-44.ap-southeast-1.compute.amazonaws.com | SG |
| 3.21.207.218 | ec2-3-21-207-218.us-east-2.compute.amazonaws.com | US |
| 16.171.112.168 | ec2-16-171-112-168.eu-north-1.compute.amazonaws.com | SE |
| 3.0.93.116 | ec2-3-0-93-116.ap-southeast-1.compute.amazonaws.com | SG |
| 23.178.112.100 | outbound1a.letsencrypt.org | US |
| 13.60.189.223 | ec2-13-60-189-223.eu-north-1.compute.amazonaws.com | SE |
| 52.221.203.109 | ec2-52-221-203-109.ap-southeast-1.compute.amazonaws.com | SG |
| 16.171.61.198 | ec2-16-171-61-198.eu-north-1.compute.amazonaws.com | SE |
| 23.178.112.210 | outbound2k.letsencrypt.org | US |
| 54.254.225.158 | ec2-54-254-225-158.ap-southeast-1.compute.amazonaws.com | SG |
| 13.212.119.125 | ec2-13-212-119-125.ap-southeast-1.compute.amazonaws.com | SG |
| 3.137.175.210 | ec2-3-137-175-210.us-east-2.compute.amazonaws.com | US |
| 16.171.42.224 | ec2-16-171-42-224.eu-north-1.compute.amazonaws.com | SE |
| 34.217.113.97 | ec2-34-217-113-97.us-west-2.compute.amazonaws.com | US |
| 13.213.4.255 | ec2-13-213-4-255.ap-southeast-1.compute.amazonaws.com | SG |
| 18.236.77.81 | ec2-18-236-77-81.us-west-2.compute.amazonaws.com | US |
| 3.144.94.205 | ec2-3-144-94-205.us-east-2.compute.amazonaws.com | US |
| 13.48.44.150 | ec2-13-48-44-150.eu-north-1.compute.amazonaws.com | SE |
| 3.135.230.112 | ec2-3-135-230-112.us-east-2.compute.amazonaws.com | US |
| 54.244.211.15 | ec2-54-244-211-15.us-west-2.compute.amazonaws.com | US |
| 13.229.156.227 | ec2-13-229-156-227.ap-southeast-1.compute.amazonaws.com | SG |
| 16.170.35.184 | ec2-16-170-35-184.eu-north-1.compute.amazonaws.com | SE |
| 3.1.72.144 | ec2-3-1-72-144.ap-southeast-1.compute.amazonaws.com | SG |
| 13.61.22.140 | ec2-13-61-22-140.eu-north-1.compute.amazonaws.com | SE |
| 13.214.170.145 | ec2-13-214-170-145.ap-southeast-1.compute.amazonaws.com | SG |
| 54.200.69.86 | ec2-54-200-69-86.us-west-2.compute.amazonaws.com | US |
| 13.229.223.4 | ec2-13-229-223-4.ap-southeast-1.compute.amazonaws.com | SG |
| 3.142.247.43 | ec2-3-142-247-43.us-east-2.compute.amazonaws.com | US |
| 13.53.102.122 | ec2-13-53-102-122.eu-north-1.compute.amazonaws.com | SE |
| 3.1.23.222 | ec2-3-1-23-222.ap-southeast-1.compute.amazonaws.com | SG |
| 13.49.46.32 | ec2-13-49-46-32.eu-north-1.compute.amazonaws.com | SE |
| 44.244.72.161 | ec2-44-244-72-161.us-west-2.compute.amazonaws.com | US |
| 18.141.234.139 | ec2-18-141-234-139.ap-southeast-1.compute.amazonaws.com | SG |
| 35.91.18.144 | ec2-35-91-18-144.us-west-2.compute.amazonaws.com | US |
| 13.58.185.19 | ec2-13-58-185-19.us-east-2.compute.amazonaws.com | US |
| 13.49.159.147 | ec2-13-49-159-147.eu-north-1.compute.amazonaws.com | SE |
| 18.143.92.162 | ec2-18-143-92-162.ap-southeast-1.compute.amazonaws.com | SG |
| 3.136.37.150 | ec2-3-136-37-150.us-east-2.compute.amazonaws.com | US |
| 35.90.77.13 | ec2-35-90-77-13.us-west-2.compute.amazonaws.com | US |
| 16.16.76.248 | ec2-16-16-76-248.eu-north-1.compute.amazonaws.com | SE |
| 54.151.166.93 | ec2-54-151-166-93.ap-southeast-1.compute.amazonaws.com | SG |
| 3.138.125.8 | ec2-3-138-125-8.us-east-2.compute.amazonaws.com | US |
可能不需要。工具类型爬虫通常为网站所有者使用此类工具对网站进行相关服务请求才会出现。当然，实际情况需站长判断后再作决定。
您可以通过在网站的 robots.txt 中设置用户代理访问规则来屏蔽 letsencrypt bot 或限制其访问权限。我们建议安装 Spider Analyser 插件，以检查它是否真正遵循这些规则。
# robots.txt # 下列代码一般情况可以拦截该代理 User-agent: letsencrypt bot Disallow: /
您无需手动执行此操作，可通过我们的 Wordpress 插件 Spider Analyser 来拦截不必要的蜘蛛或者爬虫。
