如何做搜尋引擎蜘蛛日誌分析
搜尋引擎蜘蛛日誌檔案是一種非常強大但未被站長充分利用的檔案，分析它可以獲取有關每個搜尋引擎如何爬取網站內容的相關資訊點，及檢視搜尋引擎蜘蛛在一段時間內的行為。
| IP地址(6) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 139.196.43.82 | 139.196.43.82 | CN |
| 139.224.43.99 | 139.224.43.99 | CN |
| 106.15.51.57 | 106.15.51.57 | CN |
| 106.14.92.200 | 106.14.92.200 | CN |
| 47.102.43.120 | 47.102.43.120 | CN |
| 139.196.41.228 | 139.196.41.228 | CN |
| IP地址(449) | 伺服器名稱 | 所屬國家 |
|---|---|---|
| 106.15.2.160 | 106.15.2.160 | CN |
| 101.132.223.151 | 101.132.223.151 | ? |
| 47.103.107.45 | 47.103.107.45 | CN |
| 106.15.136.110 | 106.15.136.110 | CN |
| 47.103.92.160 | 47.103.92.160 | CN |
| 47.103.95.126 | 47.103.95.126 | CN |
| 47.103.93.65 | 47.103.93.65 | CN |
| 106.15.33.153 | 106.15.33.153 | CN |
| 47.103.96.50 | 47.103.96.50 | CN |
| 47.101.64.201 | 47.101.64.201 | CN |
| 106.15.188.92 | 106.15.188.92 | CN |
| 47.103.107.246 | 47.103.107.246 | ? |
| 47.103.100.239 | 47.103.100.239 | CN |
| 47.103.141.72 | 47.103.141.72 | CN |
| 47.103.124.85 | 47.103.124.85 | CN |
| 47.102.121.27 | 47.102.121.27 | CN |
| 47.103.72.158 | 47.103.72.158 | CN |
| 47.100.195.225 | 47.100.195.225 | CN |
| 47.103.74.195 | 47.103.74.195 | CN |
| 47.103.98.166 | 47.103.98.166 | CN |
| 47.103.70.92 | 47.103.70.92 | CN |
| 139.224.162.137 | 139.224.162.137 | CN |
| 101.133.168.233 | 101.133.168.233 | CN |
| 47.100.14.61 | 47.100.14.61 | CN |
| 47.103.125.159 | 47.103.125.159 | CN |
| 47.100.13.123 | 47.100.13.123 | CN |
| 106.14.79.192 | 106.14.79.192 | CN |
| 47.102.101.92 | 47.102.101.92 | CN |
| 101.133.160.220 | 101.133.160.220 | CN |
| 47.103.222.46 | 47.103.222.46 | CN |
| 47.103.219.31 | 47.103.219.31 | CN |
| 106.15.7.132 | 106.15.7.132 | CN |
| 47.101.131.148 | 47.101.131.148 | CN |
| 106.14.141.72 | 106.14.141.72 | CN |
| 47.103.196.187 | 47.103.196.187 | CN |
| 47.103.215.230 | 47.103.215.230 | CN |
| 106.14.14.227 | 106.14.14.227 | CN |
| 106.15.57.73 | 106.15.57.73 | CN |
| 139.224.245.2 | 139.224.245.2 | CN |
| 47.103.122.65 | 47.103.122.65 | CN |
| 47.103.221.54 | 47.103.221.54 | CN |
| 47.102.123.116 | 47.102.123.116 | CN |
| 47.103.215.78 | 47.103.215.78 | CN |
| 101.133.169.211 | 101.133.169.211 | CN |
| 139.224.253.30 | 139.224.253.30 | CN |
| 47.103.193.243 | 47.103.193.243 | ? |
| 106.15.235.181 | 106.15.235.181 | CN |
| 47.102.195.83 | 47.102.195.83 | CN |
| 139.224.117.172 | 139.224.117.172 | CN |
| 47.103.220.33 | 47.103.220.33 | CN |
| 47.103.135.39 | 47.103.135.39 | CN |
| 47.102.185.111 | 47.102.185.111 | CN |
| 47.100.244.60 | 47.100.244.60 | CN |
| 47.116.76.173 | 47.116.76.173 | CN |
| 47.116.78.25 | 47.116.78.25 | CN |
| 106.14.149.237 | 106.14.149.237 | CN |
| 47.100.166.201 | 47.100.166.201 | CN |
| 47.116.11.58 | 47.116.11.58 | CN |
| 47.116.79.211 | 47.116.79.211 | CN |
可以考慮攔截。。爬蟲通常會下載公開的網際網路內容，這些內容預設情況下可以免費訪問。不過，如果你不希望你的內容被用於未經授權的目的，你應該攔截它們。
您可以通過在網站的 robots.txt 中設定使用者代理訪問規則來遮蔽 SkyworkSpider 或限制其訪問許可權。我們建議安裝 Spider Analyser 外掛，以檢查它是否真正遵循這些規則。
# robots.txt # 下列程式碼一般情況可以攔截該代理 User-agent: SkyworkSpider Disallow: /
您無需手動執行此操作，可通過我們的 Wordpress 外掛 Spider Analyser 來攔截不必要的蜘蛛或者爬蟲。
