BUbiNG BUbiNG

蜘蛛/爬虫名称
BUbiNG
开发商
Universita degli studi di Milano.
官网
www.unimi.it
当前状态
活动
  • BUbiNG
  • BUbiNG
  • BUbiNG
  • BUbiNG
用户代理字符串
BUbiNG (+http://law.di.unimi.it/BUbiNG.html)
类型
首次出现
2013-10-04 13:20:30
最后出现
2019-11-22 21:51:48
遵循robots.txt
IP地址
570
来源
IP地址 服务器名称 所属国家
52.87.252.80 ec2-52-87-252-80.compute-1.amazonaws.com US
93.103.137.65 93-103-137-65.dynamic.t-2.net SI
89.212.20.214 89-212-20-214.dynamic.t-2.net SI
93.103.154.48 93-103-154-48.dynamic.t-2.net SI
195.78.123.5 ip005-123.netbone.cz CZ
64.62.252.163 ? US
64.62.252.164 ? US
64.62.252.162 ? US
147.229.8.39 athena2.fit.vutbr.cz CZ
64.62.252.169 ? US
用户代理字符串
BUbiNG (+http://law.di.unimi.it/BUbiNG.html#dc)
类型
首次出现
2018-03-03 03:13:56
最后出现
2018-03-25 22:51:46
遵循robots.txt
IP地址
8
来源
IP地址 服务器名称 所属国家
52.201.172.119 ec2-52-201-172-119.compute-1.amazonaws.com US
34.202.159.35 ec2-34-202-159-35.compute-1.amazonaws.com US
52.23.213.46 ec2-52-23-213-46.compute-1.amazonaws.com US
54.174.39.248 ec2-54-174-39-248.compute-1.amazonaws.com US
35.153.193.122 ec2-35-153-193-122.compute-1.amazonaws.com US
54.208.249.249 ec2-54-208-249-249.compute-1.amazonaws.com US
34.207.57.181 ec2-34-207-57-181.compute-1.amazonaws.com US
34.232.76.186 ec2-34-232-76-186.compute-1.amazonaws.com US
用户代理字符串
BUbiNG (+http://law.di.unimi.it/BUbiNG.html#wc)
类型
首次出现
2018-03-03 03:39:04
最后出现
2018-03-25 22:09:40
遵循robots.txt
IP地址
680
来源
IP地址 服务器名称 所属国家
147.229.8.59 knot38.fit.vutbr.cz CZ
147.229.8.170 knot36.fit.vutbr.cz CZ
147.229.8.167 knot37.fit.vutbr.cz CZ
147.229.8.39 athena2.fit.vutbr.cz CZ
147.229.8.130 knot30.fit.vutbr.cz CZ
147.229.8.140 knot35.fit.vutbr.cz CZ
147.229.8.136 knot33.fit.vutbr.cz CZ
147.229.8.224 knot04.fit.vutbr.cz CZ
147.229.8.105 athena5.fit.vutbr.cz CZ
147.229.8.135 athena9.fit.vutbr.cz CZ
52.87.252.80 ec2-52-87-252-80.compute-1.amazonaws.com US
93.103.137.65 93-103-137-65.dynamic.t-2.net SI
89.212.20.214 89-212-20-214.dynamic.t-2.net SI
93.103.154.48 93-103-154-48.dynamic.t-2.net SI
195.78.123.5 ip005-123.netbone.cz CZ
64.62.252.163 ? US
64.62.252.164 ? US
64.62.252.162 ? US
64.62.252.169 ? US
52.201.172.119 ec2-52-201-172-119.compute-1.amazonaws.com US
34.202.159.35 ec2-34-202-159-35.compute-1.amazonaws.com US
52.23.213.46 ec2-52-23-213-46.compute-1.amazonaws.com US
54.174.39.248 ec2-54-174-39-248.compute-1.amazonaws.com US
35.153.193.122 ec2-35-153-193-122.compute-1.amazonaws.com US
54.208.249.249 ec2-54-208-249-249.compute-1.amazonaws.com US
34.207.57.181 ec2-34-207-57-181.compute-1.amazonaws.com US
34.232.76.186 ec2-34-232-76-186.compute-1.amazonaws.com US
34.201.161.68 ec2-34-201-161-68.compute-1.amazonaws.com US
174.129.54.205 ec2-174-129-54-205.compute-1.amazonaws.com US
35.168.8.213 ec2-35-168-8-213.compute-1.amazonaws.com US
52.91.148.207 ec2-52-91-148-207.compute-1.amazonaws.com US
107.20.26.147 ec2-107-20-26-147.compute-1.amazonaws.com US
52.91.154.143 ec2-52-91-154-143.compute-1.amazonaws.com US
52.72.12.225 ec2-52-72-12-225.compute-1.amazonaws.com US
54.86.94.45 ec2-54-86-94-45.compute-1.amazonaws.com US
54.166.133.220 ec2-54-166-133-220.compute-1.amazonaws.com US
54.208.43.88 ec2-54-208-43-88.compute-1.amazonaws.com US
用户代理字符串
BUbiNG (+http://law.di.unimi.it/BUbiNG.html#wc)
类型
首次出现
2018-03-03 03:39:04
最后出现
2018-03-25 22:09:40
遵循robots.txt
IP地址
680
来源
IP地址 服务器名称 所属国家
34.201.161.68 ec2-34-201-161-68.compute-1.amazonaws.com US
174.129.54.205 ec2-174-129-54-205.compute-1.amazonaws.com US
35.168.8.213 ec2-35-168-8-213.compute-1.amazonaws.com US
52.91.148.207 ec2-52-91-148-207.compute-1.amazonaws.com US
107.20.26.147 ec2-107-20-26-147.compute-1.amazonaws.com US
52.91.154.143 ec2-52-91-154-143.compute-1.amazonaws.com US
52.72.12.225 ec2-52-72-12-225.compute-1.amazonaws.com US
54.86.94.45 ec2-54-86-94-45.compute-1.amazonaws.com US
54.166.133.220 ec2-54-166-133-220.compute-1.amazonaws.com US
54.208.43.88 ec2-54-208-43-88.compute-1.amazonaws.com US

BUbiNG是一个可扩展的、完全分布式的爬虫,目前正在开发中,它取代了UbiCrawler

BUbiNG支持机器人排除标准。如果你想排除你的网站被BUbiNG抓取,请参阅《网络机器人页面》。

简而言之,你可以在你想排除抓取的网络服务器根目录的robots.txt文件中放入以下内容。

User-agent: BUbiNG
Disallow: /

目前,BUbiNG尊重robots.txt文件的变化(通常每小时一次),但不支持META标签的机器人排除。

如需任何信息,请随时发送电子邮件至 law@di.unimi.it。

常见问题

FAQs
查看更多 >