robots.txt on wiki and forum
#1
The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:


Code:
User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
  Reply
#2
(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:


Code:
User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.
  Reply
#3
(06-10-2016, 01:38 AM)tllim Wrote:
(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:


Code:
User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).
  Reply
#4
(06-10-2016, 11:23 PM)lvella Wrote:
(06-10-2016, 01:38 AM)tllim Wrote:
(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:


Code:
User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).
I will explore with IT manager again, he against the idea on opening.
  Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  A dedicated forum for guides and tutorials? shockr 5 2,427 01-16-2017, 04:39 PM
Last Post: tllim
  deutschsprachiges Forum g0su 15 7,480 01-13-2017, 06:01 PM
Last Post: TheLostBit
  HTTPS for the forum and other webpages baryluk 1 1,231 01-02-2017, 02:25 PM
Last Post: tllim
  Where is the posts from the old forum irishbiker 4 1,968 05-04-2016, 03:20 AM
Last Post: Ghost
  Intel launches $250 kit to build robots with Kinect-like vision Shady2tree 2 1,879 04-19-2016, 07:28 AM
Last Post: Shady2tree
  Forum Revamp Ghost 20 8,002 04-19-2016, 06:22 AM
Last Post: tkaiser
  Wiki Update please norm24 8 3,869 03-12-2016, 11:56 AM
Last Post: Tarjei85
Information Improving the forum Zoidiano0 9 4,223 01-24-2016, 06:43 AM
Last Post: Ghost

Forum Jump:


Users browsing this thread: 1 Guest(s)