robots.txt on wiki and forum
(06-10-2016, 01:38 AM)tllim Wrote:
(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:

User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).

Messages In This Thread
robots.txt on wiki and forum - by lvella - 06-09-2016, 09:00 PM
RE: robots.txt on wiki and forum - by tllim - 06-10-2016, 01:38 AM
RE: robots.txt on wiki and forum - by lvella - 06-10-2016, 11:23 PM
RE: robots.txt on wiki and forum - by tllim - 06-11-2016, 08:55 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  A dedicated forum for guides and tutorials? shockr 5 6,994 01-16-2017, 04:39 PM
Last Post: tllim
  deutschsprachiges Forum g0su 15 20,258 01-13-2017, 06:01 PM
Last Post: TheLostBit
  HTTPS for the forum and other webpages baryluk 1 3,194 01-02-2017, 02:25 PM
Last Post: tllim
  Where is the posts from the old forum irishbiker 4 5,886 05-04-2016, 03:20 AM
Last Post: Ghost
  Intel launches $250 kit to build robots with Kinect-like vision Shady2tree 2 4,343 04-19-2016, 07:28 AM
Last Post: Shady2tree
  Forum Revamp Ghost 20 25,369 04-19-2016, 06:22 AM
Last Post: tkaiser
  Wiki Update please norm24 8 11,260 03-12-2016, 11:56 AM
Last Post: Tarjei85
Information Improving the forum Zoidiano0 9 11,294 01-24-2016, 06:43 AM
Last Post: Ghost

Forum Jump:

Users browsing this thread: 1 Guest(s)