+- PINE64 (https://forum.pine64.org)
+-- Forum: PINE A64(+) (https://forum.pine64.org/forumdisplay.php?fid=4)
+--- Forum: General Discussion on PINE A64(+) (https://forum.pine64.org/forumdisplay.php?fid=3)
+--- Thread: robots.txt on wiki and forum (/showthread.php?tid=1319)
robots.txt on wiki and forum - lvella - 06-09-2016
The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:
Code:
User-agent: *
Disallow: /Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
RE: robots.txt on wiki and forum - tllim - 06-10-2016
(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.
Code:User-agent: *
Disallow: /
Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
RE: robots.txt on wiki and forum - lvella - 06-10-2016
(06-10-2016, 01:38 AM)tllim Wrote:(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.
Code:User-agent: *
Disallow: /
Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).
RE: robots.txt on wiki and forum - tllim - 06-11-2016
(06-10-2016, 11:23 PM)lvella Wrote:I will explore with IT manager again, he against the idea on opening.(06-10-2016, 01:38 AM)tllim Wrote:(06-09-2016, 09:00 PM)lvella Wrote: The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.
Code:User-agent: *
Disallow: /
Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?
The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).