robots.txt on wiki and forum

[lvella]

The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:

Code:

User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?

[tllim]

The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:

Code:

User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?

We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

[lvella]

The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:

Code:

User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?

We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).

[tllim]

The lack of resources about Pine64 on search engines was suddenly explained when I found out that both the forum and the wiki had their "robots.txt" files configured to forbid indexing:

Code:

User-agent: *
Disallow: /

Can the person responsible for the domains please enable indexing of the contents and make our user lives easier?

We purposely disabled this due to it will generate a lot of soliciting traffic and slow down forum.

The forum allows only one search per 30 seconds, but Google gives all of us for free multiple searches per second. I don't know what is the burden of the bots on the servers, but looking for where I stand, it seems hardly worthwhile to forbid them, given the benefits (and that only good bots honors robots.txt).

I will explore with IT manager again, he against the idea on opening.