scary.png Happy Halloween! Лови жутко страшный список шаблонов и модулей со скидками.
Jump to content
Five5555

Нагрузка на сайт поисковыми ботами Robots.txt

Recommended Posts

Всем привет!

Постоянно отключается сайт из-за превышения лимитов. Нагрузку создают яндекс и гугл боты.

Помогите настроить robots.txt, чтобы снизить нагрузку

Вот тот, который используется сейчас:

User-agent: *
Disallow: /*index.php*
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*?tracking=
Disallow: /*&tracking=
Disallow: /m/
Disallow: /mobile/
Allow: /catalog/*.css
Allow: /catalog/*.js
Allow: /catalog/*.png
Allow: /catalog/*.gif
Crawl-delay: 15

User-agent: Yandex
Disallow: /*index.php*
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /m/
Disallow: /mobile/
Allow: /export/yandex_yml.xml
Crawl-delay: 10
Clean-param: tracking

Host: https://sfera.by
Sitemap: https://sfera.by/index.php?route=feed/fast_sitemap

 

Share this post


Link to post
Share on other sites
1 час назад, Five5555 сказал:

Всем привет!

Постоянно отключается сайт из-за превышения лимитов. Нагрузку создают яндекс и гугл боты.

Помогите настроить robots.txt, чтобы снизить нагрузку

Вот тот, который используется сейчас:

User-agent: *
Disallow: /*index.php*
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*?tracking=
Disallow: /*&tracking=
Disallow: /m/
Disallow: /mobile/
Allow: /catalog/*.css
Allow: /catalog/*.js
Allow: /catalog/*.png
Allow: /catalog/*.gif
Crawl-delay: 15

User-agent: Yandex
Disallow: /*index.php*
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /m/
Disallow: /mobile/
Allow: /export/yandex_yml.xml
Crawl-delay: 10
Clean-param: tracking

Host: https://sfera.by
Sitemap: https://sfera.by/index.php?route=feed/fast_sitemap

 

яндекс и гугл не создает, там штук 500 ботов которые врываются на сайт и по 500 раз открывают страницы в секунду, и это не яндекс и гугл, поставь модуль lighting , у меня стоит, мне эти петухи вообще не страшны) хотя они и так не особо создавали, но были пару штук которые реально открывали 500 страниц за секунду 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
You are posting as a guest. If you have an account, please sign in.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

×

Important Information

On our site, cookies are used and personal data is processed to improve the user interface. To find out what and what personal data we are processing, please go to the link. If you click "I agree," it means that you understand and accept all the conditions specified in this Privacy Notice.