User-agent: GPTBot Disallow: / User-agent: Sogou Disallow: / User-agent: Sogou web spider Disallow: / User-agent: * Allow: /p/*/ci/master/tree/$ Allow: /p/*/ci/main/tree/$ Allow: /p/*/ci/default/tree/$ Allow: /p/*/HEAD/tree/$ Disallow: /p/*/code/ Disallow: /p/*/git/ Disallow: /p/*/svn/ Disallow: /p/*/hg/ Disallow: /p/*/search Disallow: /p/*/code-0/ Disallow: /p/*/tree/ Disallow: /p/*/search_feed/ Disallow: /p/*/search_help/ Disallow: /p/*/wiki/*/edit Disallow: /p/*/wiki/browse_tags/ Disallow: /p/*/discussion/create_topic/ Disallow: /p/*/discussion/stats Disallow: /*/ci/ Disallow: /*/commit_browser Disallow: /*/commit_browser_data Disallow: /*?barediff= Disallow: /*?diff= Disallow: /*/bugs/new/ Disallow: /*/support-requests/new/ Disallow: /*/patches/new/ Disallow: /*/feature-requests/new/ Disallow: /*/tickets/new/ Disallow: /*/attachment/ Disallow: /projects/*/files/*/stats/timeline Disallow: /projects/*/files/stats/timeline Disallow: /projects/*/files/*/stats/map Disallow: /projects/*/files/stats/map Disallow: /projects/*/files/*/stats/os Disallow: /projects/*/files/stats/os Disallow: /projects/*/files/stats/json Disallow: /projects/*/files/*/stats/json Disallow: /projects/*/files/latest/download Disallow: /projects/*/latest.json Disallow: /projects/*/report_inappropriate Disallow: /projects/*/reviews/new Disallow: /projects/*/rss Disallow: /projects/*/lists/*/unsubscribe Disallow: /projects/*/moderate_review Disallow: /projects/*/rate_review Disallow: /projects/*/best_release.html Disallow: /projects/*/postdownload Disallow: /projects/*/get_updates Disallow: /*_escaped_fragment_ Disallow: /auth/ Disallow: /auth/do_login Disallow: /software/visit? Disallow: /software/compare/*-vs-*-vs-*-vs-*-vs- Disallow: /software/compare/*/add-software Disallow: /software/product/*/reviews/new Disallow: /software/product/*/claim Disallow: /software/vendors/new Disallow: /software/vendors/inquire Disallow: /software/*?q= Disallow: /software/*&q= Disallow: /rest/ Disallow: /user/registration Disallow: */bin_counts Disallow: */milestone_counts Disallow: */stats_data Disallow: *//users$ Disallow: */users$ Disallow: /u/*/link/ Disallow: /p/*/link/ Disallow: */feed.rss Disallow: */feed.atom Disallow: /p/*/news/feed$ Disallow: */activity/feed Disallow: */activity/pjax Disallow: /p/*/admin/ Disallow: /directory/release_feed Disallow: *?css-reload= Disallow: *&css-reload= Disallow: /settings/mirror_choices Disallow: /directory/tp Disallow: /software/tp Disallow: /sd-jobs # /directory param rules: permit ?q= and ?page= on their own, but no other params # longest match takes precedence. listed in length order Disallow: /directory/*? Allow: /directory/*?q Disallow: /directory/*?q*& Allow: /directory/*?page= # mailman flags Disallow: *?style=flat Disallow: *&style=flat Disallow: *?style=threaded Disallow: *&style=threaded Disallow: *?viewmonth= Disallow: *&viewmonth= Disallow: *?viewmont= Disallow: *&viewmont= Disallow: *?viewday= Disallow: *&viewday= Sitemap: https://sourceforge.net/sitemap.xml Sitemap: https://sourceforge.net/allura_sitemap/sitemap.xml Sitemap: https://sourceforge.net/directory_sitemap.xml Sitemap: https://sourceforge.net/software_sitemap.xml Sitemap: https://sourceforge.net/blog/sitemap_index.xml Sitemap: https://sourceforge.net/articles/sitemap_index.xml Crawl-delay: 1