mysql - CPU High Load because of search engines mass crawling -


i have server containing many web sites, , when sites have huge updates, search engines :

bing (specially), yandex, yahoo

come server using many ip addresses , don't attention cpu load , mysql queries , try fetch web pages in same time,

so how can limit them whole server

currently blocked these search engine ips thank you

you have create robots.txt search engine read , know not allowed index (crawl) whatever specify on it.

for bing, instance, here guide: https://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec


-

Popular posts from this blog

node.js - How do I prevent MongoDB replica set from querying the primary? -

read-operations-only: is there way have mongodb replica set, make mongodb instance on box connected to, mongodb gets queried? i have 3 ec2 instances behind aws load balancer. on each ec2 instance runs mongodb, part of replica set. i have express endpoints on nodejs , connect replica set follows mongodb.mongoclient.connect('mongodb://1.1.1.1,2.2.2.2,3.3.3.3.', function (err, db) { if (err) { complete('{}'); } else { i distribute load of queries evenly across 3 instances of mongodb replica set instead of default having queries routed ec2 instance primary mongodb defined, because destroys point of load balancer (which balance load of queries across 3 instances evenly). my understanding when connecting mongodb replica set, primary instance selected unless primary instance down, , therefore purpose of every secondary instance serve backup. in example people call hot,cold,cold. because there 2...
Read more

c# - Randomly pick a specific int from a 2D Array -

i'm trying randomly specific integer (1) 2d array list filled 0's , not many 1's. made this, , works: while (wallslist[randomx, randomy] != 1) { randomx = randomizer.next(34); randomy = randomizer.next(34); } the downside of it, it's takes time find 1 time int (1), , have process on 1000 times since new 1's added , removed 2d array each time. takes 3m launch program know if there optimized version of this, searched lot , found solution 1d arrays. time. you have sparse array. why not represent list of x/y int pairs? then, if x/y int pair in list, it's 1, if not, it's 0. then, find random value/cell containing 1, pick random value list. you use list like new list <tuple<int, int>> { new tuple<int, int>(1, 5), new tuple<int, int>(2, 7) }
Read more

php - Angularjs http.delete is not working after deploying project on server -

i working on laravel5.1 project. while working on localhost using xampp http.delete works fine after hosting project on remote server delete option not working though other properties working fine. here portion of code $scope.removesaletemp = function(id) { $http.delete('api/saletemp/' + id). success(function(data, status, headers, config) { $http.get('api/saletemp').success(function(data) { $scope.saletemp = data; }); }); } 'api/saletemp/' goes saletempapicontroller public function destroy($id) { saletemp::destroy($id); } view part: <tr ng-repeat="newsaletemp in saletemp"> <td>@{{newsaletemp.item_id}}</td><td>@{{newsaletemp.item.item_name}}</td><td>@{{newsaletemp.item.selling_price | currency}}</td><td><input type="text" style="text-align:center" ...
Read more