ruby - EventMachine: What is the maximum of parallel HTTP requests EM can handle? -

ruby - EventMachine: What is the maximum of parallel HTTP requests EM can handle? -

I am creating a distributed web crawler and am trying to get the most out of every single machine's resources. Use the em-http-request to create an asynchronous HTTP request through an event, and use em-http-request to create an asynchronous HTTP request. For now I have 100 iterations that run at the same time and it seems that I Can not move beyond If I increase many iterations then it does not affect the crawling speed. However, I get only 10-15% CPU load and 20-30% network load, so the crawl in many rooms is fast.

I am using Ruby 1.9.2. What is the method to improve the code effectively to use resources or am I also doing it wrong? def start_job_crawl @ redis.lpop @queue do | Link | If link.nil? EventMouseine :: add_timer (1) {start_job_crawl ()} and # parsing link, using the asynchronous HTTP request, content pars (link) end and end and #main reactor loop emeran {e.m.k.k.q. @ RDIS = EM :: Protocol :: Redis.connect (: Host => "127.0.0.1") @Radisback. Code | Say "Radis error: # {code}" end # 100 parallel 'thread' This EM :: Iterator. Increase NY (0..99, 100) Unit | Number, iter | If you are selecting ()

      (Which is the default for EM), the maximum is 1024 because limited to 1024 file descriptors () is limited.  
 Although it seems that you are using  kqueue , then it should be able to handle more than 1024 file descriptors at a time.








03:22

















Get link





Facebook





X





Pinterest





Email





Other Apps




Comments





Post a Comment







Popular Posts




google chrome extension - Reason for Uncaught Error: Attempting to use
a disconnected port object -



    While retrieving the request from the content script, I get this error in the background page. Does anyone know that this error What can be the reason?   Full stack trace:    Unwanted error: Attempt to use Chrome / Renderer Extension Binding Disconnected Port Objects: 147 Chrome.port Post Message Chrome / Renderer Extension Binings: 147 ChromeHird.port DicePatchOnConnect.Connect Event Chrome / Renderer Extension Binings: 89IExtension.fu SendResponse.state background.js: 1573db.readTransaction.tx.executeSql.paramStr background.js: 1038    This only happens when Reloading extension is not being helpful for a few hours - it is not that after restarting the Chrome browser it will be a solution that becomes normal for a few hours, content script may continue to send requests for background No response can be sent from the background. Is there any way that I can catch this unwanted error and reset the listener?   I am using chrome.extension.onRequest.addListener for my communication. Bef...






visual studio 2010 - How to reinstall MVC 2 tools for VS2010? -



    I uninstalled ASP.Net MVC 2 device for Visual Studio 2010. How can I restore it? MVC2 downloads seem to include only the VS2008 device.   Whatever I said, says that MVC 2 has been included in VS2010, but the installation of an improvement did not fix it, and MVC 2 is not listed. Add VS2010 / Select a component in the Remove screen.      You will get your VS2010ToolsMVC2.msi VS2010 DVD under WCU \ ASPNETMVC - hopefully it will do!    






Git Bash script to check whether repo has any commits? -



    I have a script on which I need to work differently if there is one, which is one or more , Commits. What is the best way to do this?   Something like this will appear in the proxy code.    #! / Bin / bash if [[`git log_count`] ==" 0 "]]; Then there is no commodity for this repo. # What goods ... and echo are "one or more committed!" # What are the other things?    Any ideas?      you can    git rev-parse -verify HEAD    (after :)     This is a pipeline command, and its behavior vs. "  git diff " There is very little possibility to change   I find it more pronounced  rev-parse -verify  means "is this a valid object name?"    This is your  show-ref .  You ask "Is there anything in this safe / head in the safe?"  I ask, "what current  head  exists?"   In practice, both of these are fair tests, because once you have a branch, so to achieve the  HEAD , some mock values will indicate manually lack of editing It is very diffic...








Powered by Blogger