Sounds like a weird reason to switch the language and framework used for your product. Blocking libraries block just the same in Node and in Twisted. Why not just stop using the Django ORM or more generally, stop writing code that blocks the main thread for long periods of time?
The problem of the main thread waiting on a response from the database is easily solved without rewriting all your code, which incidentally will have the same issues, since there's no new magic there.
If your software is a giant, ugly hairball because you wrote it in a hurry (I know I did this many times!) don't blame it on the language or framework. What I'm wondering is: Python/Twisted/Django rank equally or similarly on all the points from the list, except for Team Experience, which has to be much higher with the old stack than with a completely new technology and set of tools. This makes me think that there was a "play with shiny things" point after all.
It's been a while since I've written nodejs in anger so I may be out of date, but as a rule libraries in Node don't block, they present async interfaces and do what they have to in order to keep from blocking.
Python has no such conventions, making many libraries into dangerous minefields for the Twisted developer.
I'm personally not a huge fan of node, and I prefer to write non-blocking code in Aleph/Netty (which is closer to twisted in terms of potential to block due to blocking java libs) myself.
But, I can understand on a large team not wanting to deal with the potential for blocking libs.
That said, if performance is that important, it should be part of your test suite, and should catch that.
I've come up with the same conclusion regarding the Django ORM, as they have. An excerpt:
I believe our single biggest mistake from a technical side was not reigning in our use Django ORM earlier in our applications life. We had Twisted services running huge Django ORM operations inside of the Twisted thread pool. It was very easy to get going, but as our services grew, not only was this not very performant, and it was extremely hard to debug
Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements.
Also for many applications I find Django to be a huge overkill - and in more recent times I've focused a lot more energy using microframeworks such as Flask and Bottle.
"Django has in many ways become the industry standard in regards to fast development - but it, as so many other ORMs, does not scale and produces horrible SQL-statements."
I disagree - many ORM's in python world? Try using sqlalchemy it will put django's ORM to shame, it will build fast good looking queries most of time, and if you have a rare case of something not good enough - you can easly tweak it. Can't say anything about Storm but saying that "industry standard" django ORM is OK because "others have same flaws" is just distorting reality to fit this thesis ;-) Ugly truth is django ORM is not ok and it bites many people when they are at the point where it's real pain to fix this - the more developers are aware of this the better.
I'm in the process of building an api in Django (GeoDjango, actually, with PostGIS). What are the gripes with the ORM and how could it be better? (Not that I'm disagreeing; I'm curious. I'm not experienced enough to know what makes an ORM okay or not.)
Theres a couple of issues i had with it personally. It doesnt work that well with nosql. Also i found it difficult to customize or do certain things that i wanted to do. Some of them very basic. For example i wanted to only return individual columns rather then all of them. I spent hours looking for how to do this. I finally found out you do .values or .select or something to that effect at the end of your call. Unfortunately i tried this and got errors. So i gave up in frustration.
.defer() and .only(), they're right in the list of methods on Querysets: https://docs.djangoproject.com/en/1.3/ref/models/querysets/#.... Sorry, but it's a bit hard to take this critique seriously, given how trivial it is. I have a list of complaints about Django's ORM that is about a mile long, so don't think I'll defend it blindly, but let's try to stay grounded in reality when we criticize something.
That mile-long list of complaints is what I'm after. Not to compel you to write a book about it (but if you did, I'd read it :-P), but what are some of those things?
No idea what defers and only are. Actually maybe only was what i was looking for. The django documentation for many things such as this is almost non existent. Some of it is from older versions as well. And im sorry but how is a newbie supposed to find this stuff. Youve probably been working with python and django for a while. Which is great. But when youre just starting and playing around having to spend hours looking simplistic stuff like this up when you have a million other things to do really gets frustrating.
Most people don't understand the work "complect" ([kəmˈplɛkt] vb (tr) Archaic to interweave or entwine). They probably (as I did) thought it was a typo for 'are complicated', making your whole comment a tautology. Or perhaps they thought you were trying to start a flame war by mentioning Rails.
I mentioned Rails because Django and Rails are extremely similar as they try to solve the same problem with similar approaches. The differences in how the two work come down to philosophy. Rails trades a lot more simplicity for ease of use with its "convention over configuration" approach. Django chooses more simplicity over ease of use.
If I were to diagram those two and Node.js, I'd say it looks like this:
Yup. It was such a good talk, I assumed it had made the rounds among HNers that I didn't need to link to is. I assumed wrong.
It is a brilliant talk. The only part that he left out of the discussion, but is worthy of a talk in its own right is the notion of the community (formation and characteristics) as one of the artifacts that result from the constructs of a language.
Python is a programming language. Node.js is not. What you meant to say is that you switched from Python (presumably running on top of a Real Web Server) to javascript running on Node.
The argument could be made that they are both interpreters, but please, get it right - you are writing javascript, which is a pretty shitty language in which to develop apps these days. The only reason we use it is because it has near-universal browser support - and then only for UIs.
Also, not to parrot, but it's true - node is cancer. It's for fad chasers who have no idea how real servers manage to serve volume.
Fortunately, most people inexperienced enough to choose node are inexperienced enough to never need to scale, so it'll work out okay for them - for implementation-specific values of 'okay'. :)
> you are writing javascript, which is a pretty shitty language in which to develop apps these days.
Woah. I do both Python and JS, and guess what - there are a fair number of people who hate Python's significant whitespace. And to them, it's also a "hacky" language (with all kinds of __foo__ stuff). So, shitty-ness is in the eyes of the beholder.
> It's for fad chasers who have no idea how real servers manage to serve volume.
There are real world apps like Trello, which is on Node, and are scaling well. Get your facts right, or stop trolling.
Yup, shitty-ness is in the eye of the beholder. I wish that every time a discussion of syntax came up, we'd be as quick to make reference to Wadler's law as we are to make reference to Godwin, when Nazi's are brought into the picture.
Arguing syntax is more often then not pretty absurd, unless you are arguing how the syntax actually changes semantics (like with S-expressions and overloaded parens).
I think it's amusing that when I claim that javascript is a shitty language, instead of trying to claim that javascript isn't shitty, you point out the shittier attributes of the language I was comparing it to.
Regardless of whether you like python or not, javascript pretty much sucks. Coffeescript helps a bit, but if you aren't running in browser, why hobble yourself like that?
The "Javascript is a shitty language" argument is getting old. I feel like you're only admitting that you're too lazy to learn how to program with a prototype-based language. You even suggest Coffeescript which is big on making Javascript like an OOP language when it's really not.
As far as a reason why you might "hobble" yourself is because Javascript is great at asynchronous design. You should try checking it out for real instead of jumping on the opinion bandwagon.
How does the age of the argument impact its validity.
> you're only admitting that you're too lazy to learn how to program with a prototype-based language.
If you don't like Erlang you are just admitting you are too lazy to learn the actor model. See I'll just call you "lazy" it will validate my argument.
> because Javascript is great at asynchronous design.
Not disagreeing with you here, but it would help if you'd actually mention why. You said something about it being a prototype based language. Is that why you think it is great at asynchronous design?
The language is limited by IE dragging their feet on unbreaking their implementation or advancing standards, Webkit and V8 staying at a very conservative middle ground (Google pushing Dart and compilation as substitutes), and Mozilla being the only engine developer showing willingness to evolve the language, because they are the only ones using it to implement the rest of their browser. Node.js is held back by a language implementation they don't control and feels underpowered in the language department, indirectly being a victim of resistance to change in the browser landscape.
Python and other non-browser languages are far more powerful and expressive, and — since you refer to Python 3 — willing to further improve.
__call__; __new__; __getattr__ and __getattribute__; and __metaclass__ - you can do some black magic in Python straight out of the box that takes a lot more work in statically typed languages.
Or methods like __init__, __str__, __cmp__, etc. I know they are part of Python's protocols, which are basically a form of the template method pattern, and that they do make sense. But to some people they feel hackish; they woud prefer to have "normal" name methods for these protocols instead.
For example, I think if you'd have to explain how to change the string representation of some kind of object, saying "just define the 'str' method and then call it, like 'something.str()'" is simpler than "define a '__str__' method -- don't worry about all the underscores -- but then don't call it directly; instead of that, use the global 'str' function".
I don't know if that extra indirection (i.e. calling 'str', which, in turn, calls '__str__'; or 'next', that calls '__next__' in Python 3) adds more value than the simplicity of not having that extra indirection in the first place.
If it used normal names, Python couldn't evolve new methods for its protocols with language support. It would need some kind of annotation saying “yes I really mean to get this method called by the language and didn't just happen to use the same name by coincidence”.
> You don't actually need that though - there is a completely obvious way to write stuff without having to use that line.
Can you elaborate on that point? I thought that was the canonical way to do exactly that... http://docs.python.org/library/__main__.html. If there's an easier/less verbose method (pun not intended) I'd love to know about it.
It is this environment in which the idiomatic “conditional script” stanza causes a script to run
The conditional script stanza is optional, and intended to make a .py file function as both a script and a module (i.e. be importable). To me, it's unclear why you would want to do that. The only thing you have to do to avoid using __main__ is not require any file to be both a script and a module/library. Then just put any entry-point code at the top level of whatever script invokes your program. Unfortunately the __main__ stanza is pretty widely used though.
On the topic of modules, there is a wrinkle in requiring __init__.py to be present in any library subdirectory that you want to be part of the namespace. It would be nice to have a more intuitive way for that.
Oh, I do it all the time. It allows me to write a script that I can just run from the command line when I want, like 'fetch_new_images'.
I especially love it when I'm first coding, cause I like to test as I go. That allows me to run it to see what works and what doesn't.
When I'm finished, I know that I can just plop that file into a directory and use it as a library afterward. The majority of the code that I write implements that, and I find it a godsend, personally.
> It's for fad chasers who have no idea how real servers manage to serve volume.
I think that pquerna has some pretty good ideas about how "real servers" work, but leaving that be, you could actually explain what's wrong with the system, rather than attack the people.
Having some Erlang experience myself, I can find plenty not to like about node.js. However, the Erlang guys have fired round after round into their own feet to the point where even if it's inferior, node.js has probably already surpassed Erlang for many server tasks.
They had, years ago, a clearly superior platform for event-driven programming (none of these weird callbacks). They have more or less failed to capitalize on it in terms of getting mainstream traction.
"It's for fad chasers who have no idea how real servers manage to serve volume"
If you look at who's building serious systems with Node.js, that statement doesn't hold up. CloudKick / the Rackspace monitoring team are hardly amateurs - neither are Joyent, LinkedIn, eBay or Yahoo.
Facebook didn't choose PHP. They continued to use it, as there was a lot of legacy to port if they wanted to.
These people actively measured (using a spreadsheet at that) the various parameters of a programming language and settled the one that fits their requirements.
Exactly! Facebook continues to develop in PHP because throwing out your existing system and replacing it with new code base (and a new language!) is almost always fraught with risks.
That spreadsheet is not "science" even though it has numbers in it. I can put higher numbers for Python and arrive at a different answer.
Wow. What are you so frightened of? It's obvious from your post that you have invested a great deal of time in python and you are terrified that Node is going to take away your livelihood. That is not going to happen. Python is here to stay, just as Javascript is. Both will continue to be improved over time, but there is no reason to freak out and lash out at other people. Relax.
And I'd like to see the results of your study where you have proven that "most people inexperienced enough to choose node are inexperienced enough to never need to scale". You should publish such a bombshell study.
But in truth, your fear is causing you to make nasty attacks and, well, just make shit up. Come on, just admit it, Sneak.
Were it that Python was my livelihood my life would be a lot simpler. Instead, my livelihood comes from unfucking bad decisions like "hey, let's use PHP!" or "hey, let's rewrite the whole thing in javascript and use node!" or "MongoDB seems great, let's switch from MySQL!"
It's not fear of new, it's fear of broken. I encourage you to use whatever shiny stack-of-the-month you enjoy, as it basically amounts to my competitive advantage over people like that. I'll be over here using Apache and nginx and MySQL and python and perl and other languages that don't make me write abominations like '==!', same way I have forever. Best of luck to you.
"It's obvious from your post that you have invested a great deal of time in python and you are terrified that Node is going to take away your livelihood."
No it's not. You're just making one hell of an assumption.
I blame them and not the technology stack they used. It's idiotic to mix blocking technologies with non-blocking technologies as blocking the IO loop is fatal for performance as you won't be able to process any events while the IO loop is blocked.
And this person still has not learned from their mistake, they think that using SQLAlchemy would have solved their problem. If they wanted to make Twisted work they should have picked a database that can be used in a non-blocking way.
The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop.
SO, basically you just learned about non-blocking and think node.js is a golden bullet. It's not.
And it's exactly the same situation as anything they could have used in Python, as soon as they add actual code to get work done (as opposed to hand over work and wait for a callback).
"""The reason why I like node.js is that everything is built in a non-blocking way. Twisted has the same philosophy until people think they can get away using a blocking library inside the IO loop."""
That's not what they were doing, though. At the end of the non-blocking fiesta you want to actually return results to a waiting connection -- actual results, rendered templates etc. That will block and take time -- especially in node.js which is single threaded.
So they just traded Python's threads/processes/lightweight threads for node.js processes.
No, I did not learn about non-blocking approach recently. I have built solutions in Java and in node that scaled to over 300.000 open connections pocessing billions of messages. Besides that you are missing my point, which basically is to not mix blocking with non-blocking code.
This is another reason why i moved off of django. This arrogant attitude of belittling other people when they have problems. I expect this from the rails community not from python. However django is increasingly becoming a "its the django way or the highway buddy". Which is dissapointing frankly.
Nice how i get voted down. Thus proving my point about the python and django communities. So glad i moved to node. I will still consider using python for backend stuff though.
Your point has not been proven. You should forget the meta stuff (python people are like this, django people are like that) and counter with arguments when someone attacks your point of view on the topic at hand. No one knows who you are, just say what you have to say (about the topic) based on your experience. If you draw conclusions, strengthen them with data if possible, or argue why you believe that they're valid. People will call you on them if they feel you're in the wrong. It doesn't mean they hate you. You should do the same.
Also, the tone on a keyboard is often easily abrasive. The pen is mightier than the sword. The keyboard is harsher than the mouth. Don't fall for the bait focus on the arguments.
The problem in part is with hackernews actually. I dont find it very conducive to good discussions. I rarely comment myself actually. Most comments seem to be he said/she said and argumentative. Im actually surpised people dont ask for this site to be improved. And i guess it never will. But it would be nice to have constructive conversations about these things rather then arguments.
On any site where there is a voting mechanism, you're going to have a certain amount of groupthink. And likewise, on nearly every such site (especially HN), complaining about being down voted is often like throwing gasoline on a fire.
There has been discussion about how to improve the site, sometimes from PG himself, sometimes from others. It isn't perfect, but in comparing the signal-to-noise ratio of HN to some of those other sites, I find that HN shines. Do you have any specific ideas on how to improve things here?
I've been following this comment thread and I have to say that I find most of the argumentative and/or confrontational attitudes to be a direct result of your complaining. If you don't like Django that is fine. But don't expect others, who not only use but may also contribute and have a strong vested interest in it, to sit by idly while you trash their work and belittle the character of their community.
The depth of libraries available in the Python or JVM ecosystems is going to dwarf what's available for Node. Were you able to find everything you need for Node?
The actual number for Java is a little bit skewed probably. You're only using one maven central repository (which is already big enough). But there are other repositories as well such as RedHat's (JBoss, Hibernate, etc). SpringSource's (Spring frameworks, grails probably, and everything under SpringSource umbrella).
And also keep in mind that scripting languages have higher number largely because people use them for writing scripts/command-line as well. And if they insist to use Java for scripting well then... they deserve to be punished by walking up and down the stairs 20x per library per decision.
I would venture to guess that the lack of such libraries is more the result of community focus.
The rails community (and by extension ruby) is generally more product focused, whereas the python community (and by extension django) are more science focused.
I intentionally ordered the language and the framework differently for each because most people who use ruby/rails, got into that community for the sake of the framework, whereas most of the people who use python/django, got into that community for the sake of the language.
Finally, python has a much greater adoption among in the academic world, so it's natural to expect that its community will have built more academic libraries like those you mentioned.
Node.js and Javascript are seeing adoption among those interested in real-time applications, evented systems and applications that bridge the chasm between the client and the server, so it is natural to expect that more libraries focused on problems within those domains.
One thing I like that I've seen from the node.js community is that the fact that they are developing with the same language on both the client and server, that they consider them one in the same. The only thing separating the two is latency and connection reliability. The latency issue is psychologically not much different than having a bias for doing things in memory and avoiding disk IO server-side. The lack of connection robustness is likely to evolve into solutions that mirror the problems that the erlang/OTP community has spent a lot of time solving.
The ecosystem around languages are heavily influenced by the strengths and weaknesses of the communities that adopt them, and the community that adopts them is largely the result of which problems that language is well suited for either by language design or historical coincidence (e.g. Javascript is in the browser)
Count of modules seems like a useless metric. A better one would be modules that are considered ready for production. Even then, the only ones that matter are the ones you will end up needing for a project.
Having said that, the only ones I consider moderately painful is when we need to deal with node-thrift, which on the Python or JVM is well maintained by existing groups.
Part of this though is because web apps are for better or worse increasingly being almost nothing but javascript. And it becomes tedious moving back and forth between javascript and another language. Even if that language is really good. So i think people are just going with javascript for "everything" and rewriting their libraries in it.
How does it calculate the size of the code base for each language? Most people would be using jquery as a minified oneliner. How can that inflate the numbers so much?
It doesn't measure line length, which could be a good way to detect minified JavaScript. It uses file extensions and byte size. There is some exclusion of commonly bundled libraries, but it's a rough heuristic on file paths.
Of course it depends what libraries youre looking or. If youre looking for libraries for build the "basics" of a web app then there are plenty of libraries available. If yore doing other things though and looking for libraries that dont exist maybe its because youre using the wrong technology.
Not necessarily. Sometimes the lack of libraries isn't solely because it is the wrong technology, but because it is the wrong community. Language and community are pretty intertwined.
LISP for example is a great technology for a lot of things, but library-wise it suffers because within that community there is a tendency for people to "roll their own" solutions.
Java has tons of libraries, but I'm not sure most people would say it's the right technology for a lot of things. It's generally the right technology for projects that involve lots of developers with mixed levels of ability (before some Java person down votes me for this, consider that Gosling himself made this point back in the mid-90s when he invented the language)
so what makes javascript callbacks easier to understand than twisted's deferreds? aren't they basically the same thing?
and how does "we picked Node.js" logically follow from "It is obvious that the JVM platform is one of the best ways to build large distributed systems right now"?
What I posted was just where it was left after a meeting 9 months ago.
Yes, we definitely weight against our feelings of how we failed to employee Twisted Python. This was our experience, are you suggesting we shouldn't consider our experiences when evaluating something?
It really came down to a choice between pursuing a JVM based system, and Node.js.
Yup, we probably did want to do something new. But guess what, its worked. I'm sure we would have haters the other direction if we were posting about how Node.js has failed us, but it hasn't, yet.
i'm surprised that you chose a technology so similar to one that you had problems with before. especially when the deciding factor seems to have been "it's exciting" rather than "it does the job well". yes, you can argue that "exciting" is important to attract a top-notch team, but then i am not sure why a top-notch team would have problems with twisted (and not have the same general issues with node).
how will node fix the issues you had with twisted?
Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. Including as i mentioned i blieve in another comment here that using node allows you to only have ot work with one language. Which is great. So its true that python has similiar technologies such as tornado and twisted. But if node works then why not just stick to one technology to have to maintain rather then multiple ones. Also node has a great community that is increasing. Many python libraries dont have this. Meaning really documentation. For example tornado has very little documentation.
"Id have to disagree with that myself. While node and mongodb are both "shiny" they are because they solve a problem that people have and offer solutions to problems that are very appealing. "
A lot of people use them for solving problems they DON'T have.
Lots of people use both Mongo and Node.js at ill-fated attempts at premature scaling (similar to premature optimization).
Again i will disagree with you on that. One of the benefits of node and why its becomingly increasingly popular is that it allows ease of development with the benefits of scale built in. So you dont have to worry about a steep learning curve for a simple application that only a few people will ever use. Or that youre framework is easy to develop with but what happens if it becomes popular. What then. This is even more the case with todays applicaitons which incrasingly are "api" based with the web framework portion being used just to serve the ajax code and for urls and whatnot. With node you get all of it together.
Also if node used some other language that wasnt a standard this might also be true. But again it uses javascript or coffeescript which is becoming more and more used all f the time. Even in mobile applications and gaming now with canvas. Obviously if you know python or ruby or php extremely well and can build whatever you need quickly with that then theres no reason not to. I was new to python so it wasnt a major switch for me.
Node is popular because the initial learning curve is so easy, but I think it's a bit deceiving. There's a considerable step up from writing a basic node app and building a full "real world" app.
When you first start out, you don't realize that flow control is going to be an issue, you don't realize that you're likely going to need to figure out how to do stream processing. Those aren't especially difficult to learn, but it's a delayed learning curve. You don't know that you'll need them until you run into a dead end. There are libraries/modules that help with that now, but it's not trivial to figure out which is needed (at least not for someone just getting started).
Node is an interesting tool, and I think it solves some issues elegantly. I think some people use it for projects it's not suited for, but that happens with every language/framework/tool.
I want to scale my app on modern, multi-core computers. I know, I'll write everything so that it all runs in a single thread!
This is the same fallacy that gets repeated every time node is mentioned. If you're scaling up past a single machine you have to figure out how to share state between multiple machines anyway, so you may as well run a process per core. Then you have a single way to share state, rather than one way between cores and another between machines.
So unless you're writing a desktop app that needs to scale up but only to all of the cores of a single machine, you're sometimes better off simplifying the state-sharing logic.
Uh, that just proves my point that you don't need a multithreaded app to do so.
And your 'cancer' language is unhelpful. What are you afraid of? That somebody somewhere is using a bad language when a better one is available? That the NodeConf organizers reserved all of the good hotel rooms?
I don't think that node.js is doing either of those things. How does one quantify "sinking down the drain"? It's not zero-sum. How is Cloudkick writing services in javascript any worse or better than Cloudkick writing those services in python or java?
I think the real danger is people thinking they've learned everything there is to learn and closing their minds. Which includes telling people that "node.js is cancer". Node is letting people do all kinds of interesting things, and it isn't killing off anybody's favorite language.
"""One of the benefits of node and why its becomingly increasingly popular is that it allows ease of development with the benefits of scale built in."""
And that's --the second part-- is where they and you are wrong, as tons of teams have found out. Scale is never built in, you have to work for it.
"Scale built in" is another way of saying "Mongo is webscale".
They should be alright in terms of adding new features. And if Node.JS hits scalability issue? well tough luck I suppose.
I see that JVM is a very serious contender but as already noted: License issue. And let me add this: I'm betting my ass that there's the "I don't want to write Java" whisperers as well. Now now, don't BS me. I know how the so-called "engineers" think of Java these days. Sure, you could use Clojure, Scala, or JRuby. But seeing the competitor is Node.JS, pretty easy choice don't you think?
My experience writing JS code for Node.JS based platform (ExpressJS, etc) is that there's a bigger chance that you'll write more code. Code to make the code more modular (sounds weird isn't it?). Code to make sure you work-around the warts of JS. Each line of code should be highly scrutinized due to JS warts.
This thread is going to be a typical fun nerds-fight. I guarantee you. I'm going to grab a popcorn and watch nerds doing keyboard slamming.
I'd really like to see that spreadsheet, since there are many python frameworks other than twisted available. There are eventlets/greenlets, there's gevent, there's diesel and monocle for minimal things, etc. Makes you wonder what actually made them change the whole technology... Than again, they could probably even reuse pyjamas to still write node.js code in python...
OT: "We should of used SQLAlchemy. We should of built ..." I thought this expression was Bristol-specific. Apparently it's taking US too.
In the Southern US, at least, "should have" commonly gets shortened into "should've," which sounds sorta like "should of." That's still no excuse for writing it.
If you want to use a reactor pattern and you don’t want to use JavaScript, you can use EventMachine on Ruby. Twisted is the same thing for Python. I’m sorry to hear you were having problems with Twisted, but EventMachine is absolutely rock solid. I have gotten a lot of work done with it.
Now, let’s say you don’t want to do callbacks, but still take advantage of the Reactor pattern in EM. I wrote a patch for Sinatra that uses Fibers to wrap callbacks, hence allowing you to continue to program synchronously as you used to, and it’s called Sinatra Synchrony (http://kyledrake.net/sinatra-synchrony).
But in order to do non-blocking IO, you don’t need to use a Reactor pattern, because Ruby internally does not block on IO (the GIL does not affect this). And of course you get solutions like JRuby (built on the JVM), which provide for threads (and Rubinius 2 coming soon).
My APIs written with any and all of these concurrency options can get thousands of hits per second. It’s quite scalable, but still provides all the rich libraries, reusable code, and testing support that makes my APIs high quality, which to me (and my users) is more important than making them fast. With this approach, my APIs are both high quality and fast. And I have never experienced a single crash of any worker processes in production. I would know, because my process monitor is smart enough to observe the workers in my deploy code and informs me, while adding another worker to replace the fallen one. Did I mention I can do zero-downtime hot deploys? Again, all implemented in Ruby.
I’m not trying to brag here, but there’s this weird current of arguments that these existing solutions don’t scale. It’s a myth, I don’t know where it comes from, and I’m worried that if I don’t speak out against these arguments, I’m going to wake up in a giant pile of half-baked Javascript spaghetti written by people that didn’t even understand the real problem in the first place (blocking IO people, research it!).
After developing a v1.0 of a project with django i ended up also moving to node.js as well as mongodb as opposed to postgresql. I found myself increasingly wrestling with django. I really like python and the django community and think django is good for certain types of applications. But at least for what im building it wasnt the right solution. Im also using node.js for another project which is also currently built with django. It has different requirements and i might use python on the backend whereas the other project will most likely just be node.js and mongodb.
I'm curious as to why Go wasn't a final contender? It started off with Go being a crowd favorite and ended with it nowhere. I assume it has to do with the young package repository but what's the reasoning you used?
Go's not-in-core packages are a mixed bag. Lots of "one off" experiment projects. Which is fine, thats where it is in its life. Node.js was in the same place packages wise 18 months ago.
I've used fibers-enabled EventMachine in one part of our application. It's pretty nice, because you don't have to do callbacks and the Ruby fibers are pretty nice actually.
We're using it for handling jobs from Resque, which reads and writes from our database and do a ping to a 3rd party server.
But. The amount of bugs in the em-enabled libraries is just amazing. For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers. Also Resque blocks by default, so of course doing a non-blocking and non-forking version was a priority.
The worst thing here are the error messages and backtraces. There are none (DeadFiberException for failing assert) and with hacks you can get out a bit more information if the job crashes.
Now thinking this again, it would be a good idea to write it again with Node.js. Just because it's still more mature compared to EventMachine, and all of it's libraries are reactor core friendly by default.
>For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers.
That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
The thing is - my first thought would have been to leave the ORM alone and focus on the DB side (like more sophisticated connection pooling/pgbouncer, etc. ). Which is why I'm interested in what went wrong in the ORM that made it screw up when used in a non-blocking kind of a setup.
> That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
I will do a blog post about this when I finally have some time (christmas holidays, maybe). Em-activerecord worked very nice first, but when I hit it with thousands of concurrent jobs, some workers just dropped dead saying MySQL couldn't answer. This was annoying, because the workers didn't really fail in Resque, just failed to do their job and if this was in production it would've cost us thousands.
So, my own ORM is just a database superclass with em-connection-pool, configuration, openstruct and couple of class and instance methods (insert, update, find, query). And now this thing is really really fast, using lot less of sql connections (delayed job had 300 connections, now we'll need only ~40 connections) and scales really well.
The thing here is, that when I'm not using Rails at all, I don't know why I should use evented Ruby instead of Node.
This is why theres people who say you shouldnt use an orm at all. I have to admit intially i liked the django orm however after a while i found that i prefered or wanted to make the called directly myself. There was just oo much mystery going on behind the scenes that i couldnt find out about.
You can probably use both node and ruby. From what ive seen sites that are using node use it more for the backend to serve ajax then for serving the web pages. Although i have seen few site with very ajax based interfaces that apparently use just node. It really depends on your ui i guess. Good luck with whatever you choose though!
forgive me for digging deeper - but could you narrow down the problem _why_ your architecture better. My gut feel is that you are leveraging the connection pool better - because thats the only thing that can explain DB connection issues. The other thing could be that your em-activerecord connection objects might be too heavy, so you're hitting some kind of ulimit/open-file-descriptors problem (basically OS issues).
This seems like exactly what would hit anybody who is trying to build a worker model for processing data... hell even for sending emails. It would be interesting to see where you are pushing the limits - it could even be something deep like
I tried to dig deeper to em-activerecord, but it really was a better idea to build a very simple orm layer with specific sql queries instead of a more complex solution.
This problem is noted by the developers of em-activerecord and em-synchrony a several months ago already. There seems to be no progress, which might lead to a harder problem or that people are not using these libraries in bigger services.
But my point was, that Node.js seems to be much much more finished product compared to EventMachine for example. With EM you have to live without most of the Rails and for example testing the EM code with unit tests is not so nice job to do...
What I liked the most with EM are the Ruby fibers. The best solution for hiding the callbacks and writing nice and readable code by far. Too bad the fibers are a bit of a ghetto still, like somebody said earlier.
I'm using em in production also and I am seeing quite few bugs too. I wanted to ask you some eventmachine related questions, how can I get ahold of you, I didn't see your info in your profile.
It's a shame that fibered Ruby feels like such a ghetto. I'd really like to see it be a viable concurrency choice, especially for apps that call a lot of third party services.
The problem of the main thread waiting on a response from the database is easily solved without rewriting all your code, which incidentally will have the same issues, since there's no new magic there.
If your software is a giant, ugly hairball because you wrote it in a hurry (I know I did this many times!) don't blame it on the language or framework. What I'm wondering is: Python/Twisted/Django rank equally or similarly on all the points from the list, except for Team Experience, which has to be much higher with the old stack than with a completely new technology and set of tools. This makes me think that there was a "play with shiny things" point after all.