"We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue."

This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varies all over the map. Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had. That is made worse by good APIs where you spend all that time reviewing them only to note they are well written, but each version which could have not so clued in people committing changes might need another review. So you can't just leave it there. And when you find the 'bad' ones, you can send a note to the project (which can respond anywhere from "great, thanks for the review!" to "if you don't like it why not send us a pull request with what you think is a better version.")

What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation.

There was an interesting, if unwitting, simulation of this going on inside Google when I left, where people could check-in changes to the code base that would have huge impacts across the company causing other projects to slow to a halt (in terms of their own goals) while they ported to the new way of doing things. In this future world changes, like the recently hotly debated systemd change, will incur costs while the users of the systems stop to re-implement in the new context, and there isn't anything to prevent them from paying this cost again and again. A particularly Machievellan proprietary source vendor might fund programmers to create disruptive changes to expressly inflict such costs on their non-customers.

I know, too tin hat, but it is what I see coming.

Open/closed is a red herring here. Projects slowing down as they succeed seems to be a universal phenomenon, from startups to civilizations. Specialization leads to capture. I think almost exclusively about how to fix this: http://akkartik.name/about (you've seen and liked this), http://www.ribbonfarm.com/2014/04/09/the-legibility-tradeoff

Disclosure: google employee

In practice, the way to avoid this is to keep the software as simple as possible. Try to adjust to your user's most pressing current needs, not every need they might conceivably have. Killing features and deleting code is as important as launching features and writing code; make sure that your incentive systems reward this. Very often, third-party code gets pulled in to scratch one particular itch; if it's no longer itching, rip the code out. If it is still itching and you've built significant parts of your system around it, you may want to think about replacing the innards with a home-grown system.

It's better to be careful - very careful - about what you add, and to have a story for migration, than to remove features.

This paradox, BTW, could be thought of as the full-employment theorem for entrepreneurs. As long as it is rational for a business to avoid change for fear of having to remove or support it later, then there will exist changes that a company with no customers and no codebase could implement that no incumbent would dare. Some of these are bound to be useful to some segment of the market, and that's why you get continued disruption in technology markets.

Some of the iterations are good, some are just for control, mostly though it is to keep developers locked in and chasing rather than innovating. Software has to change and iterate though, but too slow or too fast can be harmful or the worthiness of iterations varies when platforms change.

On the topic of Node.js and Express though I think it is production solid and it is very fast (faster than php, ruby, python). I love it and I think here Netflix developers are at fault for not testing something a bit outside of the express default usage that is tested thoroughly. Their solution of changing to Restify so quick on a problem probably tells you how they got into this problem even if Restify is indeed better for this purpose. With any new change comes research/testing/possible problems open source or closed.

In the end this is why microframeworks, which both of the node frameworks are, do win out as they are easier to inspect and live on after the hype (monolithic frameworks not so much).

To contrast with what you said, I've worked at Microsoft, which is almost the company that invented NIH and we had the same problem. I think it's because, to paraphrase Alan Perlis, programming nowadays is less about building pyramids than fitting fluctuating myriads of simpler organisms into place.

Disclaimer: I work for a company providing commercial support (scaling) for an open source project (Drupal).

Just because you have the option to read the implementation details of every library and service you are using doesn't mean that you have to. You only have to learn enough about it to decide whether you think it is a good addition to your stack, and to use it to do whatever you are trying to get done. But open source gives you the ability to figure out why you're using it wrong, or how it is broken when that time comes.

If you are saying that open source is not documented well enough because the developers fall back on "check the source," that is a different argument where commercial software may be better, but this is not true for any of the more common open source software I've used.

Making a decision of which software to depend on in your application is something that is always difficult whether you have access to the source code of all of the choices or not. It's a decision that you make with limited information. You only have extensive knowledge of the tools that you already have experience with, so using alternatives is always a risk, but understanding and managing that risk is part of the developer's job.

And with regard to keeping up with changes, you always can remain with previous versions for some time to avoid the slowdown associated with shifting to new APIs.

I think you missed my point though, which was that there isn't any sort of fitness function being applied to open source. In the closed source environment that it 'price' (caveat walled gardens). By paying for the software customers "vote with their wallet" on the things they like/want/use. What is more if two folks are putting out the same capability they are incentivised to compete on a vague sense of value. That creates a 'culling' function which operates independently of the software creators (the only way to sell more is to be "better").

Open source doesn't have that forcing function, there isn't really even a reputation function involved where bad FOSS would give folks a bad reputation that a user could pick up on (yes we probably all know one or more committers who are jerks but that isn't quite the same thing). And there is no barrier to entry so we get multiple variations on the same conceptual solution (I've lost count of the number of CMS's or blogging packages there are out there for example).

Finally there is the various bits of entanglement, you wrote "And with regard to keeping up with changes, you always can remain with previous versions for some time to avoid the slowdown associated with shifting to new APIs." but that has not been my experience. As you're evolving your product interdependencies in the APIs and packages you are using force them to upgrade as well. There is no island of stability in the FOSS world (well maybe OpenBSD servers or something but I have not seen them in the 'stack') Backports only go on so long and suddenly you're no longer getting updates from your source as Ubuntu showed when they dropped updates for 12.x LTS. All those systems still shell shockable or heart bleedable, or something else. Because to upgrade that part, needs to upgrade the next part, and so on and so on until you've upgraded everything. And you pay that price again and again and again and again.

Much of this is just different than the other model which is not necessarily a bad model. But from a systemic point of view it is hard to partition the work and that leads to inefficiencies. A single change in an API implementation which results in dozens of engineering groups re-implementing stuff, versus a migration model that allows people to move over gradually. The promise of open source is "no capital expense" but I worry the down side will become "much higher operational expense". And when the opex cost of open source is > than the capex + opex cost of closed software, then closed software wins. Of the folks who have tried to prevent that equation from shifting in favor of closed source, I only see Redhat as a success story.

There are a number of open-source projects - web.py, Mongo, Angular, Ember - that I am actively avoiding because IMHO their benefits to me don't outweigh risks or previous hassles experienced. Much better to use tried-and-true open source projects like MySQL, Postgres, vanilla JS, and Django that I have had good experiences using. There are many others like Polymer or Docker that I am intrigued by, would love to give a try, but am sitting on the sidelines until other people uncover the most obvious pitfalls.

Economically, open-source projects are just startups where the monetary cost of use is zero. That doesn't mean that the total cost of use is zero, and as you're evaluating potential products, you need to be careful to figure in time expenditure spent debugging as a cost. But you are incredibly naive if you believe that just because you are paying a million dollars for a piece of software, you will spend less time debugging and integrating it than a well-used piece of open-source software. Hell, I once was the college intern writing that million-dollar piece of software. All it takes to sell something is chutzpah.

   People who find that an open-source program creates more hassles than it 
   solves don't use it. Folks who blog about their hassles induce other people 
   not to use it.

Docker, btw, is awesome. You can use it to replace virtualenv (which only handles python libs) and system lib isolation in a much more scriptable, more robustly deployable fashion. So you can take python projects developed at different times that use different revs of packages (and those packages can require different libs) and make deploying them way better. Try it, it's just awesome.

But bad open-source software can live on to upset the productive lives of others forever, as the distribution cost is essentially free. With bad closed-source software, if the producer goes out of business (without a buyer), typically it becomes unavailable.

> people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software

I guess the theory is that if you have a customer/vendor, contractual relationship, then you can task the closed-source vendor to describe, fix, or alter their API.

But my (albeit limited) experience is that the customer is not always right; sometimes the vendor just says "that's the way it is" and you're stuck either living with it or porting to a different company/technology. Conversely, it's often possible to hire a vendor or consultant to provide expertise and support for some aspect of an open-source stack. And because it's open-source, if you don't like your consultant you can go find another one without having to change your technology.

In short, I think the speed of innovation thing might have more to do with internal teams trying to manage too much diversity of technology, than with some inherent shortcoming of open source.

I think...your experiences at Google have altered your world view to the point where you don't see how thing are happening at other organizations. Google's monolithic codebase where everything builds against everything else may work(?) for them, but the alternative is disciplined module management.

You don't have to ever bump a version of working code if it's doing its job. Good open source projects should absolutely (publicly) test against performance regressions. New versions should be minor/incremental and source compatible.

I've never worked on a codebase the scale of Googles', but I fail to see how you can't mitigate your concerns, nor do I see commercial software the solution.

Having worked at Google in the interim, and having a number of acquaintances at Microsoft (which uses the multiple-repository approach) I can see the pros and cons of both. The biggest benefit of the single codebase isn't technical, it's cultural. When you have a number of interdependent modules, then every change request and new feature has to go through that module's owner. If it's not their top priority (and it won't be), then getting your work done suddenly has a hard dependency on a team who is...generally pretty unresponsive, at least from your POV. The result is a lot of finger-pointing and political infighting, where every division thinks that every other division is a bunch of bozos.

The nice thing about Google's system (which, IIUIC, was Sun's as well, and is also Facebook's) is that when you have a hard dependency on another team and their priority list doesn't line up with yours, you can say "Well, can I make the change myself and you review it?" and the answer is usually yes. That means that people's default worldview is to assume busyness, not malice or stupidity, which makes the company as a whole function much better together. Yes, it creates a huge mess that someone will eventually have to clean up. But now you have a lot of options for how to clean it up, not a single point of failure: you can have the feature implementor do it, or the code owner, or it may get replaced entirely if the system is rewritten, or the feature may be unlaunched and no longer necessary, or someone may write an automated Clang or Refaster tool to fix a bunch of instances at once.

I'd compare it a lot to democratic capitalism: it's the worst system that exists, except for all the rest. When I was at Google, we all complained about how everybody else checked in changes that added complexity to our code. But if we couldn't do that, we'd all be complaining about how everybody else prevented us from getting our work done. I know which problem I'd rather have.

Is that true? Again, I've never done development on that scale, so I could totally be wrong, but I'm seeing this a problem with how changes are handled and releases are published.

I do work on some Open Source projects that are used by some big companies, the project has very good reliability because of CI, specification testing, perf testing for regression to the point that merging in minor bug fixes and pushing a minor release is pretty trivial.

Let's say only some of the teams practice good module management, ok fine, you fork the repo, fix the changes, publish your fork as your new dependency and move along on your way.

Does this take discipline? Absolutely, but it severely reduces complexity in the code. It mandates a certain level of pureness by forbidding certain dependencies in the vast majority of the codebase. The upsides are tremendous when compared to the slight inconvenience of fixing the occasional bug in someone else's module.

Open source works because the projects that become popular can stake out a particular territory and say "This is our responsibility, this is your responsibility, these are the hooks you can tap into, and if you don't like it, fork or use a different project." Flip the perspective and imagine yourself as the user of an open-source project. There's always a huge amount of work you have to do to customize exactly what the framework gives you to what the product needs. This is okay, because that's the bargain you know you're signing up for when you use the open-source framework, and it is after all what you're getting paid to do. If you don't believe it saves you more effort than it costs you, then don't use it. (In fact, there's a huge graveyard of projects on GitHub where the author tosses their personal way of doing things over the wall and then wonders why nobody uses it or contributes to it. This is why: the benefits don't outweigh the costs.)

Now imagine that you're at a big corporation. There is a lot of work that somebody has to do to make the product work right. All of you are getting paid by the same entity, the company. Who does the work? The team who wants it? The team whose codebase it's in?

Open source projects have the luxury of defined interfaces; they get to set the boundaries of what their code does themselves, and can turn away customers who need more than they provide. Internal teams do not: if they say no to a critical feature (regardless of how different it is from their existing interface), the product doesn't ship, and then an executive gets mad. So often adding features is non-negotiable, the features are significant enough that there's a real risk of impacting the stability or performance of the core product - and yet it's still the right thing for the business to do. The big problem is that each individual team wants clean code and well-defined interfaces, yet the benefit accrues to the company as a whole...hence, resentment between teams about who's got to spend significant work mucking up their pristine design.

>All of these incompatible forks need to down-integrate the change and work around the little tweaks they've added. You have a mess.

Is that really better than having a mess without any demarcation of the software? Either way people are going to make a change. Your way involves no disincentive to making quick fix that just your team needs. Modularized codebases means that before submitting that PR or forking, you're forced to consider how this will affect everyone who uses the library.

I understand your point about modularizing codebases causing more friction between teams. That's quite insightful and I haven't considered that. However, I think that gets down to a cultural issue of what do you value more, clean code that is easier to comprehend, or fostering geniality between teams.

There's no getting around this, you or someone that you trust (not necessarily at your company) needs to read and review both the API, and implementation details for open source software that you use. Open source software isn't a hardware store you can dip into to get the latest parts that you need for your project. Adding a dependency makes your code depend on other peoples code. Just like internal code should go through a code review, so should external dependencies. This also explains why for a lot of companies, it's easier to write their own thing rather than need to keep on top of other people's changes.

I happened to write about this yesterday which explains my thoughts further http://danielcompton.net/2014/11/19/dependencies.

Then, you've got to be intimately familiar with every piece. There's no excuse when the source is readily available. I give these guys credit though they seem to take responsibility instead of just saying "express sucks". Some of their design choices seemed a little shaky.

Its actually quite clear - most routes are defined by a regex rather than a string, so there is no built-in structure (if there's a way at all) to do O(1) lookups in the routing table. A router that only allowed string route definitions would be faster but far less useful.

I can't explain away the recursion, though. That seems wholly unnecessary.

Edit: Actually, I figured that out, too. You can put middleware in a router so it only runs on certain URL patterns. The only difference between a normal route handler and a middleware function is that a middleware function uses the third argument (an optional callback) and calls it when done to allow the route matcher to continue through the routes array. This can be asynchronous (thus the callback), so the router has to recurse through the routes array instead of looping.

You can use Ragel[1] to build your automaton.

[1] http://www.colm.net/open-source/ragel/

Nice idea.

Edit: I did consider rewriting the runtime as a forth style interpreter as well but the time never appeared.

The function it performed to rebuild the DFA was to create an NFA for the new pattern stream (it worked on char *) and then walk both the existing master NFA and the merge them. Then it was converted from NFA to DFA via dragon book copy and paste algorithm (my discrete math isn't great). Both NFA and DFA were represented as a bunch of C structs tied together by pointers. Then a table generator walked the new DFA and generated a new state table. It was very naive but covered the common case where large parts of the byte stream to be matched were similar.

I've got MacVim up and writing it again. May post it as a Show HN if I get the time to finish it.

Your worst case is O(nd + O(constructing the new route's DFA)), where "n" is the number of nodes in the previous DFA and "d" is the number of nodes in the DFA of the new route.

Note that constructing r's DFA can be done beforehand, and hence the actual time the router will be offline is only O(nd).

(Assuming a simple 2 DFA -> 1 NFA -> DFA merge. Your worst case is that every possible pair from the two, and singleton of the two is present in the final DFA, i.e. ({node from old, node from new} -> nd + {node from old} -> n + {node from new} -> d) -> O(nd + n + d) -> O(n*d). Note that this isn't the classic O(2^n) behavior, because we know that, due to the old DFA and new route DFA being DFAs, only one node in n and one node in d can be present at a time in the final stateset.

A better bet may be to use the classic "lazily-constructed DFA" approach. In other words, keep it as an NFA while only converting (sets of) nodes to DFA nodes when necessary. If you are worried about memory, you can even do this only for "hot" sections of the NFA, or only for those sections of the NFA that have the most performance gain for the number of nodes added.

Sure, you could use all routes that match, but is there a way to specify the order for all handlers, and whether or not you should continue after one handler is done?

Sure you can. When you build your NFA, each accepting state is tagged with the route to which it corresponds. Converting to a DFA will merge states. Any accepting state in the DFA tagged with more than one route indicates a possible ambiguity. How you handle that ambiguity is up to you.

One strategy is to just fail at route-building time. Another is to order the routes by priority (which priorities maybe coming from order in the source code) and eliminate from each accepting state all routes except the one with the highest priority.

The latter approach gives you the "defined order" semantics you want.

Unless you go the NFA route, but I'm pretty sure that costs non-constant space.

So, in terms of the GP's post, yes, this is an O(1) (with respect to number of routes) solution.

I suspect this is a slight overestimate, but the point stands - non-trivial apps could quickly build very large state graphs.

Other than that, if you are concerned about automaton size you can use a non-deterministc automaton instead of a deterministic one. You spend a bit more time (at each iteration you have to update a list of "active" states, while in the DFA case there is always a single active state) but on the other hand the automaton itself is much more compact.

That's not very large, since you can store a DFA very compactly in a table (C array with structs). The table stores the transitions, which only have the character and the target state. Then you either keep an array with state offsets in the transition table, or you modify transition tables such that each transition points at the first transition of the target state.

If I have the routes /foo/bar and /foo/bar/(\d+) I can generate the regexp ((^\/foo\/bar$)|(^\/foo\/bar\/\d+$))

I'm not at all surprised, the quality of software in node is pretty low, I've seen numerous issues in node libs being just as boneheaded. I swear, the fact that the express devs overlooked a key optimization is crazy. Rails, by way of example, uses the Journey engine to solve this problem (https://github.com/rails/journey)

And how would you know which one got matched? The regex match isn't going to tell you that. Also, it needs to recognize if multiple were matched, which is definitely not going to be done by the built-in regex matcher.

It's certainly possible, but pretending it's trivial isn't helping, either.

  >>> import re
  >>> route = re.compile(r'(?P<fb>^/foo/bar$)|(?P<fbd>^/foo/bar/\d+$)')
  >>> route.match('/foo/bar').groupdict()
  {'fb': '/foo/bar', 'fbd': None}
  >>> route.match('/foo/bar/1').groupdict()
  {'fb': None, 'fbd': '/foo/bar/1'}

Nice to know that what I made is considered the best solution algorithmically.

    > /(^\/foo\/bar$)|(^\/foo\/bar\/\d+$)/.exec('/foo/bar')
    [ '/foo/bar',
      '/foo/bar',
      undefined,
      index: 0,
      input: '/foo/bar' ]

    > /(^\/foo\/bar$)|(^\/foo\/bar\/\d+$)/.exec('/foo/bar/3')
    [ '/foo/bar/3',
      undefined,
      '/foo/bar/3',
      index: 0,
      input: '/foo/bar/3' ]

You could write a simple router based on this in < 50 lines of JS. I'd do it now, but I have work to do.

Read my top-level post again - multiple routes can be called on the same request, so express has to be able to find all of the matches, not just the first one. This is a mistake in the original article, as the author doesn't appear to understand the power of express routers.

> Second, you know which one matched based on the index of the capture groups, which is deterministic.

Deterministic, yes. But now you're also writing your own regex parser, because you have to know how many capture groups each sub-regex contains in order to figure out the indexing.

    "/foo/bar/3".match(/((^\/foo\/bar\/.*)|(^\/foo\/bar\/(\d+)$)|(^\/baz))/) => 

Don't forget non-matching groups! Which brings up my point yet again - everyone is pretending this is a dead-simple problem to solve, but all of the proposed solutions are buggy. Not really an option for such a widely-used library.

I'm not claiming it's impossible. I'm claiming it's more difficult than anyone is willing to admit.

That being said, most frameworks don't have multiple matching routes, and I can't imagine that many people miss them. I've never used a framework that supported that, and I can't say I've ever felt I've missed out. Middleware is the right approach to that problem generally.

The point being, now you're describing a completely different, less powerful framework. Which, yes, of course it's faster. You'll notice that's what Netflix ended up doing - they moved to `restify` from `express`.

You might as well just only allow routes to be strings and then you have the constant-time map implementation that the article describes as the expected implementation.

Iterating through the array looking for the first non-null value, then looking it up in a JS object O(1) matching capture group positions to routes is much faster than having to dispatch a bunch of functions. Additionally, it can be JITed to much faster code I would wager.

So, expressJS = N regexp matches, while this method = 1 regexp match + N null checks in the resulting array + 1 hashmap lookup in an object to return the proper route. The second solution is going to be much faster, even for large routing tables, esp. given how optimizable that array lookup code is.

As far as multiple matches, after you found a match, you would then test against a new regex that is everything to the right of the previous regex's captured group.

((a)|(b)|(c)|(d)) -> if b matches, you then test against ((c)|(d)), and so on.

http://jsperf.com/regex-list-vs-full

Most of the time in the single-regex case is likely spent on allocating the matches array (which for a router with 300 routes would have over 300 undefined elements on every execution)

V8 is not quite the typical dynamic language runtime. Code that would be dead slow in others is optimized really well by the amazing JIT.

It is interesting how much allocation changes things though, very cool that you made this.

Or you use a better regexp library than the default JS one and use named captures.

That way, if you have any route definitions that match early on, especially if they allow execution to continue ( by calling next() ), they will run before any following code does.

This allows your code to be very expressive and intuitive when reading it, as things have a defined order.

I imagine the recursion is also done so that you can use a route object anywhere in that ordered stack and it will apply all its own handlers, in the order they were defined in the code.

Not looking to start any wars, but I was under the impression that if you know what you're doing* node.js is pretty awesome.

This particular bug had to do with a misunderstanding regarding the express API.

* for the most part: understand async and closures/memory leaks.

- "Here we see our latencies drop down to 1 ms and remain there after we deployed our fix." <- Performance (from TFA)

- They can hire from a vastly larger pool of developers. Since they've got both budget and brand recognition, they should have no problems hiring high end JS guys & gals.

> Java is both considerably more speedy for this sort of workload

They're quoting 1ms latencies, that's pretty speedy. Maybe Java could do <1ms and/or support slightly more concurrent requests. How much difference does it actually make when you're already responding after 1 millisecond?

> there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers

Oh, I would have thought there are far more JS* folks out there than Java.

> Budget and brand recognition are not necessarily in favor of node.js over other stacks. That simply helps with getting the better engineers in general.

That's what I meant - them being in favour of Netflix as an employer - they aren't "Bob's Digital Agency" so they can have their pick of the litter of node devs.

* I'm not sure if the "server" distinction is meaningful here - if you know JS, you know node.js - simply a matter of familiarizing yourself with node-specific APIs.

99% of cases, your bottleneck will be I/O.

In the 1% of cases where your webserver's bottleneck is the CPU, you have much bigger problems than using a hipster ;) language: you're doing it wrong (tm) on an architectural level:

(a) Your processor-intensive/long-running tasks need to be in seperate worker processes and

(b) you need more webserver instances.

I usually try to avoid absolute statements, but I think this may be accurate:

The only CPU-intensive thing your web server code should ever do, is hash passwords.

Can I get a rebuttal to my points along with the downvotes, though?

Is someone disputing that CPU intensive tasks should be moved off the webserver?

Have you personally had experiences where your honest-to-$deity webserver bottleneck was the CPU? I'd be incredibly surprised. In most cases I'd guess you're underprovisioned and/or badly architected.

* This is where we come to somberly discuss interesting things.

1. Your networking pipe into the server is slow.

2. You don't have a reverse proxy in front of the webserver to buffer the slow connections.

3. You don't make decisions about what information to show the user in the webserver; your UI isn't tailored for the particular user whose request you're servicing.

4. You haven't made productivity trade-offs to move work from the developer to the computer.

Google's webservers (for Search) are CPU-bound, and I can guarantee that they are neither underprovisioned nor badly architected, and CPU-intensive tasks are moved off the webserver (the total amount of CPUs for webservers is miniscule compared to the number of CPUs used for the indexing & serving system). But Google has the fastest fiber networks in the world, a massive load-balancing infrastructure, its philosophy is to serve each user only the HTML/JS that they need (to cut latency), it relies heavily on A/B tests & experimentation, and it invests significantly in developer-productivity tools so that all the bookkeeping for the last two points are done by computers rather than humans.

It's not a huge step for Netflix that already has an API approach that is very UI centric as outlined here http://techblog.netflix.com/2013/01/optimizing-netflix-api.h.... To me personally this doesn't seem like a healthy step forward but I hope someone can provide some reasons why you'd want to move to Node.js from, in the case of Netflix, Java.

My preference for node is based primarily on its sane concurrency model. I basically use it as a handy scripting language for doing rapid prototyping of libuv programs. My on-paper plan is to fall back to libuv and C if I find myself really stuck for CPU, but this is something that's never actually happened to me. (I have had to abandon or modify libraries with performance pathologies, though)

There are other libraries with in other languages the same sane model but they tend to be ecosystems within the language community that aren't compatible with the rest of the language. Doing concurrency both correctly and performantly in Java, in particular, is a pain in the ass.

  n = 0;
  avg = 0;

  function twothings() {
    thing1(function(ret) { ++n; avg += (ret - avg)/n; });
    thing2(function(ret) { ++n; avg += (ret - avg)/n; });
  }

In Java, you have access to real threads and can actually run two pieces of code concurrently. This introduces a lot of potential issues, but also allows you to take advantage of multiple cores.

If all you want is non-blocking IO (what node gives you), there are a handful of great libraries that provide that on the JVM.

Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much.

I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opinion Node has way too many issues to run in money-making scenarios.

You can say that about a lot of languages other than node. How many billion dollar companies have their software written in PHP - a language that many people would agree has far more glaring issues than node.

My point, is that I believe your comments miss the larger picture. There is far more involved with deciding which language to build a product with than "which language is best."

I don't disagree with you about Go being a great language - but for most companies, it is not even remotely practical to use. Hiring talented Go developers is hard because there are so few. Their current employees may or may not know Go, what do you do about that? Etc.

If anything the comment on hiring highlights the use of Node being problematic. It's difficult to write robust systems in Node even as a seasoned Node developer.

Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regularly publish "how it's done" documents that see wide applause despite often completely contradicting their prior missives.

The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

Don't get me wrong, I am not saying that these optimizations are bad. In fact, these engineers measure things properly and then rewrite. It's just that there is no business case for all this time spent. It's like writing web servers in C. Sure, it's possibly the faster than everything else but seriously? Who maintains all this.

But you made me think what I said. I would say this sort of gets into the programmer / engineer debate. Netflix has good progammers like most companies. Good engineers (in my books) are very hard to find. These people are good at being at peace with tradeoffs and understand the requirements of a business at a much deeper level than programmers. Programmers just want to have fun with computers (and there is nothing wrong about that).

Nice, contained way to show data like this.

I investigated them a year or two or so ago, because I found that AngularJS could work quite nicely within an .svg document, which opened up some exciting possibilities. My recollection is that at the time, there were some critical cross-browser problems with font rendering that made the topic very kludgy and complicated. I except that matters have improved since, but I do not know to what extent.

Also SVG GUI tools haven't been great. Inkscape is strongly print focussed, or at least was the last time I used it. There's no reason to set a page size on a web svg...

Someone needs to build an SVG equivalent of Flash Pro.

> ...also saw that the process’s heap size stayed fairly constant at around 1.2 Gb.

This is because 1.2 GB is the max allowed heap size in v8. Increasing beyond this value has no effect.

> ...It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers.

It it is non-trivial (not possible?) to do this in O(1) for routes that use matching / wildcards, etc. This optimization would only be possible for simple routes.

I'd be impressed if they did it consistently in O(1) for static routes. I think they were looking for O(log(number of different routes)) instead of O(n).

For small values of n ;)

A tree of maps would be consistently log(n), like any map. Even a hashtable would hash to buckets eventually, and are log(n).

Honestly, I have no idea why people aren't taught Cuckoo-maps by default. It's simple, has true O(1) worst-case lookup, and has easy deletion. About the only problem is trying to prove insertion time.

Possibly worth mentioning, but there's really nothing stopping people from adding dtrace support to Express, it could easily be done with middleware. Switching frameworks seems a little heavy-handed for something that could have been a 20 minute npm module.

"This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array"

refresh our route handlers from an external source

This is not something that should be done in live process. If you are updating the state of the node, you should be creating a new node and killing the old one.

Aside from hitting a somewhat obvious behavior for messing with the state of express in running process, once you have introduced the idea of programmatically putting state into your running node you have seriously impeded the abiltity to create a stateless fault tolerant distributed system.

1ms / entry? What is it doing that it's spending 3 million cycles on a single path check?

I could be wrong but it seems like the design of the route lookup mechanism (the global array) was actually a bit of a red herring, the real issue was the ability to attach multiple instances of the same handler to the same route.

Something was adding the same Express.js provided static route handler 10 times an hour. Further benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time.

OSes cache directory entries for a reason.

I mean, even Python manages 40,000 checks / second:

  >>> timeit.timeit("os.path.exists(data)", setup="import os; import random; import string; data = os.path.join(r'C:\Windows\System32', ''.join(random.choice(string.ascii_letters) for _ in range(10)))", number=40000)
  0.9998181355403517

Hopefully that lowers your concern level.

Javascript, executed in the console of a recent Chrome:

    > 'ab'.match(/a|ab/)  
    ["a"]

    > egrep 'a|ab' <<< ab  
    ab

I was under the impression that all major regex engines used NFAs converted to DFAs lazily, with fallbacks to a slower engine for features that cannot (or cannot practically) be implemented using an NFA (unbounded backtracking, that sort of thing.)

What is the advantage to doing this?

This has me scratching my head. The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow). And I can't see, visually, which calls are repeated. They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).

At least, I _can_ imagine how you could characterize this problem without novel eye-candy. Use histograms. Count repeated calls to each method and sort descending. Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense. It sounds more like 'instrumentation', decorating the function calls to produce additional debugging information. Either way, once you have that, there are various common ways to isolate performance bottlenecks. Few of which probably require visual graphs.

There's also various lesser inefficiencies in the flame graphs: is it useful (non-obvious) that every call is a child of `node`, `node::Start`, `uv_run`, etc.? Vertical real-estate might be put to better use with a log-scale? Etcetera, etc.

Flame Graphs provide SVGs by default. You should be able to zoom if your broser supports it. The current version also supports "zooming" in to any frame in the stack, resetting that frame as the base of the display. Also WRT the base frames of 'node' et al its because Flame Graphs are a general use tool for stack visualization, it might be 'main' for a c program or the scheduler looking at a system.

> They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).

Colors by default have no meaning and the palette is configurable. The current lib can also assign colors by instruction count/ipc and width by call count, if you have access to that.

> Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense.

It is sampling. Flame graphs re typically used with something like perf/dtrace/oprofile which dumps stacks at a couple hundred to thousand hertz. Actual call tracing is (typically) not feasible for large/prod stacks.

  > our misuse of the Express.js API was the 
  > ultimate root cause of our performance issue

> What did we learn from this harrowing experience? First, we need to fully understand our dependencies before putting them into production.

Is that the lesson to learn? That scares me, because a) it's impossible, and b) it lengthens the feedback loop, decreasing systemic ability to learn.

The lesson I'd learn from that would be something like "Roll new code out gradually and heavily monitor changes in the performance envelope."

Basically, I think the approach of trying to reduce mean time between failure is self-limiting, because failure is how you learn. I think the right way forward for software is to focus on reducing incident impact and mean time to recovery.

1. Don't get suckered by interfaces, share code. If you create code for others to share ("libraries"), stop trying to hide its workings.

2. You don't have to learn how everything works before you do anything. But you should expect to learn about internals proportional to the time you spend on a subsystem. Current software is too "lumpy" -- it requires days or months of effort before yielding large rewards. The first hour of investigation should yield an hour's reward.

3. "Production" is not a real construct. There will always be things that break so gradually that you won't notice until they've gone through all your processes. Give up on up-front prevention, focus instead on practicing online forensics. And that starts with building up experience on your dependencies.

More elaboration: http://akkartik.name/post/libraries2

My attempt at a solution: http://akkartik.name/about

My motto: reward curiosity.

So in this case, guarantee you have a strong means of evaluating performance and maybe even include it by default just to be sure.

From the API docs:

> Multiple callbacks may be given; all are treated equally, and behave just like middleware. The only exception is that these callbacks may invoke next('route') to bypass the remaining route callback(s). This mechanism can be used to perform pre-conditions on a route, then pass control to subsequent routes if there's no reason to proceed with the current route.

"First, we need to fully understand our dependencies before putting them into production."

This is why developers without significant experience should not be making decisions about the tech stack.

Source: myself - https://github.com/strongloop/express/pull/2237#issuecomment...

I built one for Java: https://github.com/augustl/path-travel-agent

The routes are stored as "nodes". There's a root node. It has a hash map of child nodes, by name. It also has a list of "parameterized" nodes. When a node gets a path segment, it will first look in its hash map. If nothing is there, it'll call the parameterized nodes in sequence. Typically there's just one parameterized node.

For the following paths:

  /projects
  /projects/new
  /projects/special
  /projects/:project-id

The node for "projects" will have to items in its hash map, "new" and "special". It will have a single item in it's parameterized node, for :project-id.

I updated the README just now with a slightly more detailed explanation :)

* Netflix had a bug in their code.

* But Express.js should throw an error when multiple route handlers are given identical paths.

* Also, Express.js should use a different data structure to store route handlers. EDIT: HN commentors disagree.

* node.js CPU Flame Graphs (http://www.brendangregg.com/blog/2014-09-17/node-flame-graph...) are awesome!

Now that I look at it, there's a TOCTOU bug on the fstat/open callback, too: https://github.com/tj/send/blob/master/index.js#L570-L605

This should be doing open-then-fstat, not stat-then-open.

#moderationfail

I don't understand the need of refreshing route handlers. Could someone explain they needed to do this, and also why from an external source?

https://www.exratione.com/2013/03/nodejs-abusing-express-3-t...

I guess the Netflix situation is one of those that doesn't occur in most common usage; certainly dynamically updating the routes in live processes versus just redeploying the process containers hadn't occurred to me as a way to go.

In some ways, these engineers are not that different from academic researchers, in that they are devising experiments, verifying techniques, all in the pursuit of the question: why?

> Something was adding the same Express.js provided static route handler 10 times an hour.

Why didn't it increase the heap size? Maybe it was too small to be noticeable?

I couldn't agree with this more. Understanding where time is being spent and where pools etc. are being consumed is critical in these sorts of exercises.

https://github.com/strongloop/express/issues/2412

For the audience, this was originally titled: "Node.JS in Flames"

At least it will help people get their buzzword bingo cards filled up sooner.

The word you want is "lessons".

Jeez, not only is your comment offtopic and pedantic, it's also wrong.

I believe the word you want is "incorrect."

/sarcasm

No, at best it's a deliberately wonky usage for comic effect [1], like "Internets". It's quite possible the author is being gamesome. [2]

[1] https://simple.wikipedia.org/wiki/Borat:_Cultural_Learnings_...

[2] http://youtu.be/Wc1pxO_-5S8?t=6m43s