The problem is provisioning and managing networks in a large scale data center deployment. And while networks are reasonably simple in the small (Hey plug in the cable and the light goes green, done!) in a larger (> 800 or so up to thousands and thousands) network lots of little things need to be dealt with.
When Google first started building their own switches I thought it was the stupidest idea ever. But when you look at it objectively, having the connectivity is essential and putting the 'smarts' in a place where it can be easily updated/modified/copied etc it vital. The network is just wire and switch companies put a lot of 'value add' in their switches which basically puts their second tier network programmers writing code you can't code review but can kill your network at any time. At Blekko the few outages we've had were all caused by a switch software programmer. How scary is that?
So if your wondering, what this means for the rest of us, it means there is a market for a rock-solid-dumb-as-a-post set of switches that do absolutely nothing to the traffic except forward it. They restart in milliseconds, not seconds. They achieve lowest cost per port because they have very little firmware, and the firmware they do have can, for all practical purposes, be proven correct. The money is spent on really reliable transceivers and low noise cross connects and just enough SNMP work to give the upper levels of the system a clue as to whether they are overloaded or not. They might do link aggregation. We'll see if they appear or not.
I have an idea for an open platform that just has PCIe slots and fpgas. 3rd parties can write the firmware that runs on the FPGA layer. Add in banks of DIMM slots and you can run memcache or redis directly on the minimal box. No operating system. Since network cards boot PXE, the firmware on the device could get loaded over the network. Routing tables could get compiled from a HLL -> Verilog -> FPGA.
That is great for research purposes. This platform would use off the shelf PCIe NICs and ram. I think of it as the PostgreSQL of networking gear, cheap and all purpose. It won't have speed or density but it would have robustness and flexibility. Services that currently run on general purpose servers could get targeted towards open network hardware platforms. I don't have any proof, but I don't think there would be shortage of people doing interesting things like making hardware in memory key value stores.
FPGAs would be cost competitive in a system like this. Routing multiple 100MB/s streams of shouldn't be a problem.
> I don't have any proof, but I don't think there would be shortage of people doing interesting things like making hardware in memory key value stores.
what you are describing are also known as CAM tables. for some preliminary introduction check this out : http://www.pagiamtzis.com/cam/camintro/
How would you compete on port density with PCIe? 72 port 10GbE for 1U (http://gnodal.com/Products/GS-Series/GS7200/) is already available. Once the "pure play nothing but a switch" manufacturers come out this price will be rock bottom.
You are correct, this couldn't initially. What the platform I outlined has is flexibility and cost. But mostly flexibility. Probably the high end of port density would be ~40 (using dual port nics) and this would be for 2u. But you could start to run stuff on the switch that his historically run with ha proxy, varnish, nginx, etc. If there are sata ports, one could code themselves a backblaze pod that doesn't suck and include grep directly on the stata port.
Arista came out last year with something vaguely(although lacking the modularity of cards)similar to what you've got in mind. I think it was originally targeted for high-frequency trading apps where proximity of the colocation such as 350 Cermak in Chicago makes a huge difference in latency. I never heard much more about it though.
There are already builders out of China that you can approach for custom solutions. I also wouldn't expect Marvell or Intel(especially with their acquisition of Fulcrum Microsystems last year) to sit by idly.
Another interesting thing we'll start to see is the use of concurrent programming languages by the established vendors to stay heavily involved and dare I say relevant? Juniper announced a few months back they were going to start using Scala and Akka from Typesafe (http://typesafe.com/company/news/23506). It is not hard to guess this is for their SDN and OpenFlow plans.
When Google first started building their own switches I thought it was the stupidest idea ever. But when you look at it objectively, having the connectivity is essential and putting the 'smarts' in a place where it can be easily updated/modified/copied etc it vital. The network is just wire and switch companies put a lot of 'value add' in their switches which basically puts their second tier network programmers writing code you can't code review but can kill your network at any time. At Blekko the few outages we've had were all caused by a switch software programmer. How scary is that?
So if your wondering, what this means for the rest of us, it means there is a market for a rock-solid-dumb-as-a-post set of switches that do absolutely nothing to the traffic except forward it. They restart in milliseconds, not seconds. They achieve lowest cost per port because they have very little firmware, and the firmware they do have can, for all practical purposes, be proven correct. The money is spent on really reliable transceivers and low noise cross connects and just enough SNMP work to give the upper levels of the system a clue as to whether they are overloaded or not. They might do link aggregation. We'll see if they appear or not.