In general for small hardware I go with two options.
Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current commercial hardware in this space which is perfect. That should change shortly.
Much beefier systems with "real" amounts of RAM and flash. I generally skip the Broadcom/RPi/etc. and go straight to an Atom or i3, but that's just for convenience, since I only need a small number of them. There's probably something more reasonable to do in the embedded space, but I'm indifferent on qty 1 devices between $500 and $200, especially if I already have most of the components for the $500 thing lying around, and it saves me an hour or two getting it working.
There are a lot of things missing from widely available commercial hardware in this space right now.
I've been looking at Intel's DPDK recently [1], and it looks really interesting for really fast packet processing in the 'beefier' kind of systems you describe.
It would be a bit higher cost, but could mean you could make a router like the EdgeRouter but with good performance without the proprietary accelerator (I believe that Intel's drivers are open source).
1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/
> 1. Data Plane Development Kit, which lets you skip the kernel IP stack (which takes thousands of CPU cycles to process) and do packet processing in userland taking just tens to hundreds of cycles per packet. http://dpdk.org/
I wish OS developers saw this as a problem. There is no reason kernel stacks should be so slow for tasks where all processing is done in the kernel. (For packets destined to userspace, you've got the syscall overhead to deal with.)
I recently tested the Linux network stack's PPS performance with an Intel X520 10GbE NIC. I used Debian testing, with the 3.12 kernel. My destination machine was an i7-3930K at stock speed. I wrote a simple kernel module adding an NF_IP_PRE_ROUTING hook returning NF_DROP with no processing, which would be the simplest possible code path. For a packet generator, I used another older machine with another X520, using the "pfsend" tool included with PF_RING, and the card in PF_RING DNA mode. That was easily able to saturate the link at line rate (14.8M PPS).
The result: the kernel was only able to sustain about 2.8M PPS.
I then loaded the DNA driver on the destination machine, used the included "pfcount" tool, and no packet drops - it was receiving the full 14M PPS.
I tested DPDK recently and had similar results.
I also modified the Linux ixgbe driver ixgbe_clean_rx_irq() function, and added a step in between the "fetch packets from RX ring" and "put packet in SKB and send to network stack" functions. Even when I added a bunch of useless comparisons for each packet, I was able to get ~12-13M PPS. I could get line rate by just dropping and not doing any processing.
Definitely. I am not sure what metrics Solarflare beat others (presumably including Intel) on, besides software support (OpenOnload implements the BSD sockets API and doesn't require software changes, while DPDK and others implement their own). In my testing, the X520s have been capable of both sending and receiving at line-rate.
It's possible the Solarflare NICs could be better at some metric like latency, by a matter of microseconds (only speculating here), but I can't see how they'd consider that worth the extra money for HTTP servers.
I wonder how low the price of an Atom C2358 based system with 4x Intel Gig-E and either mini-PCI or some other solution for wireless would be. That would probably be what I'd build a "high end" home router, or low-end business router, on -- you'd still be able to hit PoE 802.af 2009 (25W).
DPDK supports Atom. This Atom has both virtualization and AES-NI, so you could do some interesting security things (the fireeye-style "run malware in a VM", and linerate crypto).
~$120 in parts means a retail price around $500. That doesn't seem at all unreasonable to me, but there's always the "sell it for $250 and make $5-10/mo MRC" option. I think Cisco/Linksys tried that but it was pretty weak.
Someone who is less bad at surface mount soldering than I am can take a 702/703, pop the 2MB flash and 8-16M RAM chips off, and replace with 16/64. There are people in China who do this commercially, and some eBay stores.
I've bricked (well, melted) 2 of them so far doing it myself.
It sounds pretty hard to do using nothing but a soldering iron. The nice thing about the C2 is that you have all the I/O pins nicely exposed and there's a fairly helpful community built around them. It's not necessarily impressive in terms of documentation, but it's certainly better than something that's pretty much reverse-engineered.
FWIW, staples.com is selling the (discontinued) WD My Net N750 for US$30. It's an AR9344 box w/ 16 MB of flash, 128 MB of RAM, a 5 port GigE switch, 2 USB 2.0 ports, and a 0.1" pitch TTL serial header already installed on the PCB. It's supported by OpenWRT trunk, and most of the bits to support it are in FreeBSD, so that might be a "just works" option one day.
You can get some performance out of it if you're using Ubiquiti's kernel, because it ships with a series of highly specialized proprietary modules that offload packet handling to hardware accelerated modules, for their custom proprietary network stack. But their kernel is old and shitty, and if you want to do anything interesting with the ERL, you have to build your own kernel.
So for any mildly interesting usage, it fails considerably hard.
And even if you do opt to use their closed-source binary blob kernel modules and software stack, as soon as you try to do any sort of advanced networking, you run up against the limits of their acceleration modules, and the OS is forced to fall back to standard non-accelerated mode, and performance drops. Try using PPP, any interesting iptables, MPTCP, OpenVPN, interesting routing tables, or anything cool: performance drops.
I would be very happy to see some small networking hardware with multiple independent Ethernet controllers pop up on the market for a decent price; every other month or so my disappointment with the ERL moves me to re-research this. Unfortunately, the EdgeRouter Lite is not the hardware anybody wants for interesting low-cost networking.
It's a $100 box. I've been happy with mine as a replacement for an AirPort Extreme. It handles my 30/5 broadband while doing QoS for my VoIP at no more than 10% CPU. Anyway, what would you recommend instead at that price point?
I've cheered Ubiquiti's wireless hardware in the past on here [1], but I have to agree with you. Stay away from their wired stuff. We've had problems with the EdgeRouter and the ToughSwitches.
There's a reason for the old kernel - basically the company that owned the Vyatta project really started neglecting the free version by the end and it hasn't seen much proper maintenance in ages.
I'm starting to get invloved in the fork that was recently taken, called VyOS [1]. Updating the base to a newer version of Debian is a high priority, but there's a fair bit of work in it.
Since Vyatta Core (the community version) is essentially dead, and there is at least one Ubiquiti guy involved in the fork, we hope that VyOS will eventually be the base for EdgeOS.
And as for the hardware acceleration - I wish the drivers were open source but even as it is, it's really a necessary evil to get very good performance on low cost hardware.
I want a secure (but end-user-openable) boot loader, good crypto performance (to do ~linerate AES, and some PKC operations, for things like opportunistic ipsec eventually), and flexible interfaces (ideally, N x Intel GigE and 2 x dual-band wifi radio chipsets).
Real RED/BLACK would be a huge feature, too, and potentially hardware mode switch, or some other technology to work with the trusted boot loader in a way that malware can't hurt. Watchdog. PoEable for the higher end, and in the low end products, absolutely 5v <2.1A (i.e. Apple iPad charger), and ideally <900mA (USB3) or <500mA (USB2).
I also do not like Vyatta's weirdness. It looks like Linux, acts like Linux, but then you have to use all of the awful Vyatta commands to actually do anything.
If you are expecting a Linux distribution then yeah, it is weird. If you are expecting a network appliance then it's par for the course and you are grateful that you can drop to shell to do other things. JunOS is pretty much the same way.
It's not awful - just different. They specifically did it that way because people who configure large networks are much more used to this kind of interface (a la Cisco and Juniper hardware) and like this way a lot better.
I'm hoping to really work on improving the documentation in the new Vyatta fork, VyOS, because poor documentation really let Vyatta down.
Related: I picked up a couple of Neoware CA21 thin clients on eBay real cheap recently. 800 MHz, 512 MB RAM, and 1 GB disk-on-module. I built customized (NanoBSD) images and run them read-only off of a flash drive. I connected a Garmin GPS 18 LVC to one and have my own stratum 1 NTP server for < $100.
Unfortunately by the time it's released, the hardware will already be a bit old. But nonetheless, the specs are certainly sufficient for most cool things.
Yea, for something like a single-purpose DNS recursor or NTP server, any cheap hardware will do. You got your terminal server for what? 15 dollars? Much more reasonable than the 100 bucks an ERL costs.
That $100 is funding the ongoing development of EdgeOS though. It doesn't seem to make much sense to buy an ERL if you're just going to put another OS on it. Shrug.
Newer Atheros routers are 2 chip solutions with radio chip hanging on PCIE connection. It is possible to cut this connection and insert PCIE Bridge between.
Note that UBNT claims line rate throughput and provides proof via a commissioned Tolly report (no idea on their legitimacy).
The fbsd image yields ~250mbps between two gigabit hosts.
I've been eyeing one of these for a while and putting fbsd on it interests me, but not if it means a huge peformance drop (even if I wouldn't really miss it).
Assuming UBNT is correct and the device supports line-rate throughput, I don't think there's any reason why FreeBSD couldn't easily support that.
The freebsd-mips mailing list already has a few threads discussing this; if you want to know why the router isn't achieving line-rate speeds "out of the box", you'll probably get the best answer there (and maybe even someone who knows how to fix the problem):
To get any decent performance you have to use a horrible binary blob kernel module that offloads (certain very specific use cases) to hardware acceleration. See this post for more info: https://news.ycombinator.com/item?id=7282258
The downside of this right now is that there are no binary packages for mips64 and there is no space on the device to store a ports tree so you need to cross compile everything.
I am running gentoo on mine at the minute, I did have FreeBSD on it for a bit.
I tried a larger flash drive but the driver doesn't like it and won't mount the root partition, need to find one it likes. Or use nfs/smb but that is a pain.
I successfully connected a USB hard drive to mine. If you're interested enough, email me and I'll go look up which inexpensive model of enclosure I'm using. The bootloader is indeed finicky but I guess I got lucky enough to have a HDD enclosure that just worked.
I must have some lying around, will give them a go. The bootloader works with my USB stick, but Linux doesn't like it. Will see if FreeBSD does or not.
Could you store the ports tree on a network share, mount that share on the device, and compile that way? Seems like that would be easier than cross-compiling, if it'll work.
Neat. I've been using an ERLite for the past ~6 months. It does its job, but I'm not super excited by it. I find the Vyatta interface to be a little annoying (most enterprise network gear I've worked with just copied Cisco's CLI exactly, Vyatta is just different enough to be surprising).
I'm thinking about swapping it out for a Mirabox[1] I have laying around. Its an ARMv7 with dual Gigabit interfaces and 1GB of memory. With the 3.14 kernel the SoC is quite well supported. Now if only I could run pfSense on it.
Qualcomm-Atheros AR9331 based systems, with 64MB ram and 16MB flash (kind of hard to get in that config cheaply). You can get decent performance, potentially hang a 5.8/2.4ghz radio on it too, and power it off a single laptop USB port (200mA normal power use, and with stuff, still less than 500mA.) Can put second ethernet, wifi, 3G, etc. there. There's no current commercial hardware in this space which is perfect. That should change shortly.
Much beefier systems with "real" amounts of RAM and flash. I generally skip the Broadcom/RPi/etc. and go straight to an Atom or i3, but that's just for convenience, since I only need a small number of them. There's probably something more reasonable to do in the embedded space, but I'm indifferent on qty 1 devices between $500 and $200, especially if I already have most of the components for the $500 thing lying around, and it saves me an hour or two getting it working.
There are a lot of things missing from widely available commercial hardware in this space right now.