And I think that it deserves tremendous respect that you have done so. You could have hacked HN in python (or lisp?) in a weekend with the knowledge that it would scale, yet you decided to go with Arc. Very bold. And a great way to battletest a new language.

It's great to see people eat their own dogfood.

That said, the lisp world could use a good web framework, so perhaps this will inspire some lispers to create one.

I don't want to speak for pg, but I highly doubt this. He chose the architecture he did for good reason. Looking things up in memory is much faster than looking them up in the database. The only issue is that the site is too big to fit entirely into memory now, so he needs to write something that cleanly manages the memory space. (BerkeleyDB calls this a pager; data is stored in pages that can be in memory or disk.)

With that in place, the site should perform fine.

Relational databases are certainly useful in some situations, but there are usually better options for websites. You should consider learning about other options before telling everyone to use Rails. Rails is not the only way to do things, and it's rarely the best way.

I don't think your "poor fit" comment is true. You don't need to model the threaded nature of comments at all in your database. Since there's a relatively small number of comments per post, and most access to comments is probably show-me-all-comments-for-this-submission, you usually just need a single foreign key, each comment to its original grandparent submission.

In this specific context the relevant advantage of SQL over rolling your own is that there is less distinction between keeping data in memory and on disk. Otherwise, you can certainly do anything with memory + SQL that you can with memory + disk.

If you use an object database, you can store your data exactly as it is represented in memory. That is much cleaner, IMO.

(Relational databases are like programming languages. Just like you can use any programming language for any task, and you can hack any data you want into the relational model. But that's not always the best way. Sometimes a key/value store, or an object store, or a document store is a better model. Using a better model means you need to write less code, which means your app will have fewer bugs.)

I thought it was just a sign that I should get back to work, but maybe my account got put into some sort of inconsistent state?

(I can imagine an interface that is transparent at the arc level, where are strings are just passed to the backend and retrieved from it, and the backend converts them to and from byte strings. Later on it could change to use a FS or a DB or whatever.)

Which I believe should be called "lightning", right?

I presume that you're not hitting a melt-down from CPU time, so I'd assume you'd get better performance by letting the system handle paging in data from the disk and figuring out which stuff needs to be in memory (buffers) and lives out on disk.

Also one thing to watch out for when you make the 64-bit jump is that (internally pointer-heavy) applications in dynamic languages tend to use significantly more memory on 64-bit platforms. davidw talked about this some here:

http://journal.dedasys.com/2008/11/24/slicehost-vs-linode

Here's a page with some more details: http://news.ycombinator.com/item?id=452005

It's common for dynamic languages to embed typing information in pointers as an optimization. For example, CLISP uses at least 2 bits to distinguish between common types. That way fixnum numbers can be recognized and added without slow memory accesses.

The result is that you get less bits for the address. Hence less addressable memory.

For example, if you have an string type, which keeps track of it's length, then you might need 8 bytes. 4 for the pointer to the string of chars, 4 for the integer to keep count.

Here's a better example from tinyscheme:

    struct cell {
      unsigned int _flag;
      union {
        struct {
          char   *_svalue;
          int   _length;
        } _string;
        num _number;
        port *_port;
        foreign_func _ff;
        struct {
          struct cell *_car;
          struct cell *_cdr;
        } _cons;
      } _object;
    };

    typedef struct num {
       char is_fixnum;
       union {
          long ivalue;
          double rvalue;
       } value;
    } num;

In my experience, 64bit isn't a good way to save memory.

You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.

A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.

You could always use Erlang...

database are not tied to a storage medium. there is no reason why you can't run a DB (key-val or even a full RDBMS) in RAM.