Just poking my head in to say that I technically never departed ReadySet — what happened was that I co-founded the company, but was so burnt-out when it came to databases after my PhD that I decided to leave the running of the company to others. Then, US visa regulations made it so that I couldn't really be involved _at all_ if I wasn't an actual employee, which meant I truly was "just a founder" with no real involvement in the company's execution if you will. Now that I'm back in Europe, that's changing a bit, and I have regular calls with the CEO and such!
Hi Jon, thanks for recording! I'm excited to watch this. Do you happen to know why captions aren't enabled on your videos? Oftentimes the issue is that the video's primary language isn't set. Once this is done, youtube will probably caption the rest, though I'm not sure if that's true of videos of all lengths.
Hope you enjoy it! I have captions "enabled" (and primary language set) on all my videos, but my experience has been that YouTube is very hit-or-miss with whether it adds auto-caption to longer videos (somewhere around 2h seems to be the limit). Sometimes it appears later, it just takes a while, other times it just never manifests. It's unfortunate, but as far as I can tell there's nothing I can do about it :'(
Hi Jon, challenge author here! First time watching your content, it was fun to see a Rust expert go through the challenge live. Saw the first hour, I noticed that during Bencode parsing, trying to find the most elegant way to implement it slowed you down a bit. (I also have this tendency and I'm sure having so many viewers doesn't help :)) Great progress by the way in 4 hours, hope you get to finish the challenge soon!
thanks Jon really enjoyed the process. I always wondered why oci registries don't bit torrent the images. now I understand why they might not have been fond of the approach.
:wave: Author of the paper this work is based on here.
I'm so excited to see dynamic, partially-stateful data-flow for incremental materialized view maintenance becoming more wide-spread! I continue to think it's a _great_ idea, and the speed-ups (and complexity reduction) it can yield are pretty immense, so seeing more folks building on the idea makes me very happy.
As others have already mentioned in the comments, I co-founded ReadySet (https://readyset.io/) shortly after graduating specifically to build off of Noria, and they're doing amazing work to provide these kinds of speed-ups for general-purpose relational databases. If you're using one of those, it's worth giving ReadySet a look to get these kinds of speedups there! It's also source-available @ https://github.com/readysettech/readyset if you're curious.
for readyset: Is there a deb package available or something lighter weight than docker, kubernets, etc? I'd just like to run it as a regular unix process and start/stop it with systemd.
Is there a way to federate which keys are commonly accessed ? Like if I commonly access the entire table can I direct inbound traffic to different application servers and have them access different caches so each cache can pull only a subset of the data into the cache, and not worry about things like which keys are being written globally
We've thought about that, actually! We have an experimental mode where multiple copies of the same query can be created (actually just multiple copies of the leaf node in the dataflow graph, so intermediate state is reused) with different subsets of keys materialized - the idea is then that these separate readers would be run on different regions, so eg the reader in the EU region gets keys for EU users, and the reader in the NA region gets keys for NA users.
Oh hey, that's my thesis! Happy to answer any questions you may have about it :) There's also the OSDI'18 paper here which may be of interest: https://jon.tsp.io/papers/osdi18-noria.pdf
I haven't read much about Noria other than this readme [0], but would like to know if you are familiar enough to contrast Materialize [1] with it in terms of perf, overhead, approach, and fundamental (design) principles?
Back when we announced Materialize we got the same question in reverse! You can read my response from a few years back here: https://hackertimes.com/item?id=22362301
Unfortunately I'm not privy to whatever improvements ReadySet has made in the past two years, so I can't comment on differences between ReadySet and Materialize. Perhaps Jon can, though!
This is super exciting. Ever since I talked to you about Noria I've been telling people about this concept. I'm excited to see a production ready implementation of it.
I've been following the space since a bit of time, and I must say it's exciting. To me this is the future of apps where the Truth lives server-side, and everything reacts from there; With partial state evaluation lowering resource consumption to a minimum.
Kafka Streams and Apache Flink seem to be focused on real-time analytics, and I wish they'd get there to stimulate the space.
Basically, I co-founded the company around the time I graduated, but had had my fill of database research after six years of PhD. So I joined AWS to work on Rust while Alana (the CEO) took on leading ReadySet.
Nope, Dashmap is all xacrimon, and came on the scene long before my port. We've been collaborating on writing a shared benchmarking suite over at https://github.com/jonhoo/bustle/ though. For the time being, it looks like Dashmap outperforms the port of ConcurrentHashMap (called "flurry"), often by a significant amount. It seems to be mainly due to the garbage collection scheme flurry uses, but we're still digging into it (maybe you want to come help?).
I can't speak to the implementation differences between the two, but I know the author of dashmap is relatively active in responding online, so they may show up shortly to explain. In terms of performance comparisons, we're actually working on building a shared benchmarking tool for all of Rust's concurrent maps that you may find interesting: https://github.com/jonhoo/bustle.
