Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be preserved). Our system can also amend the data (for example to add some additional data to the trade later). All this causes trades to be a tree of immutable versions you need to comb through. A trade can have anywhere from 1 to 30k versions.

This takes about 20 seconds. The process opens about 200 connections to the cluster and transfers data at about 2-4GB/s.



Thanks for all the great insight.

Did you consider an explicitly bitemporal database like crux[0]?

[0] https://github.com/juxt/crux?


Our application predates crux. As to crux, I can't comment because I did not know about it when I was actively searching for a product to do this.


How many simultaneous queries of that nature can the system handle?


Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?


I'm asking about how your system scales to the number of queries, but you seem to be taking every question personally. You seem to really want to make sure everyone knows that you think Discord's problems are easy to solve. I'm not saying Discord is more complicated, but you're not really giving enough information to prove that Discord's problems are a subset of yours.

Do you support more simultaneous queries than Discord?


I think you need to recheck my username, you have mistaken me with another poster in the thread.


Actually, our threading is quite simple.

There is exactly as many threads (that do anything) as CPU cores.


I meant threading in as much as the connections or links between message data

All discord needs to store is:

{ channel_id, message_id, user_id, content, reply_to, datetime }

these days they added an extra thread_id field, sure. But the data itself is blisteringly uncomplex and there is only a single way to display it (ordered by time, i.e. the 'thread')


I think parent meant threading as in message threads.


Just recently Symphony switched order in which I have received two messages from a colleague. This completely changed the meaning of the message and we got in an argument that was only resolved after I have posted screenshot of what he wrote.

It seems, the threading might not be that simple after all.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: