RM-RF#1 Jeremy from Keyrock
"A faulty websocket can cost us thousands of dollars"
👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.
This week I interviewed Jeremy De Groodt, co-founder and CTO at Keyrock. Keyrock is a crypto market maker and liquidity provider, counting big exchanges such as Binance as customers. With several €M worth of assets managed by Keyrock, Jeremy is the perfect fit for a first encounter with a CTO with dev war stories.
🕰 For how long have you been working on Keyrock?
We started Keyrock in December 2017 and I’ve led the tech part ever since. I started the team with 2 interns and we managed to raise funds based on the prototype developed back then.
🏉 How big is your team today?
We’re now 40 in the team, of which 17 devs and 10 financial traders looking after the markets. The rest of the team is also composed of quants, folks with PhDs in maths and physics - skills that are key to propel our activity of algorithmic trading.
It’s really important to be able to count on this team as any screwup can have significant financial impact 💸. We manage several millions of euros in our order books at anytime, and it’s a nightmare if the process around them fails.
🛠 Describe your tech stack in a nutshell?
We’re organized in 3 teams that reflect our stack:
The infrastructure team takes care of devops, tooling and a lot of data engineering (e.g. for monitoring and back-testing). We ingest a humongous amount of data from real-time price monitoring of crypto exchanges, and this causes significant challenges. To give you a sense of the magnitude of our data streams: we’re in direct contact with AWS regarding Kinesis data streaming as we constantly hit their max capacity.
The tech operations team mainly handles deployments which are still mostly manual given their criticality. We do a lot of progressive / canary deployments, one market at a time. These folks are like super-devops that also handle ticketing from within the team, e.g. if the quants suspect that something is wrong with our systems, the tech ops folks will look into it directly.
The algo team is the biggest one and organized in two parts: one dedicated to financial decision-making (powered by Rust), and one dedicated to maintaining connectivity with the exchanges (powered by Node). We’re integrated with 90 exchanges which all have different APIs and do not maintain SLAs… making it sometimes hard to differentiate between issues coming from bad data from exchanges or from bad financial decisions. As such, a faulty websocket with them can cost us a lot of money.
Apart from that, our architecture is being consolidated to a dozen microservices, so that we reduce inter-services communication latency (which is literally worth money in high-frequency trading). In that same philosophy, we prefer using gRPC/Protobuf for speed rather than JSON.
😱 What is the most stressful tech situation you ever faced?
By the end of 2020, our financial performance started to deteriorate. We could not find any obvious reason for this trend, nor financial nor technical. At some point just before XMas break, it got so bad that we were actually losing more money every day, with everyone freaking out about it.
We had to make a bold move, and we did: we cancelled everyone’s holidays, had people come back to the office, and tasked the whole company on finding the cause of that issue. People were a bit pissed but very committed too because the pressure was real.
We ultimately tracked down the bug after several days, on the morning of XMas 🎁 ! It turns out that we were receiving outdated data from some exchanges (up to 30s old), which led us to make bad financial decisions. This was a stark reminder never to take external APIs performance for granted. Since then, we’ve timestamped their data like mad and have been applying penalties on outdated input in our algos. Being fully data-driven as we are also requires to go meta and measure the quality of your data 🔍.
⭐ Are you hiring any developers these days?
We’re looking for Rust developers as well as full-stack JS devs to improve our tooling and connectivity with exchanges.
We’re also looking for a TechOps person: a superdev / analyst with a Swiss knife mentality to help running our tech operations.
❤️ Finally, some learnings about yourself as a tech leader?
As a CTO, I’ve always struggled to find the right balance between my love for coding and the need to step back and maintain a strategic vision.
It has greatly helped to recruit very talented people on some fronts (infra, algos). They fully own the code and ensure I do not fall into the rabbit holes of the details. My role with them becomes that of challenging decisions at high level and pushing the company strategy.
Nevertheless I still like to code and try to commit on our codebase once in a while (without becoming a bottleneck for other devs 🤭). And I still do some side-projects on my Raspberry Pi!
Thanks a ton to Jeremy for this interview 🙏. I’ve had the pleasure of collaborating with him on several occasions (even a full-night Blockchain Hackaton) - he’s a super inspiring tech leader, so do not hesitate to reach out to join his team. See you in our next of issue of RM-RF with Sandro Munda from Forest Admin.