👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This time I interviewed Jonathan Alzetta, CTO of Wooclap. Wooclap is a collaborative platform for classes, conferences and training sessions, used by 1M+ teachers and trainers worldwide. From the early university days to scaling-up, Jonathan had an eventful tech ride. Read on for exciting anecdotes!

🕰 For how long have you been working at Wooclap?

It’s a long story that started right after my cofounder and I graduated from École Polytechnique in 2013. By then we were organising study sessions and catch-up lessons with junior undergrads - it was our first project in the education space and we got along well together (note: it’s also a suggestion from Hubspot CTO that you should try small-scale/time-scoped ventures as junior cofounders to test your relationship 🎓).

I had studied nuclear physics and also took interest in computer science. Turns out, the latter was easier to practice on my own than the former 😅. So I ended up taking a job as a developer.

With Seb, we then took part in Start.LAB and launched the first iteration of Wooclap to enable live polls in classrooms. The first app was called DYHAQ = Do You Have Any Questions? It took off a bit and morphed into a company that I joined full-time in 2015!

🏉 How big is your team today?

Today we’re a team of 60 people at Wooclap, with 21 people in the tech team, including data scientists, product managers and product designers. We’re loosely organised into squads:

• On the core product of Wooclap, the squad is self-supporting with a full-time PM and designer;

• On growth, several team members of the business and tech teams assemble to tackle specific initiatives;

• Data scientists either focus on pure data projects or get staffed on other squads depending on their needs.

A bit over a year and a half ago, we were still only 20 in the whole company. Covid has drastically increased the share of distance learning and hence the need for our solution. This is especially relevant for teachers who lack direct feedback during their online courses. And with the progressive return to in-person classes, Wooclap still remains important for teachers to empower them to interact with their students. This gives us great business opportunities, along with tech scaling challenges.

🛠 Describe your tech stack in a nutshell?

We’ve got 2 products with similar stacks:

• Wooclap is our core product for interactive classes. It’s based on Node + Express with MongoDB and React with Styled Components. It’s a pure web app even though it’s often used on smartphones: we wanted to remove any friction that could occur from asking participants to install apps on their devices.

• Wooflash is our more recent product for spaced repetition flash cards (note: Jonathan even taught himself Japanese characters using this method 🎌!). It includes a bit of Apollo / GraphQL and uses Postgres instead of MongoDB.

Both products run on AWS infra. Our data stack leverages R, Shiny and a bit of Python.

🐛 What is the most elusive bug you and your team ever had to fix?

We’ve got 2 of them which are still out in the wild because they’re hard to catch and repro - they're pretty uncommon 👹.

• Wooclap includes ordering logic to sequence questions and events in the app through drag-and-drop. We sometimes run into duplicate indices which screw up the ordering, due to race conditions we still haven’t completely figured out (note: if you ever built an app with kanban-like behaviour, you probably came across that problem too - this excellent article goes in great depth about handling user-defined ordering in your data model).

• And then we’ve got the Microsoft black box problem with our PowerPoint integration used by teachers (to embed our poll results). The inner workings of PPT are so unpredictable across versions - a real debugging nightmare. This sometimes leads to our charts not being displayed due to sizing issues, or presenter view and audience view being inverted due to a bug in PowerPoint itself 🤯.

Fortunately both of these situations are well contained even if they episodically pop up anew.

😱 What is the most stressful tech situation you ever faced?

This was around 6 years ago, in the early days of Wooclap when we used to be on-site to provide our services during conferences and other events.

We managed to sell to the organisers of a conference with 1,000 attendees - a number of concurrent users we’d never faced before. Already in the prep phase for the event, we got very custom feature requests that we had to implement in a rush. We nevertheless managed to land these in time. What happened however is that during the event, the speaker messed up the script and gave the wrong Wooclap instructions to attendees. This led them to all use the “take a selfie” feature at once, which completely crippled the wi-fi bandwidth of the venue. The connection hung out for 5 minutes till the organisers managed to phone their internet provider and temporarily boost bandwidth. This is something we’d never expected in our preparatory drills. Fortunately, the rest of the event went along smoothly.

Another related horror story happened when we ran a “competition-mode event” on Wooclap which had participants race to submit their responses first, with live scores displayed. It turned out to be a big initial mess with race conditions all around and inconsistent scoring displayed to all 🤭. This led us to drastically improve our real-time handling capabilities to reach sub-10ms latencies. We found out for example that socket.io doesn’t scale well to 1,000+ connections, and eventually decided to work with a specialised solution (Ably, similar to Pusher).

✌ What's your best piece of learning on these topics?

Along the way we’ve learned a lot about ensuring stability in environments we don’t totally control: events, classrooms, the Microsoft tools… This has made us more aware of potential issues upfront and increased our resiliency in face of these.

⭐ Finally, are you hiring any developers these days?

We're always on the lookout for aspiring talents to join the Wooteam. Right now, we're looking for security engineers and developers (frontend & backend), based either in Paris, Brussels or remote.

Thanks a ton to Jonathan for this interview 🙏. Wooclap is a beautiful Belgian success story with an inspiring mission, kudos to them 👍 If you’d like to work with Jonathan, join Wooclap’s team here. See you in our next issue of RM-RF with Antoine Louiset from Yousign.

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This time I interviewed Laurent Perrin, co-founder and CTO of Front, a communication hub powering extraordinary customer experiences. Along his nearly 10-years journey at the company, Laurent has been through some pretty crazy stuff.

🕰 For how long have you been working on Front?

We started working on the project back in 2013 with Mathilde and quickly moved to SF to join YCombinator in 2014. Two years later we raised a $10M series A and have been growing fast ever since. Our latest fundraise was in 2020, with participations from leaders of the Future of Work industry (note: Front has Zoom’s CEO in their investors, how cool is that).

🏉 How big is your team today?

Today our team counts about 320 people. We initially took a small hit from the covid period, but things rapidly bounced back as companies adapted to new work realities which Front supports. This period proved our resiliency and enabled us to confidently grow the team afterwards.

In the R&D team, we’ve got about 90 people. Most of them (70) are engineers, and we’ve also got a product & design team as well as a pure research team. Each team has their own leader. Interestingly, none of these leaders are formally reporting to me (note: this setup is also that of Dharmesh Shah, Hubspot CTO). I strive to be a partner rather than a manager. My role is hence focused on driving the overall vision and being custodian of the product and tech architectures. I’m also the voice of tech and product at the Board, to ensure we do put realistic expectations on engineering (Laurent referred here to the Boeing board which lacked engineers and led to the issues with the 737 Max).

🛠 Describe your tech stack in a nutshell?

Our back-end is a Node.js app with a constellation of small services (150) that exchange messages using queues. It’s noteworthy that our stack remained based on Node since its early days, without major shifts. In the front, we use React + Redux.

Our infra runs on AWS with Kubernetes, leveraging their proprietary SQS as well as common tech such as MySQL, Redis and ElasticSearch. Each AWS region has independent infra stacks.

We have sharded our main DB on 25 shards split by customers. We initially met a few issues linked to shard rebalancing occurring at bad times (e.g. during high user activity load). This led us to create a “stop button” to prevent rebalancing in set timeframes.

🐛 What is the most elusive bug you and your team ever had to fix?

We’ve met a seemingly futile but very frustrating bug related to our inbox messages counters (notification dots). Basically, the counters got out-of-sync with the actual messages count, leading users to believe they had unread messages when they had none (note: this was also a hard problem for Facebook back in the days, which ultimately pushed to the Flux architecture and React).

This was a complex real-time events issue. Counters are influenced by a plethora of events - with up to 200,000 updates per second at our scale. Even if we’re good on 99.999 of cases, it still means we generate 1-2 errors / second.

We initially tried to raise awareness within the teams on the need to pay extra attention to counters consistency. But as we grew the team we had to resort to more automated means. So we ended up carpet-testing counters across the whole test suite with something behaving like a global after hook.

😱 What is the most stressful tech situation you ever faced?

In 2018, we shifted our infra to Terraform, meaning we had to start with a clean-up of the legacy “manual” infra. At some point during the deploy, Terraform identified a remaining KMS key (encryption key hosted on AWS) which had not been wiped out yet, and prompted the developer to do so.

The developer deleted the key and everything went smoothly for the next 2 hours, because AWS puts that grace period before actually destroying the key… It’s only after deletion that we realised it was the one key encrypting our production database.

This was really the worst event we ever had, as basically no one could access any data anymore. We were very lucky that we had an odd replica of the DB in another region with its KMS key untouched, purely because someone had been paranoid at some point and thought it’d be a good idea to have this redundancy.

We went through 6 hours of total downtime. Now, each shard comes with its replica. We contacted AWS about this event and they have since improved the KMS experience so that it cannot happen anymore (note: AWS now marks clearly when a key is directly linked to an AWS-managed resource such as a DB, with no deletion possible). We all sympathised with the developer who went through this adventure because of the misleading UI by then.

✌ What's your best piece of learning on these topics?

Through my career, I’ve learned to shape clear expectations for the engineering teams. Before Front, I got trapped in a project with unrealistic technical expectations from the unset. Because of that I worked so hard I literally ended up at the hospital, and the project eventually didn’t deliver meaningful value. But I came back stronger and much more assertive about tech objectives.

⭐ Finally, are you hiring any developers these days?

We’ve got more than 30 open positions at the moment. Roles are located in Paris, San Francisco and Remote.

Thanks a ton to Laurent for this interview 🙏. He’s a passionate tech leader with interesting takes on the role of the CTO in a fast-growing scale-up. If you’d like to work with Laurent, join Front’s team here. See you in our next of issue of RM-RF with Jonathan Alzetta from Wooclap.

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This week I interviewed Charles De Groote, co-founder and CTO of Sortlist, a marketing agencies matchmaking platform. Charles is a very pragmatic tech leader who’s learned a ton on the job as his team grew to 100+ people. Read on for exciting anecdotes!

🕰 For how long have you been working on Sortlist?

We launched the company 7 years ago together with my 3 co-founders. I was initially the only founder with a technical background. Interestingly, Michael, the founder who originally headed Product Management, decided to learn to code from scratch after we’d launched the company. He ended up working in the dev team and has been managing developers ever since.

🏉 How big is your team today?

The whole Sortlist family is 100+ people strong today. Of these, we’ve got 15 people in the engineering team and 5 people in the product team (3 PM + 2 designers), which is headed by Robin, our Head of Product.

The engineering team is composed of a dozen of versatile devs, along with 1 QA, 1 infra manager, and 2 line managers that handle all the 1-1s etc. These folks in particular give me a lot of leverage as I was getting overwhelmed by the 1-1s, especially in the beginning when we hired mostly junior talents. Nowadays I even find time to continue coding myself 😋.

We work as 3 independent product squads and perform quarterly “mercatos” where juniors and mediors get to change teams to tackle different challenges. Senior people mostly remain in their areas of expertise.

🛠 Describe your tech stack in a nutshell?

Well, it has evolved a lot over these 7 years, along with our knowledge and skills. It originally started with a Ruby on Rails monolith and an Angular front-end.

2 years ago we started splitting the back-end into microservices. After many experiments (NodeJS + Express,  NestJS, Python + Flask, Ruby on Rails), we settled on RoR as our go-to framework. We took the same journey in the front-end 1 year ago, picking Next.js this time.

In terms of infra, we mostly run on AWS with Kubernetes. We were initially on Heroku but had to move away, partly for DB costs reasons. At €2K/month, they represented 50% of our infra costs - which we were able to bring down using AWS AuroraDB. Another issue with Heroku was inter-services communications which required us to go over the public internet to pass messages - clearly not an optimal setup.

It’s worth mentioning that we at Sortlist handle multiple apps corresponding to our multiple stakeholders: agencies, consumers, back-office… That’s 9 apps in total. We’ve hence been in fierce discussions over the opportunity to try and group it all as a monorepo to leverage common modules. However this would introduce more overhead in terms of DevOps (to silo builds etc). Given that infra is currently our least staffed area, we decided to keep it simple with separate repos.

🐛 What is the most elusive bug you and your team ever had to fix?

You surely know this famous quote from Phil Karlton:

There are only two hard things in Computer Science: cache invalidation and naming things.

Well, we suffered badly from caching issues related to cookies 🍪. Our multi-apps stack runs on multiple domains, which is made more complex because we use geographical TLDs such as .fr and .be. We made the painful mistake of sharing cookies between all these apps and domains, which is a dangerous pattern with hindsight.

What happened is that for the short period of one release, we unknowingly screwed up authentication in one app and this resulted in corrupted cookies being released into the wild and cached by our users for all apps. As a consequence we started getting more and more reports of users not able to login at all. We never managed to repro the issue while this problem came and went for several weeks. That is, until one of our employees actually faced the problem themselves. We of course got super excited and told them “Don’t touch anything, a dev is on his way to check this out!". That’s how we finally diagnosed the cookies issue. We ultimately had to go nuclear to fix it, invalidating all cookies at once 🤯.

😱 What is the most stressful tech situation you ever faced?

It’s again something caused by our relative ignorance back in the days - 5 years ago to be precise. By then we didn’t master all the intricacies of DB management, which resulted in lengthy migrations and often required putting the website in maintenance mode.

One such migration began innocently one night at 7PM and rapidly morphed into a monster event which required a ton of manual operations from my side. I ended up having to work the whole night on this situation. I normally never drink coffee, but when my co-founders came to work on that morning I had to ask them for one - probably the only time they ever got this request from me ☕!

✌ What's your best piece of learning on these topics?

We’ve learned a lot all these years about the pitfalls of over-engineering our systems. At some point for example we rolled-out our own custom reverse-proxies in Node which led to unmanageable timeouts… Now we’ve wisened up as a team and focus on pragmatic solutions to real business problems.

Personally as a leader, I’ve learned how valuable it is to surround yourself with the right people with the right experience for the job. An example is the hiring of our DevOps manager and that of our more senior developers, who’ve brought in expertise I could never have developed on my own.

⭐ Finally, are you hiring any developers these days?

We went through a temp freeze with the covid situation but are now fully back on track and hiring 1 additional DevOps engineer as well as 3 React and 3 RoR engineers.

Thanks a ton to Charles for this interview 🙏. It’s been very exciting to follow Sortlist’s story since its early days. Also cool to meet someone who’s worked with Ruby on Rails and NestJS, enjoying the same similarities between both frameworks ❤️ If you’d like to work with Charles, join Sortlist’s team here. See you in our next of issue of RM-RF with Laurent Perrin from Front.

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This week I interviewed Quentin de Metz, co-founder and CTO at Pennylane, a fintech accounting platform. Quentin is a repeat entrepreneur who first founded Pricematch which was ultimately sold to Booking.com back in 2015. Together with the same co-founders, they launched Pennylane which is already proving a success. Quentin’s insights are super interesting for sharp tech execution in a fast-growth setup.

🕰 For how long have you been working on Pennylane?

Together with my cofounders, we started brainstorming on the topic of accounting during Summer 2019. We raised €4M in January 2020 followed by €15M just one year later. Our past experience at Pricematch definitely helped convincing investors and enabled us to move so fast.

🏉 How big is your team today?

As of now we’re about 100 people in the team (incl. the accounting service team), with a team of 30 dedicated to the product (note: that’s about 2 ppl recruited / week for one year non-stop!). Of this team, 23 are devs and the rest are product designers and owners. On the tech founders side, we’re 2 tech-focused founders and a 3rd one is focused on product.

A specificity of the product team is that we’re the only part of the company that’s fully remote - which definitely helps in sourcing the best talents. With that rate of growth, recruitment and onboarding are critical activities. As such, we’ve managed to put together a nice onboarding pipeline which allows newcomers to be fully setup and production-ready after 6 weeks.

🛠 Describe your tech stack in a nutshell?

We’ve got a very no-nonsense stack, preferring boring and pragmatic tech over shiny things. Our app stack is a Ruby on Rails monolith with a React frontend. When it comes to infra, we’re hosted on Heroku for compute and use AWS for DB (with RDS Postgres), storage and DNS.

Despite the fact that we’re dealing with financial fluxes, we’re not actually facing hard number-crunching jobs or big data challenges. Interestingly, the topic of accounting is also less complex than, say, HR and payroll because there are less custom rules. Our strategy to focus on a narrow set of accounting standards has also greatly helped us in keeping complexity under control.

😱 What is the most stressful tech situation you ever faced?

I've got 2 stories from the Pricematch days, which were rich in learnings 😊.

Back in the days, Ruby on Rails didn’t use the BIGINT (64bits) data type for primary DB keys. And it turns out that we hit the 32bits limit with a table that stored the daily prices of tracked hotels, which recorded up to 1 billion records a day. This meant we had to perform a humongous migration to the BIGINT type, which took 24 hours in total during which the app was fully down 😅. Fortunately, this default has now been updated to 64bits since RoR 5.1 came out.

We then moved this table to AWS DynamoDB for performance reasons, which worked out quite well and cost us about $1/customer/month. This was completely fine given our monthly fee of $200/customer. But then we got acquired by Booking.com and the team there thought it’d be a good idea to offer this service for free to all their hotels. There were 200,000 of them and I was somehow not put in the loop in that decision… until finance came knocking on my door asking why they’d get this super large bill for DynamoDB, 200x larger than the previous one! As a fix, we ended up keeping only a subset of the latest prices and doing some machine learning on them.

🐛 What is the most elusive bug you and your team ever had to fix?

We’ve got a very focused bug-hunting mentality at Pennylane so that nothing falls through the cracks. We’ve gotten very good in particular at spotting anomalies in our DB (which is key for accounting) and tracing back the culprits.

We’ve deployed 2 valuable systems in that regard:

• For spotting anomalies, we use Metabase which performs daily CRON SQL requests on a read-replica of our DB looking for anomalies (e.g., VAT invoice elements that don’t add up) and alerts us by mail if any suspicious results are found

• For tracing back the bugs that led to these anomalies, we have patched ActiveRecord so that each model saves the uuid of the request that last touched it. When we find a buggy record, we can then use that uuid and grep it in our logs to understand its origin

✌ What's your best piece of learning on these topics?

From the Pennylane growth journey, I’ve mostly learned how important it is to invest in recruiting. It helps so much to be very clear on the kinds of traits and skills you’re looking for so you’re less prone to make suboptimal compromises. Recruiting choices tend to be self-reinforcing: bad recruits make you lose a lot of time, while good recruits give you so much leverage.

⭐ Finally, are you hiring any developers these days?

We’re aiming to double our size towards 40 people in the tech team. We’ve originally been looking for full-stack Rails + React folks, but lately we extended openings for pure front-end positions too. We’re remote-first and growing fast, so do ping us to join the team!

Thanks a ton to Quentin for this interview 🙏. The conversation we had was very inspiring to try and become a sharper and more focused tech leader myself. To get inspired too, join Pennylane’s team here. See you in our next of issue of RM-RF with Charles De Groote from Sortlist.

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This week I interviewed Sandro Munda, founder and CEO at Forest Admin. If you’ve ever been through the pains of building a custom admin panel for your apps, you’ll absolutely love their product as it does it all for you. Techwise, it’s a SaaS with an elegant architecture plugging into your DB - which brings a lot of cool challenges, as you’ll discover here.

🕰 For how long have you been working on Forest Admin?

I wrote the first line of code 5 years ago after having struggled with admin panels as a freelance building custom apps. During my customer discovery calls, I’d ask “do you need an admin panel?”, and choose to go with Ruby on Rails just for this reason to leverage RailsAdmin 🙃. Making a tech choice based on admin requirements is clearly suboptimal, which led me to identify this cool product opportunity.

I set on to build an MVP tech-agnostic admin panel that would be quick to setup (like ActiveRecord), flexible (as RailsAdmin is a poor solution beyond CRUD actions), and secure (as admin panels are by nature connected to sensitive data). I joined forces with eFounders, they saw the value in it, we tried it out at Fotolia (Thibaud Elziere’s previous company), and so the journey began.

🏉 How big is your team today?

Today, we’re about 30-people strong, of which 20 devs led by our CTO Arnaud Besnier. We recently reshaped our dev org to 3 subteams:

• The DevEx team, which develops the connectors (like our new RoR integration) to ensure a good experience for developers that roll-out Forest Admin in their stack

• The OpsEx team, which drives pure operational features, such as integrations with 3rd parties (Stripe, Zendesk, Salesforce and many others)

• The CoreEx team, which develops and maintains transversal parts of the product such as auth, billing and security

Besides this, we regularly organize into war rooms to tackle specific topics (typically including 2 devs, 1 product person, 1 designer and 1 biz owner).

🛠 Describe your tech stack in a nutshell?

We’re a JS powerhouse with Node in the back and Ember in the front. We chose Ember as it offers the most flexibility and robustness when it comes to generating the fully dynamic UIs and data models required in the admin views and actions.

Nevertheless, we like to think of ourselves as tech agnostic and also ensure that all devs keep a full-stack mindset. This is a real requirement for us, given that we integrate with multiple frameworks across languages (after Node and RoR, we’re now aiming for Django and Laravel).

Our infra is rather straightforward and monolithic, leveraging Postgres and Redis for persistence and cache.

😱 What is the most stressful tech situation you ever faced?

Well, given the integration challenges of our product, we never know what we’ll bump into next! But there’s one anecdote which is still vivid, from the very early days of Forest Admin.

The first MVP I implemented was basically hardcoded for a single user. When I ran the first onboarding with the first user, the app crashed because I already registered myself as user. I hence had to perform a full production reboot in front of the user - which led this user to claim ID 1 in our DB, while I’ve got ID 2 😅.

🐛 What is the most elusive bug you and your team ever had to fix?

I can’t think of a specific bug, but I can say that Forest Admin is probably one of the best debugging schools out there. We basically run 1/3 of our code within our users’ environments, with little to no visibility into what’s causing issues. We may get stack traces and screenshots but rarely get the opportunity to explore things ourselves. As a consequence, it forces us to ask extremely sharp questions to our users during debugging, which is quite rewarding when you start mastering it.

It also often turns out that “bugs” are caused by exotic elements in our users’ codebases. This was actually our #1 tech challenge: being able to deploy Forest Admin on real-life legacy codebases with all their intricacies and oddities. In retrospect, this is proving to be one of the most powerful barriers to entry for competitors, as we’ve met and unit-tested a huge variety of setups.

✌ What's your best piece of learning on these topics?

My biggest learning as a dev is to always resist the temptation to perform a full rewrite, even in the face of seemingly overly complex layers of code. As we’ve discovered with our support for a wide variety of setups, complex code is there for a reason and is hard to reproduce from scratch.

⭐ Finally, are you hiring any developers these days?

We’ve got plenty of open positions, mostly full-stack and tech agnostic. Our growth strategy is product-led, which means we expect developers to heavily drive the business going forward. If you’re open-minded and excited to join a growing company with a strong tech culture, you should definitely jump aboard the rocket ship 🚀.

Thanks a ton to Sandro for this interview 🙏. Sandro is such a passionate and authentic leader. We first met over lunch 5 years ago when he was just starting Forest Admin and it seems he has kept the same aura of energy when talking about the product ever since. Join his team here. See you in our next of issue of RM-RF with Quentin de Metz from Pennylane.

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This week I interviewed Jeremy De Groodt, co-founder and CTO at Keyrock. Keyrock is a crypto market maker and liquidity provider, counting big exchanges such as Binance as customers. With several €M worth of assets managed by Keyrock, Jeremy is the perfect fit for a first encounter with a CTO with dev war stories.

🕰 For how long have you been working on Keyrock?

We started Keyrock in December 2017 and I’ve led the tech part ever since. I started the team with 2 interns and we managed to raise funds based on the prototype developed back then.

🏉 How big is your team today?

We’re now 40 in the team, of which 17 devs and 10 financial traders looking after the markets. The rest of the team is also composed of quants, folks with PhDs in maths and physics - skills that are key to propel our activity of algorithmic trading.

It’s really important to be able to count on this team as any screwup can have significant financial impact 💸. We manage several millions of euros in our order books at anytime, and it’s a nightmare if the process around them fails.

🛠 Describe your tech stack in a nutshell?

We’re organized in 3 teams that reflect our stack:

• The infrastructure team takes care of devops, tooling and a lot of data engineering (e.g. for monitoring and back-testing). We ingest a humongous amount of data from real-time price monitoring of crypto exchanges, and this causes significant challenges. To give you a sense of the magnitude of our data streams: we’re in direct contact with AWS regarding Kinesis data streaming as we constantly hit their max capacity.

• The tech operations team mainly handles deployments which are still mostly manual given their criticality. We do a lot of progressive / canary deployments, one market at a time. These folks are like super-devops that also handle ticketing from within the team, e.g. if the quants suspect that something is wrong with our systems, the tech ops folks will look into it directly.

• The algo team is the biggest one and organized in two parts: one dedicated to financial decision-making (powered by Rust), and one dedicated to maintaining connectivity with the exchanges (powered by Node). We’re integrated with 90 exchanges which all have different APIs and do not maintain SLAs… making it sometimes hard to differentiate between issues coming from bad data from exchanges or from bad financial decisions. As such, a faulty websocket with them can cost us a lot of money.

Apart from that, our architecture is being consolidated to a dozen microservices, so that we reduce inter-services communication latency (which is literally worth money in high-frequency trading). In that same philosophy, we prefer using gRPC/Protobuf for speed rather than JSON.

😱 What is the most stressful tech situation you ever faced?

By the end of 2020, our financial performance started to deteriorate. We could not find any obvious reason for this trend, nor financial nor technical. At some point just before XMas break, it got so bad that we were actually losing more money every day, with everyone freaking out about it.

We had to make a bold move, and we did: we cancelled everyone’s holidays, had people come back to the office, and tasked the whole company on finding the cause of that issue. People were a bit pissed but very committed too because the pressure was real.

We ultimately tracked down the bug after several days, on the morning of XMas 🎁 ! It turns out that we were receiving outdated data from some exchanges (up to 30s old), which led us to make bad financial decisions. This was a stark reminder never to take external APIs performance for granted. Since then, we’ve timestamped their data like mad and have been applying penalties on outdated input in our algos. Being fully data-driven as we are also requires to go meta and measure the quality of your data 🔍.

⭐ Are you hiring any developers these days?

• We’re looking for Rust developers as well as full-stack JS devs to improve our tooling and connectivity with exchanges.

• We’re also looking for a TechOps person: a superdev / analyst with a Swiss knife mentality to help running our tech operations.

❤️ Finally, some learnings about yourself as a tech leader?

As a CTO, I’ve always struggled to find the right balance between my love for coding and the need to step back and maintain a strategic vision.

It has greatly helped to recruit very talented people on some fronts (infra, algos). They fully own the code and ensure I do not fall into the rabbit holes of the details. My role with them becomes that of challenging decisions at high level and pushing the company strategy.

Nevertheless I still like to code and try to commit on our codebase once in a while (without becoming a bottleneck for other devs 🤭). And I still do some side-projects on my Raspberry Pi!

Thanks a ton to Jeremy for this interview 🙏. I’ve had the pleasure of collaborating with him on several occasions (even a full-night Blockchain Hackaton) - he’s a super inspiring tech leader, so do not hesitate to reach out to join his team. See you in our next of issue of RM-RF with Sandro Munda from Forest Admin.