RM-RF#3 Quentin from Pennylane

"Our AWS bill grew 200x overnight"

👋 Welcome to this issue of RM-RF, where you get to hear about crazy dev war stories from inspiring tech leaders.

This week I interviewed Quentin de Metz, co-founder and CTO at Pennylane, a fintech accounting platform. Quentin is a repeat entrepreneur who first founded Pricematch which was ultimately sold to Booking.com back in 2015. Together with the same co-founders, they launched Pennylane which is already proving a success. Quentin’s insights are super interesting for sharp tech execution in a fast-growth setup.

🕰 For how long have you been working on Pennylane?

Together with my cofounders, we started brainstorming on the topic of accounting during Summer 2019. We raised €4M in January 2020 followed by €15M just one year later. Our past experience at Pricematch definitely helped convincing investors and enabled us to move so fast.

🏉 How big is your team today?

As of now we’re about 100 people in the team (incl. the accounting service team), with a team of 30 dedicated to the product (note: that’s about 2 ppl recruited / week for one year non-stop!). Of this team, 23 are devs and the rest are product designers and owners. On the tech founders side, we’re 2 tech-focused founders and a 3rd one is focused on product.

A specificity of the product team is that we’re the only part of the company that’s fully remote - which definitely helps in sourcing the best talents. With that rate of growth, recruitment and onboarding are critical activities. As such, we’ve managed to put together a nice onboarding pipeline which allows newcomers to be fully setup and production-ready after 6 weeks.

🛠 Describe your tech stack in a nutshell?

We’ve got a very no-nonsense stack, preferring boring and pragmatic tech over shiny things. Our app stack is a Ruby on Rails monolith with a React frontend. When it comes to infra, we’re hosted on Heroku for compute and use AWS for DB (with RDS Postgres), storage and DNS.

Despite the fact that we’re dealing with financial fluxes, we’re not actually facing hard number-crunching jobs or big data challenges. Interestingly, the topic of accounting is also less complex than, say, HR and payroll because there are less custom rules. Our strategy to focus on a narrow set of accounting standards has also greatly helped us in keeping complexity under control.

😱 What is the most stressful tech situation you ever faced?

I've got 2 stories from the Pricematch days, which were rich in learnings 😊.

Back in the days, Ruby on Rails didn’t use the BIGINT (64bits) data type for primary DB keys. And it turns out that we hit the 32bits limit with a table that stored the daily prices of tracked hotels, which recorded up to 1 billion records a day. This meant we had to perform a humongous migration to the BIGINT type, which took 24 hours in total during which the app was fully down 😅. Fortunately, this default has now been updated to 64bits since RoR 5.1 came out.

We then moved this table to AWS DynamoDB for performance reasons, which worked out quite well and cost us about $1/customer/month. This was completely fine given our monthly fee of $200/customer. But then we got acquired by Booking.com and the team there thought it’d be a good idea to offer this service for free to all their hotels. There were 200,000 of them and I was somehow not put in the loop in that decision… until finance came knocking on my door asking why they’d get this super large bill for DynamoDB, 200x larger than the previous one! As a fix, we ended up keeping only a subset of the latest prices and doing some machine learning on them.

🐛 What is the most elusive bug you and your team ever had to fix?

We’ve got a very focused bug-hunting mentality at Pennylane so that nothing falls through the cracks. We’ve gotten very good in particular at spotting anomalies in our DB (which is key for accounting) and tracing back the culprits.

We’ve deployed 2 valuable systems in that regard:

• For spotting anomalies, we use Metabase which performs daily CRON SQL requests on a read-replica of our DB looking for anomalies (e.g., VAT invoice elements that don’t add up) and alerts us by mail if any suspicious results are found

• For tracing back the bugs that led to these anomalies, we have patched ActiveRecord so that each model saves the uuid of the request that last touched it. When we find a buggy record, we can then use that uuid and grep it in our logs to understand its origin

✌ What's your best piece of learning on these topics?

From the Pennylane growth journey, I’ve mostly learned how important it is to invest in recruiting. It helps so much to be very clear on the kinds of traits and skills you’re looking for so you’re less prone to make suboptimal compromises. Recruiting choices tend to be self-reinforcing: bad recruits make you lose a lot of time, while good recruits give you so much leverage.

⭐ Finally, are you hiring any developers these days?

We’re aiming to double our size towards 40 people in the tech team. We’ve originally been looking for full-stack Rails + React folks, but lately we extended openings for pure front-end positions too. We’re remote-first and growing fast, so do ping us to join the team!

Thanks a ton to Quentin for this interview 🙏. The conversation we had was very inspiring to try and become a sharper and more focused tech leader myself. To get inspired too, join Pennylane’s team here. See you in our next of issue of RM-RF with Charles De Groote from Sortlist.