Retrospective on application technologies used at Pyrra Tech
This is part two of my retrospective. You can also read about infrastructure and databases.
This section is focused on tools and libraries used to build our web application and API products.
Quick Links
Background
Pyrra’s two main products are a web application for media monitoring and alerting, and a data API that provide an interface to our underlying data store (Elasticsearch). I designed and built most of both products, and both were greenfield projects. So I got to make a lot of technology choices, both for the software and the underlying infrastructure.
tRPC
TL;DR 🎉 Wonderful; would use again.
tRPC is a web-application framework that abstracts away network calls and makes front-end and back-end programming fully type safe. There is no API layer that you have to think about. Many web application APIs only have a single client (at least for B2B Saas), often developed by the same team, so why not just run with it and develop the back-end and front-end like it’s one system? Of course there is still a server, and the tRPC RPC format is well documented, so non-tRPC clients are possible (not sure why you would, though, unless you were making calls from another server rather than a browser).
We were able to move very quickly with tRPC, with full type safety at all times. It almost completely eliminates runtime errors due to incompatibility between the frontend and backend. Coupling tRPC with Prisma makes it very easy to have end-to-end type-safety from the database to the UI, even into forms. We could make changes with great assurance that we hadn’t missed something.
I think tRPC makes a really good choice in wrapping React Query (now Tanstack Query) and in using Zod for input and output validations. Both Tanstack Query and Zod are great libraries. The starter-kit with NextJS, Prisma and Playwright made it very easy to get going.
There were only two real inconveniences with tRPC:
Queries can could only use GET
This was a real pain in the few places that it bit us. We had some large payloads we needed to send in queries but tRPC would force all queries to use HTTP GETs and we would run into URL length limits. The only viable workaround was to implement the queries as mutations, which all use POST. Mutations and queries are handled differently by react-query (which tRPC wraps on the client side) meaning that you lose the automatic caching that comes with useQuery and have to jump through some hoops to get the mutation to act like a query. Fortunately this has been fixed in v11!
Some of the type helpers are were complicated
Sometimes you need to get the input or output types of your query and mutation functions on the client side. In v9 this involved copy-pasting some code from the tRPC site and it wasn’t maybe quite as straightforward as it could have been. As of v10 it is!
Prisma
TL;DR Overall 🎉. Great; I’m not aware of a better all-round option for Typescript.
I’m not in the camp of “I’d prefer to write SQL in my code”. I don’t really like writing SQL, I don’t think it’s an especially well designed language, and the lack of composability is a real problem. Prisma makes it very easy to build queries and get type-safe results. The typings are perfect; the query syntax is fairly straightforward. It also has an easy schema definition language, generates good migrations, and migration management is easy.
Packaging / containerising
This is a bit of a pain, due to Prisma’s executable being a Rust binary. Just having to know about binary targets is a confusing thing that seems quite tangential to the job of querying a database. It mostly comes up if you’re trying to build a container on one architecture for deployment to another (eg if you’re building containers on your laptop before uploading them to ECR, which I did during early development). It’s one of those “fix it once and forget about it” things but it can be a source of confusion.
Not easy to use as a package in a monorepo
We tried to make our Prisma setup into it’s own package that could be imported either by our API code or the web-app. It just didn’t work out very well, and required some hackiness, especially in NextJS. We ended up moving Prisma back to the web-app and turning the API into a facade for the web-app (which was nice for other reasons; Prisma wasn’t the only motivation).
Strange / Unexpected SQL
Prisma was kind of notorious for emitting confusing or sub-optimal SQL, because for the longest time it didn’t support joins! I can’t say this ever caused a performance issue, and very rarely did I need to look at the raw SQL, but it certainly wasn’t producing what my mental model expected. Now fixed - Prisma can do joins!
Every and Some
Some people find every and some to be confusing. At times I have been one of them.
Migrations could be improved
Really just one QoL improvement for me: You can’t run forward to a specific migration, rather prisma migrate will apply all existing migrations. This is sometimes annoying for testing migrations. See my comment in Github for more details.
Express Zod API
TL;DR 🎉. Worked great. Just what we needed for a lightweight API layer.
Our API codebase went through several forms before settling in its final state of being a lightweight facade for logic living alongside the web app in our tRPC + NextJS codebase. We settled on this architecture because we had lots of helper code for searching and testing Elasticsearch over in the web-app. It seemed easiest to leave that code there and implement the handful of API endpoints into the web-app, rather than trying to create new packages or (worst of all) reinventing the Elasticsearch interface in the API codebase.
I chose express-zod-api because I needed to get the initial API out quickly for a potential customer. I wanted something simple, using technologies I was familiar with (Typescript, Express, Zod) but which didn’t feel like it would hold us back in the future. Express-zod-api has nice features like generating full documentation from the Zod schema, generating OpenAPI spec, etc… It’s well maintained (though it’s a solo project AFAIK) and I would definitely choose it again in the same circumstances.
Material UI
TL;DR 🙂. Solid choice; would happily use again but would also look at other options.
MUI’s the only big open-source UI component system I’ve used, and I don’t have too many complaints about it. I have used Tailwind a bit and I wanted something that was more out-of-the-box that required minimal thinking about CSS. I chose MUI on the basis of it being unlikely to be a bad choice. We had a lot to do and default styling was far less important than building features.
A lot of thought has gone into MUI’s design. It’s consistent, has plenty of components, and I know if I run into a problem with MUI there’ll be lots of help available online. Theming works well. We did get one potential customer complain that our app looked like a MUI app and we should have designed our components from the ground up, which I thought was kind of a strange thing to focus on.
If I was building a data-heavy app again I might look at a UI library that was more integrated with charting, such as Elastic’s UI library. But I would want to be pretty confident about long-term support before taking the plunge; I do not want to be maintaining an entire UI library or having to migrate because development stopped.
Highcharts
TL;DR. 😐. It worked fine, but it feels a bit long in the tooth.
I don’t have a whole lot to say about Highcharts but it didn’t feel like it was super well integrated with Typescript (to be fair, Highcharts is OLD) and I had some battles with it. I would definitely take a look around for a more Typescript-native-feeling (and React-feeling) charting library next time.
React Hook Form
TL;DR 😕. It worked, but I often found it confusing and felt there ought to be a simpler way.
Several times I was ready to rip out react-hook-form and replace it with anything else. It worked perfectly once I got it working, but I never found it intuitive at all, and spent quite some time battling it. I know there are nuances to forms, but it just seemed so much more difficult to work with than it should be. I thought this was due to using it with MUI, but that’s officially supported via the Controller component.
Maybe it was just that I don’t really like render props; maybe render props are appropriate here (HoCs and Sagas were also appropriate, once, until people moved on…)
I know a lot of people really like the library, so maybe it’s just me. None of the other well-maintained form libraries looked obviously better, and I did check a couple of times. But the frustration level was high. I’m hoping TanStack Form turns out to be great - just based on liking TanStack Query.
See my other posts Debugging validation issues between react-hook-form and zod and Using react-hook-form useFieldArray with useForm and defaultValues for examples of the confusing things I ran into.
Redux
TL;DR 🙂 Worked well, would consider again for specific use cases.
Most of the application relied on Tanstack Query for managing state (because a lot of state was just “cache a server response”) and also because tRPC bundles Tanstack Query. That worked really well, but there were two places where we had complex UI-only state:
- Search Builder: Our UI for builder structured searches using boolean logic. This had a number little form components (“blocks”) that could be grouped together into a tree, and complex validation logic. Something needed to track the state of the tree, validate it, and generate feedback and error messages.
- Explorer: Where search results could be viewed and filtered. Filtering was all done on the front-end for performance, so I needed to track filter state and also update the query string.
I picked Redux for these, though I hadn’t used it in a few years, and hadn’t been a massive fan when I did. But Redux has come a long way, and improvements like adopting Immer have made it much better. Also, part of why I didn’t enjoy working with Redux before was that we had also jumped on the Redux Saga bandwagon, and that made everything more complex than it needed to be (my opinion: Sagas are powerful and awesome in just the right situation. In practice almost no-one needs to use them and almost everyone should prefer something simpler. Also generators suck, especially when writing unit tests)
It might seem a bit odd to have two state management libraries, but overall the experience was positive. I was able to only Redux where it was required and then use Tanstack Query everywhere else. It was straightforward to use Tanstack Query to fetch initial state and then initialize Redux with it via a setStateFromBackend action. When data needed to be persisted from Search Builder I merely had to get the current state from the Redux store and pass it over to useMutation.
In a new application I would probably reach for Tanstack Query first, especially if there’s lots of state that’s really just tracking a server response (which is really common - think of data about the current user, you tend to use pieces of that in a variety of places). For complex UI with lots of state and user interactions with that state I haven’t used anything better than Redux and I think it works well there.