I have been helping our team design an interface between components and its funny how things are always so similar yet so different. One of the things that I see a lot of smart engineers fail to do is how to design feedback into a system. People normally think of something like a TCP ack or an HTTP response code, ohh I got a 200 everything’s done! Your not wrong, but your also not entirely right.

I find Marc Brooker’s blog to have some really good examples and illustrations of what happens in open and closed systems and how multi-modal systems behave. I recommend you go sit down and read both of these blogs (honestly most of that blog is useful). He knows these things better than I (probably ever will) and explains this better than I can (or perhaps ever will).

https://brooker.co.za/blog/2023/05/10/open-closed.html https://brooker.co.za/blog/2021/08/27/caches.html

However, one thing I think Marc is missing is a tangible example of how caches are just another system. Your choice of an open or closed system and your choice of a feedback loop - dictate how your design operates - but the important missing piece is the interface. What is your interface to your upstream? What is your interface to your downstream? What is your interface to your cache? The interface is your feedback loop, and it doesnt matter what the interface is to. Therefore the interface (mostly) defines whether or not those components will be stable or metastable.

There are A LOT of different interfaces. We commonly work through a lot of them for a lot of systems, so give them some thought. Do you build a synchronous API? An asynchronous API? Plop data into a queue? Drop data and forget about it? Something else? There are use cases for all of these, but make sure you think about the implications.

I have been looking at new cars recently (long story!) and one of the things I was thinking about was stick shifts and some of the newer displays. A lot of new pop up displays are entirely electronic and controlled by a little computer, they’re nice because they display how fast you are going and can provide some fun graphics like how far the car is in front of you. These are external sensors providing you a feedback loop to modify your behavior. But, what happens when that interface crashes? Well, you can still roughly gauge how fast you are driving by how fast things are speeding by and the cars around you which acts almost as a second feedback loop. But, what if this was a stick shift? How would you know when to shift? Perhaps you can feel it, perhaps you know the gear ratios so well that you can judge when to shift by how fast you are going and what your incline is? That gets a lot harder. You really need that RPM sensor. There is still a sensor that could fail, there is always a sensor, but how many hops/transforms does that sensor go through? Every single one, by definition, decreases availability and increases latency. Importantly, some are more important than others, do you care about your engine temp 99% of the time?

What type of feedback loop are you providing to yourself? What type of feedback loop are you providing to your customers? Hey - just poll our api thats not too hard right? Hey - I am going to blast your ACK out via SNS and you can filter out everyone else’s ACKs to figure out what you want and dont lose it because Im only doing it once or twice idk. Hey - I am not going to directly tell you what happened, but you can go consume our downstreams results and see the results yourself! Sometimes the feedback loop easy, but most of the time its not. My guidelines, avoid open systems where you can. Provide a fast feedback loop where you can. Provide multiple forms of feedback where you can.

The other day my wife and I were cooking dinner and watching our two little ones, as we do. And she asks if I can add something to the shopping list. I am paying attention to a kid or zoning out or trying to get that perfect pan sear or whatever and I dont hear. What happens? Well, I never gave her an ACK. She should have kept repeating herself, but Im not a TCP protocol, sometimes I do things without acking. She could have looked at the list, saw I didnt do it, and repeated. See that double feedback loop in action? Hell, sometimes Ill say ACK and then get distracted! Whoops. But, there is that second form there. The most important aspect of this story is the implicit time aspect of this. If I didnt tell you an ACK within say 30s, Im not giving you one - I either forgot or just did it. If I didnt add it to the list in say 5 minutes - I either never got the message or I dropped it. Hey, Craig, did you add that thing to the shopping list? … This is why I tend to like double/verified feedback loops and timeouts - and I try to put them into most of my interfaces.