Skip to content

Blog

a few thoughts today

So we heard the CVS on 3rd Ave near us will be closed. I was somehow under the impression that that location is a CVS franchise, and was saddened because the Chinese woman owner seemed like a hardworker. Now that I've done my research I am glad at the very least she's just another worker and hopefully they didn't lay her and her staff off.

Also, https://whimsicalkyle.wordpress.com/.

niche tech in film

I just want to post about this sexy beast that is currently situated at the Mono No Aware film lab in Brooklyn, New York. Let me try colorfully recount what Steve Cossman, Mono's director, tells me:

This is 1 of 18 machines in the world. The hardware is handbuilt by one guy and the software is handbuilt by another. Its full cost is \$250,000 but they made one at \$30,000 for Mono. It's got 32TB of hard drive temporarily as the guy will come next week to upgrade that. It's hooked up to a Windows PC that host the processing software, export to the data tower and we've got a Mac hooked up to that for ease of data transport. Scans 8 frame a second at 4K resolution. We drove it to the lab in the middle of a snow squall, and I have to thank a cinematography.com guru for helping set it up for us.

{% include centerImage.html url="/assets/niche_mono2.jpg" desc="what a sexy scanner" title="Xena Film Scanner" %}

{% include centerImage.html url="/assets/niche_mono1.jpg" desc="Xena control module" title="what are all those knobs?" %}

That's it, just niche tech that most will not get to see. Unless they come to Mono No Aware.

scaling with openvpn

You know your company is growing if your openvpn --max-client limit suddenly needs to be made bigger than the default 1024 or else the OpenVPN server suddenly dies and everyone thought it's a firewall issue.

It started around 2PM, our resident SysAdmin-Extraodinaire Dave, was sitting in the seat next to me when he suddenly says out loud "I can't connect to any jbrains". Ray who sits opposite me reached for his keyboard, typed a few things, and confirm "huh, I can't connect to any either". Hearing that, the other support engineers started doing the same thing, verifying that they too can't connect to any jbrains.

At this point I should explain what are the jbrains. They're AppCard's brains in the field, deploying one to each merchant that we work with, handles any business logic, watch and alter POS transactions, send back data to our remote servers. From our servers, we can ssh into any of the jbrains to do maintenance, debugging, retrieving logs, etc. In short, they're the most important component in the AppCard hardware system, and now none of us can connect to them.

Almost instantly, the entire Ops team was roused into a flurry of activity. The Ops Manager didn't notice at first (he's very plugged in), not until the first of the Account Manager started pinging him on Slack, and then physically walked over to ask what's going on. Dave has left only one message on the prods alert channel saying: "jbrains are down. looking into". Within 10 minutes, a Google Meet where every engineer that has not yet left work was invited to join (in Israel it was dinner time). The CTO couldn't join but he was able to guide debugging through Slack, with some very insightful questions on "are the iptables up?". Dave himself went into the jbrain provisioning room all by himself, for focus I guess, and did his furious typing there.

Are the jbrains down? No, Account Managers are reporting discounts are still being applied. Are the servers configured correctly? Yes, and no one was changing anything recently. Is OpenVPN up? yes, we have multiple and none died, and failover processes would have been triggered. Can we connect to jbrains we have in the office? No, well, yes, but not through our VPN servers. Is it AWS? No, we can get into the servers and we can connect to the jbrain through network IP. What do you see when you connect to the jbrain directly and look at its log? it says they cannot reach the VPN server. Can't reach or can't connect? Can't reach, "no route to host". Can you connect to the VPN server? Yes, lemme attempt restart of servers. This reminds me of something, is iptable on? No, it's not on, wait, it's only disabled, not stopped, and the server restart restarted iptable. Disabling now.

It was like the sunrise. Green numbers started popping up on all the dashboards, jbrains flood the connections, logs started sprinting, the database suddenly see a spike in CPU and memory utilization due to backlog of transactions needed processing. For a moment, things were alright. People could breathe and ask "what happened? why did it happen?". Only Dave here suddenly ping on slack, "wait, nightmare not over, jbrains connections started dropping".

At this point I phased out, I knew Dave could fix it, and he did, because with the second time having to restart servers, Dave noticed when the number of connections reached 1024, the VPN servers started dropping connections. It was because of the default --max-client on OpenVPN.

Fun.

how do you database?

At my previous job, govtech/tax-tech, the database was just as important as the code. Now what do I mean by that? Mooney explained it best on this exact topic:

Given how much thought and effort goes into source code control and change management at many of these same companies, it is confusing and a little unsettling that so much less progress has been made on the database change management front. Many developers can give you a 15 minute explanation of their source code strategy, why they are doing certain things and referencing books and blog posts to support their approach, but when it comes to database changes it is usually just an ad-hoc system that has evolved over time and everyone is a little bit ashamed of it.

I believe the quote above is true. Admittedly, I'm only a 1+ YOE software engineer, but having jumped ship from a govtech consultancy to a startup, I find there is a lot to compare between how databases is treated and how this leads to a better developer experience.

What follows is a series of features I found missing at my current place of work.

1. Version Control

Schemas have version control. The system detects any changes made to the schema (In fact, the company never taught you to alter tables with SQL) because you would make table structure changes through the system. Deleting/removing columns, adding or editing comments, addibg/editing indexes (and probably many more), local changes are "synced" with the shared work server, where it assigns a version number for your structure. Migrations from local to testing environments and then, ultimately, to Prod, is simply having the environment point to the right version.

comparison is the thief of joy 2

I feel anxious today. I feel anxious a lot actually. I feel as if I'm not as good as my peers. Or that I'm not achieving as much as them. Lot's of comparisons being made lately. But need to remind myself from time to time that it's all okay. It's okay. Take your time. My existence is but a blip on this world. Who remembers the richest family and people of the 18th century, that's right, not a lot.

I have a good life, a good gf, a good job. I'll be grateful. Not fully satisfied, but I'll always be grateful for what I have.

comparison is the thief of joy

some words by /u/Oberon_Swanson

First off is the quote "You are comparing your behind-the-scenes to everyone else's highlight reel." People bring up this quote a lot in regards to social media, but it actually applies to every interaction you have ever had with anyone else in your entire life.

People don't tell you about all their little failures and they also don't often tell you about their deepest fears, regrets, insecurities, and so on. People will be eager to tell you when they got an awesome new job, they will be not so eager to talk about how they've sent out 100 applications and no responses. And even when they do, it will be at a moment of their choosing, and framed in a way that makes them look not so bad.

Thus, you actually aren't comparing your life to anyone else's. You are comparing your life to what people choose to share about theirs. There is also the rather mathematical impossibility of winning a lot of these comparisons, especially when you focus on people who have it better than you. Jonny is better than you at math, Jimmy is taller than you, and Jenny is better than you at gym class. Even if you are actually much better than average in all these respects, even among your peers, that's not what you see when you compare--you just see all the ways people are better than you. So it's just lose, lose, lose, everywhere you look. And even when you are the best around instead of feeling good you just stop caring about that and focus your attention on something else where you're not.

Another thing to consider is that each person is different. Everybody is so different that it is rather insane to even compare two people and say one is better than the other. That guy who's better than you at French? Maybe his mom watched a lot of french movies and so has he. That guy who beats everyone in basketball--he goes right home and practices, watches basketball, and he's been playing since before you were in kindergarten. Did you do those things? No? Would you trade your life and past for one filled with playing basketball? No? Then why are you feeling so bad that he is better at basketball than you?

I find it is still hard to stop these comparative thoughts from cropping up, but you just have to know they are useless and dismiss them. Don't chastise yourself when you have these thoughts, they are natural. Focus on your own goals and your own life and go after what you want. Take care of the people you care abut. Stand up for what you believe in. Try to leave the world a better place than when you found it. Chase your dreams. That's all that matters. Be happy there are other people around you with good qualities, or people who are better than you at things you want to be the best at. Would you want to have your current level of skill and be the best in the world? No.

It is true that some people will have advantages you do not have. You and the prettiest girl in the school both want to become actresses. It's not going to be as easy for you. You want to start your own business. It's not going to be as easy for you as it will be for the trust fund kid whose dad has money and connections in the industry you want to be in. You want to go to Harvard. It's not going to be as easy for you than it is for the kids whose parents are alumni, and paid for tutoring and violin lessons and sent him to study abroad over summer while you worked a shitty job just to save up for tuition. Life is not fair. This will be shoved in your face over and over again. You can do everything right and still fail where another person could have done everything wrong and succeeded. You could eat healthy and exercise and be diagnosed with brain cancer at 19 while a 110 year old drinks and smokes every day.

You must seek out the good in your life, and the good you can do, and focus on it. You are never gonna be anybody else but you. Play the hand you were dealt as best as you can. If that means you have to go to extremes to live the life you want, then go to those extremes. Many of the people you envy have gone to those extremes already without showing it. You can either spend your life whining to yourself you weren't born a supermodel billionaire pro athelete supergenius ruling monarch, or you can do what actually matters to you.

nothing else to be done

After 25 years of career, I still have to see an organization where things are so perfect that: - no refactoring is needed - no additional documentation is useful, it's all there shiny and beautiful. And it updates itself nightly. - logging/monitoring/diagnostic tools are perfect - builds are so fast that you wonder if you did press enter - all necessary linters are configured and used - everything has unit tests - and integration tests - and there's enough time for exploring alternative technologies for future development - and enough time for contributing feature/fixes upstream for the open source things you use - and you cannot build tools to answer asks from customers even faster So yes, you may not get official tickets assigned to you, but it doesn't mean there's nothing else to be done. Perceiving that need is the first step for moving from junior to more senior role, acting on that need is the second step.

Now, depending on the country you're in, social norms may make you unpopular among co-workers and managers alike if you move too much, so there's that.

Beautiful lessons by /u/mavvam

the shopping cart

The shopping cart is the ultimate litmus test for whether a person is capable of self-governing. To return the shopping cart is an easy, convenient task and one which we all recognize as the correct, appropriate thing to do. To return the shopping cart is objectively right. There are no situations other than dire emergencies in which a person is not able to return their cart. Simultaneously, it is not illegal to abandon your shopping cart. Therefore the shopping cart presents itself as the apex example of whether a person will do what is right without being forced to do it.

No one will punish you for not returning the shopping cart, no one will fine you, or kill you for not returning the shopping cart, you gain nothing by returning the shopping cart. You must return the shopping cart out of the goodness of your own heart. You must return the shopping cart because it is the right thing to do. Because it is correct.

A person who is unable to do this is no better than an animal, an absolute savage who can only be made to do what is right by threatening them with a law and the force that stands behind it.

The Shopping Cart is what determines whether a person is a good or bad member of society.

As To Minh Son says: "4chan shit"

god laughs and decides

Darwin spends five billion years optimizing your genes for reproduction, and God laughs and decides that whether or not you mate will depend on which weird parties you go to, or whatever. - some user on /r/TheMotte