I’ll give an example. At my previous company there was a program where you basically select a start date, select an end date, select the system and press a button and it reaches out to a database and pulls all the data following that matches those parameters. The horrors of this were 1. The queries were hard coded.
-
They were stored in a configuration file, in xml format.
-
The queries were not 1 entry. It was 4, a start, the part between start date and end date, the part between end date and system and then the end part. All of these were then concatenated in the program intermixed with variables.
-
This was then sent to the server as pure sql, no orm.
-
Here’s my favorite part. You obviously don’t want anyone modifying the configuration file so they encrypted it. Now I know what you’re thinking at some point you probably will need to modify or add to the configuration so you store an unencrypted version in a secure location. Nope! The program had the ability to encrypt and decrypt but there were no visible buttons to access those functions. The program was written in winforms. You had to open the program in visual studio, manually expand the size of the window(locked size in regular use) and that shows the buttons. Now run the program in debug. Press the decrypt button. DO NOT EXIT THE PROGRAM! Edit the file in a text editor. Save file. Press the encrypt button. Copy the encrypted file to any other location on your computer. Close the program. Manually email the encrypted file to anybody using the file.
This one is funny because it 100% still exists somewhere, but I haven’t had the chance to verify it again.
Okay so basically its a data recorder box (ex: brainbox) that connects to a bunch of industrial sensors and sends the data over the network with your preferred method.
Builtin firmware gives you an HTTP webui to login and configure the device, with a user # and password.
I think the user itself had a builtin default admin which was #0, which everyone uses since there wasn’t really much use for other users.
Anyway, I was looking at the small JS code for the webui and noticed it had an MD5 hashing code that was very detailed with comments. It carefully laid out each operation, and explained each step to generate a hash, and then even why hashes should be used for passwords.
Here’s the kicker: It was all client side JS, so the login page would take your password, hash it, and then send the hash over plaintext HTTP POST to the server, where it would be authenticated.
Meaning you could just mitm the connection to grab the hash, and then login with the hash.
I sat there for like 10 minutes looking at the request over and over again. Like someone was smart enough to think “hey let’s use password hashing to keep this secure” and then proceeded to use it in the compleltly wrong way. And not even part of like a challenge/handshake where the server gives you a token to hash with. Just straight up MD5(password).
It was so funny because there were like a hundred of these on a network, so getting a valid hash was laughably easy.
I never got to check if this was fixed in a newer firmware version.
#4 is a good thing. ORMs do not make queries better or safer, they make them easier for devs that don’t learn SQL or safe calls. In some cases, they have been shown to cause slowdowns.
A bit late to the party on this one, but Facepunch just opensourced a bunch of their code, I nominate that.
I don’t recognize the name, what dud facepunch make?
Garry’s Mod. Rust (the game, not the programming language).
Ahh, ok yeah makes sense
Long time ago, but by far the worst for me was when I inherited some code that a previous programmer had done. Every variable was a breakfast item. So if biscuit>bacon then scrambledeggs=10. Shit like that. It was a nightmare and luckily I only had to deal with it infrequently.
Why do people do stuff like this, is the logic not difficult enough to follow on it’s own without a secondary definition table to consult!? Fucking hell.
secondary definition breakfast table
Had a programmer like this when I was still an apprentice. He was so full of himself. Was originally a Java programmer but had to program in PHP because that was what ran on the server. I never found out why he couldn’t just put Java on the server. We had full control.
All his variables were first names. Like
$klausand$grobi. Because he was afraid of clashing with reserved keywords. The thing is, in PHP all variables begin withexactly to prevent this issue. So he brought that habit over from Java which was far superior and not such a “Mickey Mouse language”.I mean, he wasn’t totally wrong, especially back then PHP was awful. But he surrounded every function with
and(PHP was designed to be combined with HTML output outside of these tags) and had plenty of whitespace between them and couldn’t fathom why all his html files had huge swaths of whitespace at the start.His way of preventing SQL injection was to look for SQL keywords in user input and then throwing an error in the log files.
I don’t know what’s worse… That program or that you put biscuits greater than bacon…
Actually I think the greater crime is biscuits being greater than bacon
but BiscuitTop + Bacon + Eggs + BiscuitBottom is definitely better than biscuit, or bacon or eggs.
True, all are good.
bacon++;
unless they’re bacon biscuits
I don’t know how old you are but when I was in school, this was just going out of style. They saw this as job security. If you’re the only one who can work on the code, then they won’t fire you
Oh god, that’s worse than I’ve seen where a SQL query joining 10 tables aliased all of the tables as a, b, c, d, e, f, g, h, i, j.
It was a mess, and as a new dev on the project, trying to figure out which where clause was for which table and how things worked was a fucking nightmare. Trying to keep a dictionary of letters to real table names in your head as you looked at the query was very taxing. In the end, I just fixed it all to stop using aliases. Or to use short abbreviations.
Here’s a mock example:
SELECT j.delivery_eta, c.cat_desc, a.part_number, h.region_label, f.wh_loc, e.emp_last, g.state_flag, b.mfg_title, i.ship_track_code, d.order_sum, a.created_on, j.last_scanned_at, e.emp_first, c.cat_code, g.state_level FROM parts AS a INNER JOIN manufacturers AS b ON a.manufacturers_id = b.id INNER JOIN categories AS c ON a.categories_id = c.id INNER JOIN orders AS d ON a.orders_id = d.id INNER JOIN employees AS e ON d.employees_id = e.id INNER JOIN warehouses AS f ON a.warehouses_id = f.id INNER JOIN inv_state AS g ON a.inv_state_id = g.id INNER JOIN regions AS h ON f.regions_id = h.id INNER JOIN shipments AS i ON d.shipments_id = i.id INNER JOIN logistics AS j ON i.logistics_id = j.id WHERE (b.mfg_title LIKE '%Corp%' OR b.mfg_title LIKE '%Global%') AND c.cat_desc NOT IN ('Unknown', 'None', 'Legacy') AND (d.order_sum > 1000 OR d.order_sum BETWEEN 250 AND 275) AND e.emp_last ILIKE '%berg' AND (f.wh_loc IN ('A1', 'Z9', 'M3') OR f.wh_loc IS NULL) AND g.state_flag IN ('ACT', 'PENDING') AND h.region_label NOT LIKE 'EXT-%' AND (i.ship_track_code IS NOT NULL AND i.ship_track_code <> '') AND (j.delivery_eta < NOW() + INTERVAL '90 days' OR j.last_scanned_at IS NULL) AND (a.part_number ~ '^[A-Z0-9]+$' OR a.part_number IS NULL) AND ( (c.cat_code = 'X1' AND g.state_level > 2) OR (e.emp_first ILIKE 'J%' AND d.orders_id IS NOT NULL) );That’s how mainframe programmers at my workplace do SQL. I think they do it due to long table and field names and narrow mainframe COBOL files
I have a friend that uses swear words 🤷♂️
One time, I had to request firewall access for a machine we were deploying to, and they had an Excel sheet to fill in your request. Not great, I figured, but whatever.
Then I asked who to send the Excel file to and they told me to open a pull request against a Git repo.
And then, with full pride, the guy tells me that they have an Ansible script, which reads the Excel files during deployment and rolls out the firewall rules as specified.In effect, this meant:
- Of course, I had specified the values in the wrong format. It was just plaintext fields in that Excel, with no hint as to how to format them.
- We did have to go back and forth a few times, because their deployment would fail from the wrong format.
- Every time I changed something, they had to check that I’m not giving myself overly broad access. And because it’s an Excel, they can’t really look at the diff. Every time, they have to open it and then maybe use the Excel version history to know what changed? I have no idea how they actually made that workable.
Yeah, the whole time I was thinking, please just let me edit an Ansible inventory file instead. I get that they have non-technical users, but believe it or not, it does not actually make it simpler, if you expose the same technical fields in a spreadsheet and then still use a pull request workflow and everything…
The corporate world runs on excel, never the best option, but everyone knows it so…
Yep; I’ve seen excel files that at like 10MB because it’s a database in Excel
Try a few Gigabytes. I worked on site IT support for a year, we had to max out memory on a workstation because the company database was a, about 3GB, Excel file. It took minutes to open and barely worked, crashing frequently.
I’m so sorry.
I’ve seen a hotel booking made in Excel 🤷♂️
When I was interning in a cellular biology lab, I took their chemical inventory and converted it from excel to access. Complete with forms and reports. Spent some time training the permanent staff how to use it, explained how it was much more efficient than excel.
I don’t think they bought into it, but I tried.
So, this is completely off topic, but some of the comments here reminded me of it:
An elderly family friend was spending a lot of her time using Photoshop to make whimsy collages and stuff to give as gifts to friends and family.
I discovered that when she wanted to add text to an image, she would type it out in Microsoft Word, print it, scan the printed page, then overlay the resulting image over the background with a 50% opacity.
I showed her the type tool in Photoshop and it blew her mind.I am simultaneously horrified that she didn’t do any research to see if she could insert text into the image and incredibly impressed at her problem solving skills. Honestly, the more I think about it, the more I lean towards impressed; good on her!
Haha that’s so dumb. She could’ve just taken a screenshot!
I showed her the type tool in Photoshop and it blew her mind.
Or well. That.
Aw really wholesome actually. Some libraries in my area have senior friendly editing classes, I think it’s becoming more popular. Good looking out for them!
Photoshop is amazing. That said you kinda need to take a course in it to use 80% of the functionality.
And almost always, if you are not a professional, that could be done with any image editing program.
Floats for currency in a payments platform.
The system will happily take a transaction for $121.765, and every so often there’s a dispute because one report ran it through round() and another through floor().
Presumably every so often there’s a dispute because 0 + (0.3 + 0.3 + 0.3) - 0.3 - 0.3 - 0.3 is not equal to 0 (in floating point arithmetic).
And not the easy way of using (long) integers to count the cents.
Lmao.
Using floats for nearly anything in a finance platform should be grounds for immediate dismissal.
Did either of those do banker’s rounding?
Round is the safest way of using decimals for money as it corrects 10.499999999 (decimal fractions can’t be stored precisely in floats as binary can’t precisely represent all 2 digit decimals) to 10.50, where floor would take it to 10.49
It is safer to count in cents and have a policy to handle fractions of cents from divisions
deleted by creator
I found code that calculated a single column in an HTML table. It was “last record created on”.
The algorithm was basically:
foreach account group foreach account in each account group foreach record in account.records if record.date > maxdate max = maxdateIt basically loaded every database record (the basic unit of record in this DATA COLLECTION SYSTEM) to find the newest one.
Customers couldn’t understand why the page took a minute to load.
It was easily replaced with a SQL query to get the max and it dropped down to a few ms.
The code was so hilariously stupid I left it commented out in the code so future developers could understand who built what they are maintaining.
I’ve had legacy systems that would encrypt user passwords, but also save the password confirmation field in plain text. There was a multitenent application that would allow front end clients to query across any table for any tenant, if you knew how to change a header. Oh and an API I discovered that would validate using “contains” for a pre-shared secret key. Basically if the secret key was “azh+37ukg”, you could send any single individual character like “z” and it would accept the request.
Shits focked out here, mate.
Rules I’ve learned from software engineering for almost 2 decades.
- Never roll your own ORM
- Never roll your own Auth
No matter what you think, someone else did it better. Trying to do either of those outside of a hobby environment is pure hubris. “But I can do it better” - no you fucking can’t. I have wasted much much more time debugging shitty “home grown” ORM solutions that clearly some dev just was bored and did than I have spent figuring out the quirks of whatever one I’m using. Same goes for auth. Just learn it.
Never roll your own ORM
I’ve done this. Probably 10 years ago. Even today, I maintain the same application that has the ORM in it that I designed. If I could go back in time and do something else, I’d do the same thing again. Honest to god. For my use case, I feel it was warranted. It was risky, but it worked out surprisingly well.
If I could go back in time and do something else, I’d do the same thing again.
So many questions here. What language? What’s the database? How many years of experience do you have in industry?
We are the worse evaluators of our own code, some of us are down right terrible but most people regret less significant code choices in a matter of months. The fact that you still think it is good 10 years later is a massive red flag.
Java, Postgres mostly but also LDAP and random in-house-written RESTful services, almost 20 years.
- The objects we store in the Postgres database are very “hierarchical” in nature, with one top-level object and lots of child/grandchild/great-grandchild objects. (We asked for a Mongo database but the infra team at the time said "make do with Postgres.)
- As I mentioned, some of that hierarchy is in LDAP or RESTful services, not in Postgres, so we needed something capable of dealing with multiple storage backends that would stitch the objects together as necessary. So the “ORM” needed to have backends for multiple backend systems.
- We knew clients would need a vast number of different queries. So we made a RESTful endpoint that gave the full power of the ORM to (authorized) clients. If they needed different data, we’d be like “change your query like this” and they didn’t have to wait on us.
- Early in the project, we consciously designed an extensible JSON representation of our hierarchical objects. That is what’s returned from the aforementioned RESTful endpoint.
- However, we also created a “shortcuts” system to allow us to “balance” how much of the logic lived on the server vs in the client. (It can mix and match. Like “apply this shortcut, but also filter this way and paginate” or whatever.)
- We made the API of the ORM such that it could both be used to query from the database/LDAP/RESTful systems, or be used as a client SDK for the aforementioned RESTful query endpoint that the application exposed.
- It’s both “more than an ORM” (querying from non-database sort of backends) and not fully an ORM (read only, doesn’t handle schema evolution.) But it’s fair to say it’s more “an ORM” than “not an ORM”.
- The implementation of the Postgres backend part of it is heavily inspired by Django’s ORM.
We couldn’t have pressed Hibernate into this use case. It doesn’t really deal with hierarchical data and sure as hell doesn’t know how to query from LDAP. I don’t know that anything existed at the time (nor am I sure anything exists now) that would fulfill our use case.
And the alternative to what we built was a massive, unmaintainable DAO with ridiculous numbers of individual queries in it that would have to be modified or added to endlessly every time someone needed to filter a bit differently or whatever.
The SVP over my org keeps wanting to design his own RBAC/Auth/IAM system.
We have entra, auth0, and keycloak.
The reason he wants it is he doesn’t want secrets to setup auth. Like that’s how it (mostly) works, sunshine.
The reason he wants it is he doesn’t want secrets to setup auth
Yes I’d like auth without the auth please
I never fuck with auth. If I can throw it up the stack I’ll do it as much as I can. When I can’t I find an open source solution and Im sure I still misconfigure it
I kinda rolled my own ORM, it was just a glorified class to SQL parameter converter that is then passed onto some SQL code. Then there was a thingy that marshals the result. Using table value functions in SQL standardises the result so any
select *just works. It was around 50-100 lines total.
I have to ask, if it’s only contains wouldn’t you get a ton of collisions?
Expecting an apartment manager to know what a api header was nevermind how to change it is probably not likely. Security hole to be sure though.
The secrets themselves were basically guids, they had quite a lot of characters. If sent MORE than 1 character, pretty low chance they would clash. But those long guids also covered a lot of letters and number - it wasn’t terribly difficult to find one single character that cleared authorization reliably.
And maybe you’re joking lol, but multitenant meaning multiple businesses/customers using the same application stored in the same database. If Bob’s construction wanted to spy on Jim’s contracting, they’d just need to know the right header to send and could get whatever they wanted from the other customer partitions. User access should of course be limited to their own assigned partitions.
Oh, ok I interpreted multitenant wrong. I was thinking it was like a apartment complex so you have like a manager and a sales person with access and that’s it. Still a valid security risk but not as severe as what you are saying now.
Sorry for confusion
I once saw an application that would encrypt (not hash, encrypt) passwords but then when a user was logging in, they’d encrypt the password candidate and then compare the cipher texts to see if they were the same. This was using 3des, so no IV.
A registration form and backend that would return the error “please choose more unique password” if you choose a password that was already stored (in plain text) in the database against another username.
I shit you not.
Create a moderately ok password, hash it, use the hash as your nice unique password, as a private joke for when the database leaks and yours is the only password that’s hashed and you start getting spam saying they know your password hunter2 (because they incorrectly dehashed the password) or 2ab96390c7dbe3439de74d0c9b0b1767 (md5 sum of hunter2; because they correctly read it as plain text)
Damn, ‘hunter2’ is taken
All I see is
******
I worked for a mid-sized government entity where we handled PII data. Underneath us were local municipalities who were in charge of sending us that PII so that it could be registered at our level. For PII think licenses, IDs, sensitive stuff for sure.
Most of the municipalities were easy to work with, they did an SFTP drop or used a VPN or something.
A couple though were rural. Very rural, and didn’t have IT departments. They had Martha who works the counter from 1-4pm. Those places were… horrid. We had a special email where they would email us whatever formats they had. Unencrypted, completely open, we couldn’t do anything about it because it was their data and their rules, it was our job to simply accept what they had. We could of course make serious suggestions, point out how horrid this was, but at the end of the day it was their decision. So we had a job to log into an email account every day, check for an email from Martha’s hotmail account, and parse the excel file she used to read out private IDs and license numbers which she manually typed into it.
This was 20 years ago now so dear god I hope their laws improved.
Don’t worry it’s completely different now. Martha retired so now Mary Ellen sends the emails.
Triggered
Maybe this says something about me but I would write a script to get the excel file from the email and then do whatever you did to it.
At what point would it not be financially viable to just a write a program for the entry on your budget and then send it to the rural places?
above my paygrade that last question. As for the first, we did automatically pull it out of the email, but it was sent to us manually
Sorry I used to do gross margin improvement and that just seems ripe for cost cutting. Spend 100K one time, save $75k every year.
Was the email atleast encrypted?
Oh, oh no no no it wasn’t
Just old style SMTP. That’s nearly extinct on the wild internet now
It never is.
Java webapp. Customer facing. E-commerce application, so in PCI scope and dealt with credit card info and such.
There was one specific cookie that stored some site-wide preference for the customer. (Why not just put that preference in the database associated with the user? Because that would make too much sense is why.)
But the way they encoded the data to go into the cookie? Take the data, use the Java serialization framework (which is like Python’s “Pickle” or Go’s “Gob”) to turn that into a string. But that string has binary data in it and raw binary data is kindof weird to put in a cookie, so you base64 encode the result. (The base64 encoding was the only sane step in the whole process.) Then you do the reverse when you receive the cookie back from the browser. (And no, there was no signature check or anything.)
The thing about the Java serialization framework, though is that decoding back into Java objects runs arbitrary object constructors and such. As in, arbitrary code execution. And there’s no checking in the deserialization part of the Java serialization framework until your code tries to cast the object to whatever type you’re expecting. And by that point, the arbitrary code execution has already happened. In short, this left a gaping vulnerability that could easily have been used to extremely ill effect, like a payment information breach or some such.
So all a malicious user had to do to run arbitrary code on our application server was serialize something, base64 encode it, and then send it to our servers as a cookie value. (Insert nail biting here.)
When we found out that there was a severe vulnerability, I got the task of closing the hole. But the existing cookies had to continue to be honored. The boss wasn’t ok with just not honoring the old cookies and developing a new cookie format that didn’t involve the Java serialization framework.
So I went and learned enough about the internal workings of how the Java serialization framework turned a Java value into a binary blob to write custom code that worked for only the subset of the Java serialization format that we absolutely needed for this use case and no more. And my custom code did not allow for arbitrary code execution. It was weird and gross and I made sure to leave a great big comment talking about why we’d do such a thing. But it closed the vulnerability while still honoring all the existing cookies, making it so that customers didn’t lose the preference they’d set. I was proud of it, even though it was weird and gross.
The value that was serialized to put into the cookie? A single Java int. Not a big POJO of any sort. Just a single solitary integer. They could just as well have “serialized” it using base-10 rather than using the Java serialization framework plus base64.
Some minecraft mods had/have a similar problem. They use javas serialization stuff for sending stuff between client and server. There is mod that partially fixes this by only allowing whitelisted classes to be deserialized.
Gson in the corner murdered:
Wow… If you can answer was this like a single company or were you selling the service to other companies?
If selling to multiple companies did you offer a “new” version and a “security patch” for the old or just made everyone use your implementation?
This was a developed-in-house e-commerce web application at a major e-retailer. So fortunately that monstrosity of a cookie-handling mess was only ever used by one company.
You know what, though? Talking about this reminds me of another story about the same e-commerce application.
After a customer placed an order on this e-commerce site, the company’s fraud department had to evaluate the order to make sure it wasn’t fraudulently placed. (As in, with a credit card not owned or authorized for use by the purchaser.) Once that was done, the order had to be communicated to a worker at the warehouse so they could pack the right items into a box, put on a shipping label, and set the box aside to be picked up by the UPS truck which would come once a day near the end of the day.
The application used by the fraud department and the application that displayed new orders to warehouse workers was one and the same application. Whether a user had fraud-evaluating powers or pack-items-in-boxes powers just depended on what permissions their particular user had. (That may have been decided by LDAP groups. I don’t remember for sure.)
Meanwhile, the e-commerce site offered gift cards for sale online. The gift card would be shipped to the customer. And there was a box where you could write a message associated with the gift card. So, for instance, someone could buy a gift card to be sent to their nephew’s address or whatever and include a little note like “Happy Birthday. Don’t spend it all at once.” or whatever. And the fraud/pick-and-pack application would display all details of the order including any messages associated with the gift cards.
Well, I found a stored cross-site scripting vulnerability where if you put
<script>...</script>tags with some JavaScript in the gift card message box and completed the order, the JavaScript would execute any time someone viewed the details page for the order in the fraud/pick-and-pack application. And of course, the JavaScript could do within that application just about anything the user could do with their given permissions.The main danger was that a malicious actor with sufficient knowledge of how our fraud application worked could place an order fraudulently with someone else’s credit card and include in the order a gift card with a malicious JavaScript payload in the message box, and then that malicious JavaScript could automatically mark the order “a-ok, no fraud here” when a fraud department worker loaded the order details page, letting the order be fulfilled without any actual fraud review.
The fix was pretty simple. Just stick a
<c:out>...</c:out>in the appropriate place in the fraud/pick-and-pack application code. But it was an interesting example of a vulnerability in a not-customer-facing application that could none-the-less be exploited by any public customer/user without any particular special access.If you’re interested in one more interesting story about the same e-commerce application, see this comment I made a while ago.
That is interesting, I dealt with purely internal software so never considered that.
A program that HR had built so that all employees could they their payment receipts online
The username was the companies’ email address, the password was a government personal id code that you can lookup online, a don’t change, and you can’t update the password to something else.
So I told the director of HR this was a bad idea. She told me I was overreacting until I showed her her own receipt, then she finally understood that this is a really fucking bad idea.
Okay, so now she out me in charge of debugging that program.
So I setup a meeting with the director of the company they hired, he came by with the developer: a 21 yo girl who I think hadn’t finished college yet. Great start! Apparently it was her idea to do the authentication like that so that explains a few things.
So we dive in to the code.
First of all, the “passwords” were stored in blank, no hashing, no encryption, nothing. That wasn’t the worst.
For the authentication she made a single query to check if the user email existed. Of that was true, then step two was a second query to see if the password existed. If that were true, the email had been authenticated.
So let’s say, hypothetically, that they had actual passwords that people could change… I could still login with the email from anyone, and then use MY OWN password to authenticate.
This just blew my mind so hard that I don’t think I ever fully recovered, I still need treatment. The stupidity hurts
I wouldnt blame that on stupidity as much as on ignorance and naivety. Many people simply don’t think about anybody deliberately misusing their design. The idea that somebody could even want to access somebody elses receipts didn’t occur to them. And if they were still doing their studies they might not have known that you can “combine” SQL queries and ask for two things at once.
I don’t blame the girl, but whoever chose her to design a system with sensitive information.
I don’t blame a girl for doing a job that lands her food on the table. I blame the guy employing her because she’s the cheapest option
Having said that, this design was so bad that she should not have been doing any of this. If you don’t know that SQL allows you to select multiple columns then by all means, do a tutorial, it’s not that hard.
If you don’t even know what encryption is, that passwords need hashing and what not, then you should really question what you’re doing
OPs question was about the worst code I’ve seen, that was the worst I’ve seen
If you don’t even know what encryption is, that passwords need hashing and what not, then you should really question what you’re doing
I agree with your point, but I would phrase it more generally: when we’re assigned a task in a problem space we are unfamiliar with, we should always take some time to research that space before designing our solution.
After all, if we don’t know what encryption or password hashing are, how could we know that we need to learn about them first? But spending just a couple hours one morning reading about password and authentication management would have given the developer a good sense of best practices.
So she either, A) didn’t think to familiarize herself with a new topic prior to working on it, or B) did read about it and ignored general industry guidance. Both of those options are more problematic to me than simply not knowing specific things. Those are process problems that need to be addressed to build her skills as a developer.
But ultimately, in my opinion, this is really all the fault of the cheapass director who didn’t want to pay any experienced professionals to handle the task.
It wouldn’t take much google-fu to get a worked example of good authentication in whatever language. She can’t have tried, she must have just gone “programming 104 covered how to SQL, I can use that”
First of all, lack of ORM isn’t bad. It’s not a good or bad thing to use them out not use them. What’s bad is not sanitizing your query inputs and you don’t need an ORM to do that.
I think the worst thing I’ve seen is previous devs not realize there’s a cost to opening a DB connection. Especially back when DBs were on spinning rust. So the report page that ran one query to get the all the items to report on, then for each row ran another individual query to get that row’s details was probably one of the slowest reports I’ve ever seen. Every DB round trip was at minimum 0.1 seconds just to open the connection, run the query, send back the data, then close the connection. So 10 rows per second could be returned. Thousands of rows per page has people waiting several minutes, and tying up our app server. A quick refactor to run 2 queries instead of hundreds to thousands and I was a hero for 10 min till everyone forgot how bad it was before I fixed it.
It’s the round trips that kill you.
Oracle drivers for .NET are fun. Have a user client application which uses quite a lot of data, but a few thousand rows are fetched some queries. It’s way too slow for any larger query, turns out for the batch query kind of work we do, the default FetchSize for Oracle is just a performance killer. Just throw it to 128 MB and it doesn’t really hurt at all.
Worst thing i’ve seen though, apart from the 150 line long dynamic sql stored in our database, was probably a page in our program that loaded about 150 rows from the database. Normally we do create a new connection for each query, but it’s fine since Oracle has a connection pool. Whatever millisecond is trumped by the round trip. But imagine a UI so badly written, it did 4 separate database queries for EACH row it loaded into the UI list. Useless things like fetching a new ID for this row in case it is changed, reading some data for the row i think, and more. Thing took a solid minute to load. There was so many bad patterns in that page that even during the PR for improving the speed it was just dealing with a mess because you couldn’t just rewrite the entire thing, so they had to make it work within the constraints. Horrible thing to work with.
Our CFO’s social security number, contact info, and just about everything you’d need to impersonate them inside a random shell script that was being passed around like drugs at a party for anyone to use. Oh and it had an API key to our payments processor hard coded into it.
That was the tip of the iceberg of how bad the systems were at the company. All of these are from the same company:
- A fintech based company with no billing team
- An event system that didn’t event
- A permissions system that didn’t administer permissions
- A local cache for authentication sessions. Which means that requests would intermittently fail auth because the session was only on one replica. If you hit any of the other ones, you’d get an unauthenticated error
- A metrics collection system that silently lost 90% of it’s data
- Constant outages due to poorly designed and implemented systems (and lack of metrics… hmmm)
- Everything when I joined was a single gigantic monolith that was so poorly implemented they had to run at least 3 different versions of it in different modes to serve different use cases (why the fuck did you make it a monolith then?!)
- The subscriptions system was something like 20 or 30 database tables. And they were polymorphic. No one could touch the system without it breaking or that person declaring failure, which leads me to …
- A database schema with over 350 tables, many of which were join tables that should have been on the original table (fuck you scala/java for the limitations to the number of fields you can have in a case class). Yes you read that right. Table A joined to table B just to fill in some extra data that was 1:1 with table A. Repeat that a few dozen times
- History tables. Not separate from the original table, but a table that contained the entire history of a given piece of data. The worst example was with those extraneous join tables I just mentioned. If you went and changed a toggle from true to false to true to false, you’d have 4 records in the same table. One for each of those small changes. You’d have to constantly try to figure out what the ‘latest’ version of the data was. Now try joining 5 tables together, all of them in this pattern.
- Scala… I could go on a tirade about how bad scala is but needless to say, how many different error handling mechanisms are there? Scala decided to mix all of them together in a blender and use them all together. Scala is just two white paper languages in a trenchcoat. Never use it in a production system
- A dashboard for “specialists” that was so easy to overwhelm that you could do it by breathing on it due to the LACK of events that it needed
- Passwords stored in plain text (admittedly this was in the systems of the company we acquired while I was there). Doesn’t matter if they were actually <insert algorithm here>, they were visible in a dashboard accessible by employees. Might as well have been plain text
- A payments system that leaked it’s state into a huge part of the rest of the system. The system ended up being bifurcated across two systems, I was brought in to try to clean up some of the mess after only a couple of months. I desperately tried to get some help because I couldn’t do it solo. They ended up giving me the worst engineer I’ve ever worked with in my 15 year career, and I’ve seen some bad engineers. Looking back, I’m reasonably confident he was shoving our codebase into an AI system (before it was approved/secured, so who knows who had access) and not capable of making changes himself. I could make several posts about this system on its own
- I could go on but I’ll cut it off there
