Wow.
I’ve been processing a couple of billion rows of data on my machine, the fans didn’t even come on. WTF are they teaching “experts” these days, or has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I’ve been seeing with younger programmers; they’ve siloed themselves so specifically into whatever programming they “specialize” in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they’ve grown up with systems that “just work”. Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don’t even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
My experience does not reflect yours. Computer Architecture, Discrete Math (logic gate math), and Operating System Concepts were all required classes in my CS degree from just a few years ago.
Honestly that’s good to hear. I’ve run into some devs who are completely mystified on how to connect to a remote database and couldn’t tell a socket from sandwich.
In my degree, we had to write kernel mods and device drivers
Can I have my socket with rye. I like rye.
My CS degree had a hardware/IT support class, but A) it was entirely simulation based. We never touched any actual hardware. We “built” PC’s or identified physical issues in 3d sim software, set up RAID arrays in software, etc. B) it was super hand holdy and you only ever go over a problem once, so nothing on the class has stuck. I know much more from having built, troubleshot and maintained my own computers and network than I ever learned from that class, then learned more by doing in an actual IT support position before becoming an engineer.
I mean to be fair the sheer amount of material most university engineering programs require these days makes spending significant time on specific problems almost impossible. They try to shove so much theory into your head they lose track of practical implementation. Basically everyone I went to school with complained about the lack of practical application relative to theory, and I studied mechanical engineering which is theoretically and literally chiefly concerned with hardware.
You don’t teach a farmer how an internal combustion engine works. Computers are tools to software engineers. What they need to know is how to operate them, not how to maintain them.
I’m not sure how well that analogy holds up. Farmers are usually pretty well versed in mechanical systems. To the point that now that John Deere has been screwing them over on right to repair that some farmers are even becoming versed in computer programming so they can flash the firmware on their tractors.
Farmers can build a two stroke from parts.
No, but if a farmer’s tractor is overheating (as in the gard drive conparison), I’m sure they could diagnose it.
I never said that it was impossible for a farmer to learn things outside their immediate field. Just like computer programmers often have knowledge of hardware and the general technology stack.
My point, to make it explicit to a few of the illiterates who’ve replied to my comment so far, is that it is not necessary to teach a web developer how a goddamn CPU works. They can gain nothing from that knowledge because there are at least 3 levels of abstraction between JavaScript and assembly.
And my point is that the example you used does not make the point you are trying to make, but rather the opposite. I get what you’re saying, it just doesn’t apply to farmers and mechanics.
Operating your tools and being able to maintain and repair your tools are the unequivocally essential skills for everyone in every single industry.
If you can’t, you are not a professional.
The concepts of machine logic, registers/lookups/etc are essential for every programmer. If you don’t have a clear idea about how the simplest CPU functions, you don’t have any basis of understanding the abstractions in front of you, scripting in JS. Not a professional.
no but a web dev should have some knowledge basis on what the ever living fuck their AIDs code fuelled by nothing but the cheapest source of caffeine and brain damage they have even does.
This is the entire reason why half of the internet is just broken, stupid developers who don’t know how anything works, but know how to code, making dogshit implementations of anything and everything they can get their hands on.
It doesn’t matter that the learning is segmented, you should STILL be learning about computer hardware and it’s architectural choices, it’s literally the reason why programming languages work the way that they do.
No, not really. Programming requires understanding of the underlying hardware, at least to a certain extent. Otherwise performance issues will look like dark magic and optimizing anything would be impossible.
Where do you start debugging if something goes wrong with the software and your information level is this low/ do you look at network stats? CPU utilization, paging/swapping? Is the hard disk bandwidth the bottleneck? Without at least some passable understanding of a computer architecture people like this just throw up their hands, or throw whatever tricks they know at the wall and see what sticks.
A lot of farmers are learning how they work cause the companies that sell them the equipment keep fucking them over. I would argue that farmers nowadays needs to know how that works along with basic programming to get past the anti-consumer bullshit companies put in to make it nigh impossible to fix things yourself.
doesnt matter if you know how to program, john deere is just going to put some autistic encryption and ID locking on their shit, what needs to happen is for john deere to stop fucking doing this.
Most tractors are walking computers anyway, farmers are genuinely the most multi talented people you will ever meet in your life.
It is good for the programmer to know how the computer operates, as well.
Horseshit. Computers aren’t tools for a software engineer. Computers are tools to an administrator, an accountant. Computers are the sandbox you are building castles in as a software engineer. If you don’t understand the system upon which you build, its abilities and features, its limitations, it’s dependencies, you are going to make some stupid mistakes.
You need to understand discrete mathematics as a consequence of computer computation. You need to understand parallel processing and threading for muli-core processors. You need to understand networking, package management, security vulnerabilities, etc. from different architectures and protocols. And it ALWAYS helps to understand the very basics of a computer’s functioning, from hardware, CPU architecture, machine code, assembly/low level programming, memory management, etc.
print('Hello, World!) is day one shit for a reason. Programming language and logic is the basics. The real expertise comes from your 3rd and 4th year materials. Databases, architecture, theory of computation, discrete mathematics, networking, operating systems, compilers, etc.
computers are a tool to anybody who uses them?
If you’re using a tool, it goes without saying, you should probably have at the very least, a cursory understanding of it’s function. Lest you injure yourself gravely.
What the fuck
How is he going to fix his tractor? Wait days for John Deere to send somebody? Let the crop rot on the vine?
CS departments were doing poorly, but now they’re putting out farmers? No wonder all these new graduates can’t find a job.
I mean every programmer says they intend to quit and pick up farming. Might as well give them the knowledge to be successful at their late career while they’re at it
Just keep trying to justify your own lack of competency I guess. ¯\_(ツ)_/¯
the only reason farmers are afloat financially is BECAUSE they can rebuild an engine if needed.
Just look at the john deere right to repair shit. It’s literally a huge problem.
You’ve never met a farmer in your life.
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
I’m not asking everyone to be able to become a hardware specialist, but if you can’t even figure out “my computer gets hot” I’m not going to be able to trust anything you do. Identifying a heat issue does not take a rocket surgeon.
If it was an nvme ssd i could almost believe it. Some come with totally underspecced heatsinks
He hired a bunch of 19-25 year old. Not experts
Hey! Thats offensive to 19-25 year olds, there are many who just finished college/university and are more than aware.
They’re just role playing like in movies, with no idea of the consequences.
How on earth is it offensive to say they’re “not experts”? They’re not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
Except they’re not, as you will know their tweet would be false after your first year of any technical (IT oriented) education.
Technical enough to be hired, is all I meant. 🙄
Even then, no. These were all obviously nepotism hires who would not have otherwise qualified.
Meh, some of them won some hackathons and scholarships, it’s pretty clear they’re otherwise at least somewhat bright but they don’t have any relevant domain knowledge.
In other words, the type of person most likely to be prone to hubris and catastrophic failures.
Apologies, if I came over as hostile. I did not get your meaning through text.
First year? That shit is like A+ cert level knowledge or below, and A+ is damn near worthless. They would know that in the first few hours of a study guide
I was being generous when you consider the people in school who somehow pass, even when they don’t know a thing 🥲
Your original comment was ambiguous as to if being an “expert” and “being 19-25” are mutually exclusive.
lol a 19-21 yo isnt going to have a degree lol,
If they went into uni straight out of high school, they could. A lot of Bachelor holders would be around that age, since they start at 18.
There is nothing wrong with being 19-25. There’s something wrong with being wholly incompetent.
There’s not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That’s not Musk’s minions though.
I think it’s important to differentiate incompetence from ignorance. Ignorance is not knowing. Incompetence is not being able to fulfill the requirements for your assigned task. If you cannot fulfill the requirements for your given task, then you should not be given said task.
Bunch of 1337 hax0rs script kiddies who don’t understand anything but they suck elon’s balls or something idk.
These are the type of people that have deleted the French language from their GNU/Linux system.
I’ve read a story on the forbidden website where a “database” was a single table with a single column holding a single row that contained the actual data as a CSV blob. I’m willing to bet the muskies are not beyond such acts of genius.
It’s terrifying that this is plausible.
We call it New Redis!
has Elmo only hired people who claim that they can “wrangle data” and say “yes” ?
There’s two issues going on:
- Elmo’s sociopathic approach to laying people off is public knowledge, and top experts have the luxury of not even applying for his jobs.
- Elmo’s ability to judge engineering talent has likely been wildly exaggerated thanks to how he has successfully bought organizations full of talented people, in the past.
You have to understand that the average Trump voter probably knows everything they know about computers from watching the ‘wacky-zaney hacker with personality issues/quirks’ “hack” into things by tippity tapping their fingies on a keyboard in your average copaganda performance.
This is something those types of people will believe.
You’re on the mark. I’m like Help Desk Level 2, I wouldnt even consider myself an actual wizard. The average person in my office thinks I’m Gandalf. Its scary how much these people dont know. And each one of them is out there on the internet.
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
My go-to tool of late is
duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.
Yes, his Boy Harem.
Unless I’m misreading it which is possible it’s awfully late, he said he processed 60,000 rows didn’t find what he was looking for but his hard drive overheated on the full pass.
Discs don’t overheat because there was load. Even if he f***** up and didn’t index the data correctly (I assume it’s a relational database since he’s talking about rows) The disc isn’t just going to overheat because the job is big. It’s going to be lack of air flow or lack of heatsink.
I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he’s on a desktop and he slept a raw NVMe on his motherboard without a heatsink
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Can’t be a relational database, Musk said the government doesn’t use SQL.
He said many things.
Please remember that he is a genius. Only geniuses say a lot of things.
I rest my case.
Removed by mod
Lol he also said cybertrucks don’t suck ;)
deleted by creator
yes but also why say 60K when you could have literally said anything? I mean surely the fact that he thinks 60K rows a big number is already explaining alot lol.
It’s bait.
They probably have an explanation tweet at the ready to make more sense of it. They just want enough 'hurr durr these idiot" comments before they reverse Uno card this with more context.
Based on all that has been going on, I feel like they don’t really have the capacity to think more than one step ahead. They do sth stupid and then they usually follow up with “lol joke” or “lol you can’t understand”
Wait, seriously?
I don’t get how people don’t see this stuff. Yea the average trumpet isn’t out there planning Jack shit. But there are think tanks that are. Remember anti-smoking campaigns. Anti climate change campaign. The precision to purchase advertising into key areas and specific demographics that would spread a message. This is ancient knowledge with modern technology.
Imagine a room full of former Wall Street, quants, established experts from fields like behavioral science and psychology. All with the singular goal to decide where to dedicate a dragons horde of wealth to maximize effect in a world where we all have anonymous pipes directly into our eyes and ears. We never stood a chance. There’s no rich socialist funding think tanks. There’s no counter. We can laugh at the yokel all we want. But the yokel is being puppeteered by some scary fuckers with intention to seize power with the new shifting Zeitgeist. Soldiers don’t need to think. But their generals are. The left are like guerilla fighters going up against an imperial army full of Patons and Eisenhower’s.
Cambridge analytical, heritage foundation, international democracy Union. We’re fucked until we actually recognize why we’re fucked
“You don’t know what you’re talking about 💩💩💩”
There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they’ve never seen before and never done before.
Honestly, any sweet, white-haired old lady who keeps pictures of her dogs and grandkids on her desk who’s been doing data entry for 15 years could do circles around these clowns.
But she might also have the wisdom and perception to know we’re not supposed to be doing this “work” at all, which is why he recruits naive teenagers and college kids who are still emotionally immature to think that this is going to be their “destiny” or their opportunity to get into the big leagues of business.
I keep hearing things about these hires he has, I don’t think they’re naive, At least not as such. They seem to be more power hungry trust fund babies.
But yeah, people with a few years in them would be a moral liability in that line of work.
Yeah if you read more of these guys tweets they are clearly in politics. One message tried to claim trump loves kids (to be clear: in the abstract sense, not in the he definitely fucked kids on an island with Epstein sense). Then they tried to twist the words to say “why don’t you love kids”. It was clumsy like you’d expect from someone who is practically a teenager, but the core is an attempt to follow the usual right wing playbook.
music theaters also have rows, and they run on sql so logic checks out.
60k of rows is nothing. Fuck, where do you find these “geniuses”?
Tbf we don’t know how many columns there are /s
The IRS just switched columns and rows. So there’s 60k rows and 330 million columns /s
deleted by creator
I do data analysis for a living, I reach out to tech and complain if I can’t open a file with a million+ lines.
There’s only one reason he wants kids and not experienced adults.
He’s attracted to them?
Probably
Easily manipulated, but probably that too.
This cannot be real, wtf. This is cartoon levels of ineptitude.
Or sabotage by someone heading out? Please let this be resistance sabotage they haven’t noticed yet.
You guys arent running your software off raspberry pi’s with sdcards from the gas station?
My allowance is 5$ a month!
Look, all I’m saying is give Pis a chance.
I think my Pi could process 60k rows without overheating. And the poor thing is dangling behind my bookshelf from its power cord with a fine layer of dust coating every inch of it.
Why would you do this to that poor rPi?
Electronics get what they deserve.
You’re not supposed to place your laptop directly in the lap of your fur suit. Always leave an air gap for ventilation, smh.
Even a gamer knows that ssdd heat up but never to that level, lol.
What kind of cheap temu ssd does he have in his laptop?
He was saving money, you know, to be efficient.
Took out the fans for the aesthetic
This sounds like trying to do stuff in Excel? The computer isn’t overheating but the amount of memory needed is very high which would make it run poorly. They might interpret that as overheating?
It also makes sense if they are on calling the entire computer “the hard drive” like grandma and the fans kicked on.
Yeah, everyone commenting about being able to handle billions of rows easily, which obviously very true if you are worming with sql or similar.
But this is probably some finance kid, investment banker analyst, and only knows how to use Excel.
60,000 rows in excel with formulas, if not done efficiently, can for sure make you computer a little toasty.
Could be a gen 5 nvme drive without adequate cooling. Them bastards can run hot. Especially the early gen 5 drives.
But with only 60k rows would one even have enough time to overheat?
From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.
Considering that is nearly exactly some of the answers I’ve received during the technical part of interviews for jr data eng, you’re probably not far off.
Shit I’ve seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I’ll update, “one day”.
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
I’m a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
Seems like a good excuse to someone who doesn’t know what they’re doing and needs an excuse because why they haven’t completed it yet?
The whole post is complete bs in multiple ways. So weird.
If you work for a boss that fundamentally misunderstands what you are doing, then misleading them into thinking you’re ‘hard at work, making decisions with consequences’ is the theatre you put up to keep the cash flowing.
It’s one of the fundamental flows of autocracy, people try and represent what you want them to
It sounds like Hollywood tech lingo. Like when you’re watching a movie or a TV show and the designated techy character starts just saying computer words that make no actual sense in the real world, but I guess in CSI: Idiottown the hard drives have severe overheating issues.
Diming on her own stupidity without realizing it, like all of them do
Either she knows something novel, where processing data using voice coils is somehow beneficial, or is someone who calls their computer a ‘hard drive’, which summarily negates any legitimacy of technical competence.
Or wrote the code using AI without checking what it exactly does.
The AI probably used bogosort or something equivalent
deleted by creator
Doesn’t actually say that 60k overheated his drive. He says that he ran a run on 60k, and that he couldn’t do the whole database due to overheating. Two unrelated statements except that 60k is the lower bound for what he could process.
Doesn’t mean he knows what he’s doing though, as pretty huge datasets are processable on quite modest hardware if you do it right.
deleted by creator
I don’t want to take away from the valid point of your comment, but rocket science is almost exclusively math
deleted by creator
60k isn’t that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
60k is laughably, embarrassingly small. It’s still sqlite-sized.
Sqlite can easily handle millions of rows. Don’t sell it short
How about a 6.4TB sqlite database?
Should be enough to hold 60k rows
I have an sqlite db that is a few GB in size, game saves using the format. Sadly almost all blob data, would love to play with it if it was a bit more readable
i mean its even excel sized depending on how many columns. This is seriously sad and alarming
Hey now that’s real close to the 65,535 16-bit limit (from 20 years ago)
Holy shit if this ids lm issue that’s too funny
60k is single json file
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
He probably mining crypto on top of running his SQL queries.
What? You don’t run your hard drives in the oven while baking brownies? It makes them zesty.
There must be more join statements than column names
Don’t know what Elmos minions are doing, but I’ve written code at least equally unefficient. It was quite a few years ago (the code was in written in perl) and I at least want to think that I’m better now (but I’m not paid to code anymore). The task was to pull in data from a CSV (or something like that, as I mentioned, it’s been a while) and it needed conversion to XML (or something similar).
The idea behind my code was that you could just configure which fields you want from arbitary source data and on where to place them on the whatever supported destination format. I still think that the basic idea behind that project is pretty neat, just throw in whatever you happen to have and have something completely else out of the other end. And it worked as it should. It was just stupidly hungry for memory. 20k entries would eat up several gigabytes of memory from a workstation (and back then it was premium to have even 16G around) and it was also freaking slow to run (like 0.2 - 0.5 seconds per entry).
But even then I didn’t need to tweet that my hard drive is overheating. I well understood that my code is just bad and I even improved it a bit here and there, but it was still so very slow and used ridiculous amounts of RAM. The project was pretty neat and when you had few hundred items to process at a time it was even pretty good, there was companies who relied on that code and paid for support. It just totally broke down with even a slightly bigger datasets.
But, as I already mentioned, my hard drive didn’t overheat on that load.
I’ve run searches over 60k lines of raw JSON on a 2015 MacBook air without any problems.
I’d do that if I was given so much stupid access
Seriously - I can parse multiple tables of 5+ million row each… in EXCEL… on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
Excel have a limit of 2^20 rows, something more that 1M. Curious what version of excel are you using for that.
You’re correct - the standard tabs can only hold roughly 1.2 million rows.
The way to get around that limitation is to use the Data Model within Power Pivot:
It can accept all of the data connections a standard Power Query can (ODBC, Sharepoint, Access, etc):
You build the connection in Power Pivot to your big tables and it will pull in a preview. If needed, you can build relationship between tables with the Relationship Manager. You can also use DAX to build formulas just like in a regular Excel tab (very similar to Visual Basic). You can then run Pivot Tables and charts against the Data Model to pull out the subsets of data you want to look at.
The load times are pretty decent - usually it takes 2-3 minutes to pull a table of 4 million rows from an SQL database over ODBC, but your results may vary depending on datasource. It can get memory intensive, so I recommend a machine with a decent amount of RAM if you’re going to build anything for professional use.
The nice thing about building it out this way (as opposed to using independent Power Queries to bring out your data subsets) is that it’s a one-button refresh, with most of the logic and formulas hidden back within the Data Model, so it’s a nice way to build reports for end-users that’s harder for them to fuck up by deleting a formula or hiding a column.
Oh yes, I remember using power query for a few months once I started working with bigger databases, but I saw that moving to Python would be better carrer wise and never came back to excel to do actual work (but at the end everything get exported to excel)
My fucking events table of my synapse DB in postgres is nearly ten times as large, and I ported that from sqlite no long ago, in a matter of minutes. All of the data is on a 2*3 cluster of old 256GB SSDs, equaling about 1.5TB with Raid 0. That’s neither really fast, nor cool. But stable.
I will now attempt a full comment … brace your hard drives
Bro seriosuly fuck off my phone is overheating now. Thanks
Maybe
Maybe
They are just making shit up and doing jack shit
I really hope so.
What is this, a table for ants? Because that’s the average number of ants in an ant colony and it’s nowhere near an impressive amount of rows to be doing any sort of processing on. It wouldn’t be an impressive amount of rows if your rig was an i386DX-33 running off a 5” floppy.
Exactly, 60k rows is negligible enough in most cases that you can just treat it as free unless you’re doing a cross join on it or something, unless he’s doing something like using an unordered text file as his database with no ram or cache
Buddy’s probably running code he got from GitHub Copilot that is used to do a visualization of a bubble sort for learning purposes.
Even then, I can crank through orders of magnitude more data on my craptop
