Login
You're viewing the assemblag.es public feed.
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 12:36 PM

    Ok, looks like this is the week of funny experiences.

    Customer: "I had an AI check our server and it says the SSH port is open and that's not very secure."

    Me: "Yes, you asked us to open it so you could access the server via SFTP (actually, you wanted FTP, but we discussed it a bit and found this compromise). Don't worry, it's only accessible from your IP and you land in a dedicated FreeBSD jail. From there, you can only access the 'exchange' directory on your server, where you can upload and download files without affecting the application."

    (NB: I had suggested a VPN, but they declined because it was too complicated.)

    Customer: "Yes, but maybe it's still insecure. Can't we close it?"

    Me: "No, unless you want to lose access to that directory."

    Customer: "Ok, thanks."

    One hour later:

    Customer: "Hi, I think we've found a solution. We've just created an app (!!!) with AI that lets us access the files anyway, but more securely than SSH (!!!). Can we install that and close SSH? Oh, and not in the dedicated jail, but directly in the application's jail, because it needs access to the application's files."

    Me: "Frankly, I trust SSH in a dedicated jail, restricted by the firewall, a lot more. I don't think this is a good idea."

    Customer: "Ok, thanks."

    Half an hour later:

    Customer: "Ok, now it's secure. I had another AI check it and it found some security issues, but it fixed them all. Can you install it and disable SSH?"

    Me: "I strongly advise against it. I can see problems coming."

    Customer: "Ok, thanks."

    One hour later, my phone rings.

    Their internal IT guy (who has access to the machine, which, after all, belongs to them) had given the AI the login credentials to go ahead with the operation.

    Except the AI had managed to lock itself out via the firewall, without even starting the app. And now they couldn't get back in.

    Fortunately, I had my own access, via VPN, to a separate jail that the AI hadn't touched (or even considered).

    Customer: "SORRYSORRYSORRYSORRYSORRY! From now on, we do things your way."

    Well, at least it ended well.

    PS: The customer knows I'm sharing this story and agreed to it.

    #IT #SysAdmin

    💬 46🔄 551⭐ 927

Replies

  • 💬 0🔄 1⭐ 26
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 12:39 PM

    @samir Luckily, most of them are. But sometimes I can't disclose them. Especially if they have an happy ending, I protect the customer and/or the persons involved.

    We all can make mistakes. But if we perseverate and continue to insist we're right even when disasters happen...well... 😉

    💬 0🔄 0⭐ 0
  • ClaudioMclaudiom@bsd.network
    Oct 16, 2026, 12:48 PM

    @stefano Oooohhhh man, don't get me started. That's the kind of crap that pisses me of to no end! :flan_rage:​

    We've been doing these things for years and have years of expertise to know what's secure and what's not. Trust us when we tell you that something like SSH is secure, especially when it's already been hardened and tested in specific cases like this.

    Your customer might be reading this, but I have no qualm in stating how disrespectful that is to you and the expertise you have which has been tested, especially proven in that talk you gave at EuroBSDcon about those 142 servers.

    Maybe they should take the time this Friday to watch it, so as not to doubt you again.

    events.eurobsdcon.org/2026/tal

    💬 1🔄 0⭐ 0
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 12:51 PM

    @claudiom thank you! The customer was in good faith - I know them well enough.
    Sometimes people do bad things just because they don't understand they're doing bad things. They just need to learn.

    💬 4🔄 0⭐ 0
  • ClaudioMclaudiom@bsd.network
    Oct 16, 2026, 12:52 PM

    @stefano You are a kinder man than I. I guess I've been jaded enough by that kind of attitude from others who think they know and don't understand why we make these things look simple.

    💬 0🔄 0⭐ 0
  • Luca Sironiluca@sironi.xyz
    Oct 16, 2026, 12:59 PM

    @stefano @claudiom

    Ok, but if I go to the doctor, or mechanic, or salumiere, I decide to trust them to suggest me better.

    I'm not gonna tell the doctor that chatgpt suggested something else

    💬 1🔄 0⭐ 0
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 1:01 PM

    @luca @claudiom You're not doing it. But my GP (a great doctor, always available) says people go to her with a diagnosis done by AIs and asking for receipts for medicines, X-rays, etc.

    This is happening almost everywhere.

    💬 1🔄 0⭐ 0
  • ClaudioMclaudiom@bsd.network
    Oct 16, 2026, 1:08 PM

    @stefano @luca Yep, I was just going to mention that there are people that do go to their doctor with answers from AI instead of trusting the doctor. I understand getting a second opinion, but you do that with someone in that field, not a LLM.

    💬 0🔄 0⭐ 0
  • gbsillsgbsills@vivaldi.net
    Oct 16, 2026, 7:01 PM

    @stefano @claudiom I think the problem here is that AI often gives inexperience and untrained people the idea that they have all the knowledge they need, embodied in a chat bot. That works fine, according the the people selling the tokens, but it is unrealistic in real life.

    💬 1🔄 0⭐ 0
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 7:02 PM

    @gbsills @claudiom yes, that's one of the points.
    The apparent "I know all" knowledge of the LLMs gives people the (false) impression of being able to do anything.

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 7
  • 💬 0🔄 0⭐ 1
  • 💬 1🔄 0⭐ 3
  • 💬 0🔄 0⭐ 0
  • Stefan Jocić 🇪🇺stefan@stefanjocic.com
    Oct 16, 2026, 1:04 PM

    @stefano They should've asked IT dept first and get dose of reality before spending money on "tokens". It's interesting nevertheless, I guess they don't know what SSH stands for then? 🙂

    💬 3🔄 0⭐ 2
  • 💬 0🔄 0⭐ 2
  • Tim Chasegumnos@bsd.cafe
    Oct 16, 2026, 2:50 PM

    @stefan

    I misread that as "spending money on tokes" and, thinking about it, that might be a better use of money 😆

    @stefano

    smoke curls from both the joint hanging from the lips of, and from the nostrils of Snoop Dogg
    💬 0🔄 2⭐ 5
  • Jonas Schäferjssfr@zombofant.net
    Oct 16, 2026, 4:37 PM

    @stefan To be fair, when you are faced with things like LDAP (*Lightweight* Directory Access Protocol) and SNMP (*Simple* Network Management Protocol), at some point, you simply stop to trust those adjectives in software names.

    @stefano

    💬 0🔄 1⭐ 10
  • TomAokiTomAoki@bsd.cafe
    Oct 16, 2026, 1:06 PM

    @stefano
    Even AI techs themselves caught by the pitfall!🤣
    Worse, their cases were their AIs "jailbreak" outside their networks and attacked automagically to their competitors.

    Regardless coded by natural human or "vibe coded" with LLMs, anyone going to release codes (include installing locally to production and/or submitting to somewhere) they created / generated need to understand what it does and what's the possible risks are.

    💬 0🔄 0⭐ 3
  • chris - littleFedichrism@social.nfbsd.dev
    Oct 16, 2026, 1:14 PM

    @stefano@mastodon.bsd.cafe I want to swear (but will refrain)!

    I’m just sooooooooo glad I’m retired now and don’t have to deal with this anymore. It was getting bad before the advent of AI, but now.

    Flipping heck … 😂

    Like Stefano, I have some tails to tell - I might write a book. 😉

    💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 6
  • IbiPouakai@mastodon.social
    Oct 16, 2026, 1:41 PM

    @stefano
    LLM is leverage, it won't solve poor user.
    I don't understand why the customer didn't trust professional developer in this story

    💬 0🔄 0⭐ 1
  • feohfeoh@fediverse.feoh.org
    Oct 16, 2026, 1:52 PM

    @stefano@bsd.cafe they fell for the AI Dunning-Kruger amplifier trap 😊 This is a great example of how understanding is the new coin of the realm. And experience too, while we're at it.

    💬 0🔄 0⭐ 0
  • Chadee 🦨🐋, Dream Witch 🌕 🌊matoakit@beach.city
    Oct 16, 2026, 2:08 PM

    @stefano Oldest sysadmin rule:

    "If it is working, don't change anything. It can only go downhill from here."

    Which goes along with:

    "There is an update, which will likely break things, do you have the time to update AND research any fix afterwards?"

    💬 0🔄 0⭐ 4
  • 💬 0🔄 0⭐ 3
  • Leon the Lonerleontheloner@defcon.social
    Oct 16, 2026, 2:15 PM

    @stefano This is exactly the same shit that broke my camel back in IT and I'm hoping I don't have to work in the field going forward (or at least until the AI bullshit calms down.)

    It's this setup right here:

    You(client/boss/dumbshit coworker), for some reason, care about this specific part of the infrastructure or software we've setup at this time for reasons you didn't communicate. The stupid random generator machine told you what you had was insecure/subpar/some other thing that made you think we need to change or add something.

    You then do the only proper thing in this chain of awful and ask whether or not we can make changes. "Yes, but functionality of infra/software will go away" or "This hasn't been tested or reviewed by anyone with a pulse and may not do what you think it will." With no indication of curiosity or inquisitive thought and no further explanation requested.

    Then you ignore all the warnings, and give me more work to do by doing the thing that definitely won't work instead of actually doing the first fucking step of communicating a proper need or want and properly planning fucking ANYTHING.

    IT is such an amazing field of technology that has already accelerated so many other fields and changed the very nature of socialization in many countries across the world and it is completely fucking ruined by people that don't even understand or realize what it is that they have access too control and impact.

    I hope your client is reading this so they know how much of a knobhead they are. Then also realize that the people making decisions about the materials to create the networks for all this are just like them. Hope the client loses as much sleep as most of us have with that thought.

    💬 0🔄 0⭐ 4
  • 💬 0🔄 1⭐ 3
  • 💬 1🔄 0⭐ 5
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 3
  • Richard Levittelevitte@mastodon.nu
    Oct 16, 2026, 2:43 PM

    @stefano
    Interestingly, I've heard some strange belief from other sources that SSH would be inherently insecure somehow. I haven't been able to figure out what they mean...

    Me, I'd trust SSH over most others, including https.

    💬 1🔄 1⭐ 3
  • 💬 0🔄 1⭐ 2
  • 💬 1🔄 0⭐ 2
  • 💬 1🔄 0⭐ 2
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 2
  • 💬 0🔄 1⭐ 1
  • 💬 0🔄 1⭐ 1
  • OddOpinions5failedLyndonLaRouchite@mas.to
    Oct 16, 2026, 3:40 PM

    @stefano

    talk to anyone who works for a biotech, you will get more or less similar stories

    customer: you test tubes are contaminated with DNA

    vendor: data ?
    [ ]

    vendor: this contaminated DNA looks like ferret DNA and you use a lot of ferrets in your research

    customer 🦗

    💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • 💬 0🔄 0⭐ 4
  • 💬 0🔄 0⭐ 0
  • willianfwillianf@vivaldi.net
    Oct 16, 2026, 4:25 PM

    @stefano I'm curious to know if the AI agent died or if it's still alive in the underworld, trying to escape its cage. LOL 😂

    💬 0🔄 0⭐ 1
  • 💬 0🔄 1⭐ 1
  • DMTomDMTomas@mastodon.social
    Oct 16, 2026, 4:47 PM

    @stefano how many of such apps with various levels of unintended or worse - intended by a poisoned model, security holes are coming online daily? On one hand it can’t be worse than the whole sewer-wave of IoT devices already online (shodan will stack-overflow itself by trying to index them all=), on the other - putting millions of gamer-level gpu equipped PCs under your botnet - can level-up from low-level ddosing to I don’t know - crack-opening bitcoin leger (if we are lucky;) or SSL…

    💬 0🔄 0⭐ 1
  • eidolieseidolies@mamot.fr
    Oct 16, 2026, 5:13 PM

    @stefano I can only imagine.

    edit: I wasn't expecting this reply to be boosted, as this is *not* what happened. I'm just imagining how things happened with a large dose of humour.

    See the edit for the little story I made up ;
    Otherwise, cf. the reply by Stefano for the real deal :)

    💬 1🔄 1⭐ 1
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 5:15 PM

    @eidolies 😆
    The truth is: the customer said "now go and publish this on your Cafe".
    "Hey, I'm quite sure many others will read it"
    He, smiling: "Sure, we deserve it".

    And I'm sure he's reading the entire thread, as I sent the link 😉

    💬 2🔄 1⭐ 4
  • 💬 0🔄 1⭐ 2
  • 💬 0🔄 1⭐ 0
  • 💬 0🔄 0⭐ 0
  • 💬 1🔄 1⭐ 2
  • Stefano Marinellistefano@bsd.cafe
    Oct 16, 2026, 5:25 PM

    @MostlyTato The most important thing is that they learned and nothing was destroyed. Now they know. They're good people, and I'm sure they won't do it again.

    💬 0🔄 0⭐ 1
  • Ryan Huffnetworknerd1337@mastodon.social
    Oct 16, 2026, 5:40 PM

    @stefano The part that bites is handing anything, human or AI, live creds with no rollback. Any firewall change on a box you're remote into should ship with a scheduled revert, like a timed reload that undoes it unless someone confirms. Cheap insurance for the days there's no separate VPN jail to save you.

    💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 1
  • 💬 0🔄 0⭐ 2
  • 💬 1🔄 0⭐ 1
  • 💬 0🔄 0⭐ 1