• IP Copied!
    Click to Copy IP
    4 Players Online
  • Join our Discord!
    Loading..
    • Welcome to the Enlighten Network Forums!

      To report a bug, go here
      To appeal a mute/ban, go here
      To report a player, go here

    Emergency Downtime Incident

    BattleDash

    Network Developer and Systems Engineer
    Administrator
    BattleDash
    Joined
    Jun 28, 2020
    Messages
    450
    Reaction score
    297
    Hey all,

    About 20 minutes ago, all servers on the network crashed, the incident has been resolved now, and I'm going to do my best to explain what went wrong.

    We noticed the disk usage of our main dedicated server getting extremely high (93%), so I was clearing out some old logs and database backups we didn't need anymore. Since some of our server-specific databases like block logs use flat files, the process of clearing them only partially is a lot harder than clearing them entirely.

    First you need to loop over and parse every single entry in the database, which is quite a lengthy task on it's own, you then need to decide if the entry is worth keeping or throwing away. Before I continue, I want to say that I'm not certain this is the root cause of the issue, but it seems like the most probable contributing factor. Our main dedicated server has 128GB RAM, which is a lot, but we also have a lot of systems privately and publicly, and servers to run, so it adds up quickly. The specific database I was clearing out was 131GB of flat file storage. What I believe happened is that the process of iterating the database caused our memory to overflow into swap memory, and since the system disk usage was already so high (93%), it filled the rest of that up with temp memory for the operation, which caused our main databases and systems to crash with out of disk space errors.

    We will hopefully soon be getting a third dedicated server to distribute some of our server load to, to prevent instances like this happening again in the future.

    The server is back in full working condition now.

    Happy Playing!

    Thanks,
    BattleDash
     

    Shen

    Member
    Director
    Crinsib
    Vibes Vibes
    Member
    Joined
    Aug 16, 2020
    Messages
    50
    Reaction score
    9
    Thanks Dash!
     

    Jaren

    Still figuring it out
    Builder
    ProfessionalBot
    Joined
    Jun 28, 2020
    Messages
    36
    Reaction score
    20

    Earf

    Moderator
    Moderator
    earthfueled
    Kryptic Kryptic
    General
    Joined
    Jul 5, 2020
    Messages
    120
    Reaction score
    112
    Thanks for trying to explain it and for fixing it so quick šŸ’›
     

    AidenM15

    New member
    aidenqt
    Kryptic Kryptic
    General
    Joined
    Aug 4, 2020
    Messages
    24
    Reaction score
    7
    Yeah thanks for fixing it!
     

    Naainz

    Member
    Director
    Naainz
    .pfd Oblivion
    Member
    Joined
    Jul 3, 2020
    Messages
    47
    Reaction score
    20

    BattleDash

    Network Developer and Systems Engineer
    Administrator
    BattleDash
    Joined
    Jun 28, 2020
    Messages
    450
    Reaction score
    297
    I just want to know, is the server run from like a local IP or a hosting service?
    Like I said, we own dedicated bare-metal servers.
     

    About Us

    Starting as a small Survival server for a YouTuber, the Enlighten Network now supports a wide range of mini-games and modes!

    Quick Links

    Support Us

    The Enlighten Store funds server costs, development, and bringing you more new content to enjoy. View Store