File got corrupted on Fileserver, Permanent way to avoid this issue

In my network I have a Fileserver with 2x4 TB disc capacity. I run debian on it. The initial configuration was to put the two HDDs on Software RAID 1. This setup worked perfectly fine for about 3.5 years. It worked until I needed a large zip file containing a game. A friend was at my home and we wanted to play this game. I allready had the file on my computer, he copied it from the Server. When he wanted to unzip it, WinRar throw some errors. He did not have any write permissions, so it was not possible that he or I had modify the file by mistake. A quick comparison between the checksums confirmed that both files were different. I replaced the currpeted file on the server with the one on my computer and we could play. It was my first encouter of Bit rot. A not funny experience, because I knew that possibly more files are corrupt. I then needed a better and more robust solution to store my important files.
8 answers

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Taggings:

Comments

I think filesystem CRC checks wont really help against bit rot because most hard drives already have ECC integrated (https://nctritech.wordpress.com/2017/03/07/zfs-wont-save-you-fancy-filesystem-fanatics-need-to-get-a-clue-about-bit-rot-and-raid-5/).

Masood Lodin - Fri, 12/01/2017 - 01:25 :::

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Taggings:

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • There are two possible candidates:
    • ZFS
    • BTRFS