file system

Best way to save files?

My first idea would be to just use a filesystem and store my files there, but there are many advantages to use a database, since it's faster and supports ACID consistency. After looking at the pros and cons we decided to just use a MongoDB and save the bytestreams of the files in there. The data is more consistent, we can get them very quickly and since we already had a database setup we had not much effort to create our own "filesystem" in the database.

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Taggings:

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Taggings:

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • Using Softwareraid was possibly not the best solution. In my understanding back then when I set up the server, RAID 1 sounded very nice, no more file loss. I found out that the raid controller software checks for corruption, but he does not know which data is correct. So he chooses randomly one file of one drive, mirrors it to the other and act like nothing has happend. In this case it happened to be the corrupt one.
  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • I need a file system with data integrity features
  • There are two possible candidates:
    • ZFS
    • BTRFS
  • ZFS is first published in 2006 by Sun and the more proven solution. It can handle large disk arrays up to 256 ZiB (should be enough ;) ), has a checksum based data integrity solution (it hashes every block with SHA-256) and supports multiple RAID levels.
  • BTRFS is published 2014 by Oracle. It has also a checksum based data integrity solution (hashes with CRC-32C) and can automatically restore "bad files". Under its current version, it can handle RAID 1, which suits the solution perfectly

Using ZFS as file system

After a lot of investigation on file systems and bit rot I present you my findings:

  • There are two possible candidates:
    • ZFS
    • BTRFS

Access Disk-Image-Files with Windows (.dmg)

Mac OS X uses its own file system for image files (HFS, HFS+). Files/Drives with this file system cannot be accessed under Windows. The same problem occurs with Disk-Image files that are used by Mac OS X to bundle data and burn them to an external data disc. Under Windows there is no built in solution to handle with these HFS based file systems.
Subscribe to file system