Securing your data over time

[David Heffernan 2353]

You should have all your work in revision control. Then when your drive fails you still have it. Not to mention all the other benefits. 

[FPiette 385]

10 hours ago, Marty001 said:

I installed with default everything, so if I move the projects folder I take it that I would not need to change anything in the IDE to locate support files etc

Please read what I said again. If you move 3rd party component it is likely you have to update their path, depends on the components and how you add it to your projects.

Think about taking a backup on a regularly schedule. All disk will fail one day! Also think about using disk mirroring (RAID1) or other kind of RAID. With RAID (except RAID0), all you data is duplicated so even when a disk fails, you still have your data. Just replace the failed disk and go.

[aehimself 399]

2 hours ago, FPiette said:

All disk will fail one day!

I learned that the hard way. In our country Quantum IDE hard-drives tended to fail often and suddenly (during power up the continuous click-click-click noise; I guess the arm was unable to move the heads?) but they were cheap so everyone used that.

 

I lost ALL of my source codes of ALL of my applications one day. Thankfully, back in those days I wrote applications for myself and to a small number of people so I simply could say well, we use the latest version and no further improvements...

[FPiette 385]

25 minutes ago, aehimself said:

I lost ALL of my source codes of ALL of my applications one day.

Are you doing backups now? Are you using RAID? Most today's mainboards have a basic RAID controller and if yours don't have one, buy one: it is not expensive at least for a basic version. High end version have a battery backup as well so that in case of power loss, the current duplication will continue and cache will be written to disk.

Here we speak about source code, but today, almost everyone save his photos on hard disk as well. And many lost everything because they don't have a backup. Personally, I don't reuse SD card: I keep them as backup.

[aehimself 399]

Just now, FPiette said:

Are you doing backups now? Are you using RAID? Most today's mainboards have a basic RAID controller and if yours don't have one, buy one: it is not expensive at least for a basic version. High end version have a battery backup as well so that in case of power loss, the current duplication will continue and cache will be written to disk.

Here we speak about source code, but today, almost everyone save his photos on hard disk as well. And many lost everything because they don't have a backup. Personally, I don't reuse SD card: I keep them as backup.

Yes and yes. Since I have a dev environment at home I just added 2 extra hard drives mirrored (with daily automatic health check and warning E-mails). I do my development on my PC then syncing it upstream on a Git repository stored on this mirror. I have a physical RAID controller with a battery backup (salvaged a HPE P400 with 2 failed battery packs. Turns out it is using 4 AAA rechargeable batteries inside, so the quick fix was a matter of a bit of soldering) and the server is on a high capacity UPS.

 

All this hassle for home, you are free to call me paranoid and I'm not going to argue 🙂

 

P.s.: I'm not considering any flash-based (including SSDs) devices suitable for backup. Their write cycles are low and sometimes you recognize something failed when you are reading information back. And that's too late if it's used for backup. Minimum a mirrored HDD (which is always on, so not USB-attached which "I keep safe in the wardrobe and only use when necessary". It's good for power consumption, but it's bad for lifespan) but the best is still a tape drive - IN MY OPINION. I have personal privacy and safety concerns when it comes to "cloud" data storage.

[David Schwartz 430]

12 hours ago, aehimself said:

P.s.: I'm not considering any flash-based (including SSDs) devices suitable for backup. Their write cycles are low

For average use as a backup drive (as opposed to replacing your HDD in your main computer that you use daily) the MTBF would be around 100 years. Spinning HDDs are <5 yrs, esp. if they're in a NAS that runs continuously. Flash storage doesn't "spin" and really don't do anything if they're not being accessed, unless your hardware logic is constantly reorganizing them and running tests, which seems silly for SSDs.

 

Personally speaking, I've found most traditional HDDs fail after about 3 years. I have SSDs in my 2014 Mac Book Pro and they're still going strong. I put them in my 2014 Mac Mini that I use daily and so far no problems.

 

My last 3 employers over 10 years had SSDs in our work laptops that we used regularly. I never had any trouble, but a colleague at one place had his SSD fail after just 3 months of use. The IT guy who fixed it said it was only the 2nd failure he'd seen since they started using them a few years earlier, and this one happened to be brand spanking new. It was a Dell laptop.

 

Honestly, the price of SD cards is getting so low that you could make rotating backups just from a handful of them. I see Walmart is advertising 64GB Class 10 microSDs for $5. I see an outfit named Wish.com that's selling 1TB Class 10 UHS-1 TF microSDs for $7.64. SD cards aren't nearly the speed of an SSD like Samsung T5's but for backing up source files on a daily basis I don't think the speed differences would even be noticeable. If you got 10 of them and rotated them daily, you'd use each one 36 times a year. For devices rated in the 7-figures, I wouldn't be terribly worried about long-term reliability at that rate. You'll be dead and gone by the time one failed.

Edited August 30, 2020 by David Schwartz

[Anders Melander 1815]

21 hours ago, FPiette said:

Are you using RAID? Most today's mainboards have a basic RAID controller and if yours don't have one, buy one

It's a misconception that RAID is a means to better data security. RAID is a means to better performance or higher availability.

 

8 hours ago, David Schwartz said:

Spinning HDDs are <5 yrs, esp. if they're in a NAS that runs continuously.

If you buy cheap drives or cheap NAS with cheap drives in them then you get what you pay for. No surprise there really.

The HDDs in my system are 16 years old and some of them have been running almost continuously. They're Western Digital WD5000YS drives with a MTBF of 137 years. I have two drives mounted internally and five drives in a hot-swap rack on the front for use as backup storage. Of course I also have a couple of SDDs for the performance critical stuff.

 

HDDs have better reliability than SDDs and they are better for long term archival. While a brand new SDD might have better MTBF values than a HDD this dramatically changes with time as the drive is used. After only a few years the SDD will have deteriorated by a magnitude to have much worse MTBF than the HDD.

 

8 hours ago, David Schwartz said:

Honestly, the price of SD cards is getting so low that you could make rotating backups just from a handful of them. I see Walmart is advertising 64GB Class 10 microSDs for $5. I see an outfit named Wish.com that's selling 1TB Class 10 UHS-1 TF microSDs for $7.64. SD cards aren't nearly the speed of an SSD like Samsung T5's but for backing up source files on a daily basis I don't think the speed differences would even be noticeable.

So you're good with relying on the cheapest available devices to save you when your primary storage fails? Interesting.

Personally I would prefer a reliable backup medium so that I could afford to use faster, but less reliable, primary storage devices.

[Fr0sT.Brutal 900]

19 minutes ago, Anders Melander said:

So you're good with relying on the cheapest available devices to save you when your primary storage fails?

+1. I'd consider SD's a very unreliable storage (especially micro SD's - the smaller the more fragile they are).

[Anders Melander 1815]

9 hours ago, David Schwartz said:

Spinning HDDs are <5 yrs, esp. if they're in a NAS that runs continuously.

Oh, and another reason why this is wrong is that it's better for HDDs to run continuously. It's the power cycles that kills them (thermal wear).

[Lars Fosdal 1793]

I love SSDs and I have not had one fail yet over the last six years.  HDDs, on the other hand, has failed me numerous times.

As for backups - Online, multiple services = redundancy.

[aehimself 399]

5 minutes ago, Lars Fosdal said:

I love SSDs and I have not had one fail yet over the last six years.  HDDs, on the other hand, has failed me numerous times.

As for backups - Online, multiple services = redundancy.

The amount of numbers HDDs failed on us more is because SSD is much-much younger than HDDs are; but I already experienced SSD issues too. I still have an SSD in my laptop and in my dev server (with OS drives of guests on it) and they are still error free. I love them too; we all love them. But their lifespan is shorter than a HDDs - especially if we don't let that HDD to stop 🙂

 

Edit: online backups are nice and should be safe enough. I just happen to be paranoid enough not to trust any big company.

Edited August 31, 2020 by aehimself

[David Schwartz 430]

9 hours ago, Anders Melander said:

So you're good with relying on the cheapest available devices to save you when your primary storage fails? Interesting.

Personally I would prefer a reliable backup medium so that I could afford to use faster, but less reliable, primary storage devices.

Well, I guess it's a matter of perspective. OGs like me who still have boxes of 3-1/2" floppies stashed away remember using them to set up rotations for backups. Sure, they failed, which is why we set them up in batched rotations! Ditto with tapes.

 

I was looking around for some backups of work I did at one point back in the early 90's. It was between the "floppy-disk era" and the "writable CDs era" when I used one of them "super-high density and high-reliability" tape drives. Now I've got a few years of backup tapes with stuff on them and no way in the world to read them. I can still read those old DOS floppies and the CDs. But backup tapes? <shrug> 

 

Whatever you might think of SDs, I guarantee they won't decompose the way old magnetic media does over time. There's not much room to write on them, to be sure, and with them getting so cavernous that it'll be impossible to use them for long-term storage without taping them to a large printout that says what's there. But I don't know anybody who does that the way we used to mark-up floppies. Besides, you really couldn't put much on floppies, so it wasn't that big of a deal. CDs started to become a problem in that respect.

 

As for online backups ... there seem to be three options: smaller free accounts that piggy-back on larger subscription models (like Dropbox offers); monthly subscription offers, which tells me the company will probably be around; and single-payment "lifetime" offers, which will eventually disappear for lack of revenue. I've already been bitten by one of them that shut down 6 months after a huge surge of initial customers. You've gotta have a way to keep the lights on, if nothing else; people forget about that. The problem with the monthly subscription deals is ... when you stop paying, everything will evaporate (sooner or later).

 

I'll trust a box filled with ANY kind of media, including SDs, far longer than ANY online service after I stop paying their bills. SDs _are_ a very cost-effective one-time payment for a huge amount of storage, and you're in 100% control of them forever, unlike 3rd-party cloud storage where you really have no idea how it's being used or even of it will be there when you need it. Consider how much storage a bunch of microSDs could hold laid out to cover the same surface area as one writable CD/DVD. A CD is 700MB; a DVD is 4 or 8 GB. I'm guessing that 1TB microSDs laid out to cover the same surface area as a CD would probably yield 50 TB or more of storage. I don't know who'd need that much on a regular basis, but using, say, 60 x 1TB microSDs for a rotating backup would mean you write each one a total of 6 times per year if you make a daily backup of ALL your data and it doesn't exceed 1TB.

Anyway, unless you're creating or editing videos and large graphics files (or humongous databases) on a daily basis, only a very tiny percentage of your data changes day-to-day. So most of the data you capture with full backups from one day to the next will be >99% redundant and remain static going forward. The likelihood of "losing" stuff is going to be more depending on your ability to actually FIND IT rather than physical data loss. So there's a great market gap opportunity: a search tool that lets you index your backup media and keep it independent of the media so you can search for stuff "offline" as it were.

Edited August 31, 2020 by David Schwartz

[Bill Meyer 337]

31 minutes ago, David Schwartz said:

Well, I guess it's a matter of perspective. OGs like me who still have boxes of 3-1/2" floppies stashed away remember using them to set up rotations for backups. Sure, they failed, which is why we set them up in batched rotations! Ditto with tapes.

 

I was looking around for some backups of work I did at one point back in the early 90's. It was between the "floppy-disk era" and the "writable CDs era" when I used one of them "super-high density and high-reliability" tape drives. Now I've got a few years of backup tapes with stuff on them and no way in the world to read them. I can still read those old DOS floppies and the CDs. But backup tapes? <shrug> 

I have learned the hard way that trying to read an old 3-1/2" floppy is perilous. My experience has been that most of them shed oxide which is then very hard to remove from the head. And some of us OGs still have stacks of 5-1/4" floppies (which do not seem to share the shedding problem of their smaller cousins.)

 

[aehimself 399]

42 minutes ago, David Schwartz said:

Whatever you might think of SDs, I guarantee they won't decompose the way old magnetic media does over time.

Ah yes, We used to compost floppy disks 😐 I already imagine the next-gen "uSD libraries" in data centers. At least the replacement drives would be cheap to order from Wish.

 

[Anders Melander 1815]

48 minutes ago, David Schwartz said:

Well, I guess it's a matter of perspective.

Sure.

If you want reliable backup use a media that has been designed for it. Floppies, writable CDs and DVDs, cheap HDDs, SDDs and SDs aren't reliable.

I have TK-50 DLT tapes from the late eighties that can still be read (each tape contains a whopping 94Mb). The tape format is standard and the drive is SCSI so it's no problem finding a way to read them - and the data is still there. Try that with any other storage type after thirty years (stone tablets and punch cards excepted).

 

53 minutes ago, David Schwartz said:

Whatever you might think of SDs, I guarantee they won't decompose the way old magnetic media does over time.

I think it pretty irresponsible to make a statement like that. You can't guarantee anything like that and I'm pretty sure you will not take financial responsibility when your guarantee turns out to be false. The companies that make these devices doesn't even make such claims. It is known that SDD and SD degrade over time. Their good MTBF is only valid when they are new.

[Rollo62 539]

Regarding backups I would also take GitHub into account.

For a few bucks, about 50 to 60 USD per year, you get a perfect project storage.

Imagine how many hard drives you need for storage in this year. 

[Attila Kovacs 631]

http://ollydbg.de/Paperbak/

[aehimself 399]

3 minutes ago, Attila Kovacs said:

http://ollydbg.de/Paperbak/

Incredible! If it runs on WIndows 2019 Core, finally I can fire up that 15-year-old leporello printer to have some good use of it!

[Anders Melander 1815]

And if everything else fails there's always the NSA to fall back on.

https://www.cnet.com/news/guess-what-happened-when-backblaze-tried-using-the-nsa-for-data-backup/

https://o.canada.com/technology/internet/irony-alert-google-labels-nsa-data-centre-backup-service

[Lars Fosdal 1793]

SD cards are expected to keep data for about five years before they start deteriorating. They actually lose charge over time. 

http://www.datarecoveryspecialists.co.uk/blog/what-is-the-life-expectancy-of-an-sd-card

 

Even CDs and DVDs may deteriorate within two to five years unless stored in a dark, dry, cool place.

Blu-ray disks are supposed to be more robust.

 

It is not hard to find reliable online storage. 

The real question is what value you put on your data, and if you are willing to invest that value. 

 

[haentschman 92]

Quote

It is not hard to find reliable online storage. 

...yes.

What if your house burns down? All data that cannot be restored, especially the Delphi projects, must be transferred to the data center outside the house. (Webspace/Webserver)

[Fr0sT.Brutal 900]

2 hours ago, haentschman said:

...yes.

What if your house burns down? All data that cannot be restored, especially the Delphi projects, must be transferred to the data center outside the house. (Webspace/Webserver)

Storage box in a bank xD

[Lars Fosdal 1793]

Spotted this
 

Quote

Blu-ray usage is already in decline just a decade after it won the format war, and there's no new physical disc to replace it. Instead, everyone is heading to streaming services. The M9500 may be Samsung's last Blu-ray player.Feb 19, 2019

 

So even that bank box will be a no go soon.
We are inevitably headed for online storage.

[dummzeuch 1517]

2 hours ago, haentschman said:

...yes.

What if your house burns down? All data that cannot be restored, especially the Delphi projects, must be transferred to the data center outside the house. (Webspace/Webserver)

I'm not sure whether I'd care much about Delphi projects when my house burns down. But on the other hand, I don't have a house, I live in a rented flat, so maybe I would care.

[aehimself 399]

9 hours ago, haentschman said:

...yes.

What if your house burns down? All data that cannot be restored, especially the Delphi projects, must be transferred to the data center outside the house. (Webspace/Webserver)

Proper backup solutions require the backup area (including servers and libraries) to be physically separated to different buildings. Thus, production center burns - you still have backups and vice versa.

I also run a machine with extremely low power consumption kilometers away from my house. It's only job is to replicate the data from my prod server, when network is up.