Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
rsync 8 minutes ago [-]
Free account, indefinitely, to whoever needs it to store this data … including the storage vendor.
Just email…
ktm5j 1 hours ago [-]
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
jerf 56 minutes ago [-]
I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
ameliaquining 6 minutes ago [-]
The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.
mindslight 22 minutes ago [-]
> 10 years ago, this was a lot of data.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
MostlyStable 1 hours ago [-]
Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem
ndiddy 1 hours ago [-]
> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
mcmcmc 1 hours ago [-]
3 copies, 2 different media, 1 offsite. One backup is none
embedding-shape 37 minutes ago [-]
1 copy, one other media, is better than 0 copies. Clearly 1 != 0.
thebitguru 30 minutes ago [-]
The point still stands, but in what world can you build a 50TB NAS for $1,000 these days? I want to go there
dghughes 19 minutes ago [-]
Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!
nairboon 19 minutes ago [-]
$1000 is for the NAS, they still need to populate it with a couple of drives.
echoangle 39 minutes ago [-]
Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.
mulmen 24 minutes ago [-]
What risk? The point of PBS is “other people access the data.”
OutOfHere 48 minutes ago [-]
A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
From the article it seems like Iron Mountain was doing the actual storage, and OSS was just acting as a middleman and taking some money off the top.
chuckadams 43 minutes ago [-]
And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.
Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
embedding-shape 33 minutes ago [-]
> Don't trust Internet Archive as your only backup method though.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
toomuchtodo 1 hours ago [-]
Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
Good4boothee 36 minutes ago [-]
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
undersuit 29 minutes ago [-]
This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.
2 hours ago [-]
adolph 2 hours ago [-]
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
blltprfmnk 2 hours ago [-]
Title should be updated to reflect the fact that this is a single PBS station.
cootsnuck 1 hours ago [-]
That made me feel relief until I realized it's my own local PBS station.
toomuchtodo 1 hours ago [-]
Any interest in connecting me with someone there so I can attempt to obtain a copy for the Internet Archive?
ButlerianJihad 1 hours ago [-]
Ah, "Channel Nine PBS", not a cluster of nine pbses
Rendered at 15:50:26 GMT+0000 (Coordinated Universal Time) with Vercel.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
Just email…
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
https://www.linkedin.com/company/osstorage/people/
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
https://old.reddit.com/r/DataHoarder/comments/sq6wbq/please_...
https://archive-it.org/vault/
(huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.