this post was submitted on 05 May 2026
288 points (99.0% liked)

Technology

84434 readers
4145 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
288
The AI Hard Drive Shortage Is Making It More Expensive and Harder to Archive the Internet (www.404media.co)
submitted 2 days ago by sanitation@lemmy.radio to c/technology@lemmy.world
13 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] tal@lemmy.today 5 points 2 days ago* (last edited 2 days ago) (1 children)

Use tape libraries for the moment, with hard drives acting as a cache for them? Doesn't need to mean moving the whole backing storage to tape, just predicting what won't likely be used soon and letting the storage format indicate "go look on tape for this item". Obviously, that can result in much higher cold storage retrieval latency, but as long as you are (a) doing predictive fetching with a reasonably good algorithm and (b) have a lot of hard drives, which I'm sure that The Internet Archive does, I'd think that tape should be workable.

https://en.wikipedia.org/wiki/Tape_library

In computer storage, a tape library is a physical area that holds magnetic data tapes. In an earlier era, tape libraries were maintained by people known as tape librarians and computer operators and the proper operation of the library was crucial to the running of batch processing jobs. Although tape libraries of this era were not automated, the use of tape management system software could assist in running them.

Subsequently, tape libraries became physically automated, and as such are sometimes called a tape silo, tape robot, or tape jukebox. These are a storage devices that contain one or more tape drives, a number of slots to hold tape cartridges, a barcode reader to identify tape cartridges, and an automated method for loading tapes (a robot). Such solutions are mostly used for backups and for digital archiving. Additionally, the area where tapes that are not currently in a silo are stored is also called a tape library. One of the earliest examples was the IBM 3850 Mass Storage System (MSS), announced in 1974.

In either era, tape libraries can contain millions of tapes.

Physically automated tape library devices can store immense amounts of data, ranging from 20 terabytes[13] up to 2.1 exabytes of data[14] as of 2016.

For large data-storage, they are a cost-effective solution, with cost per gigabyte as low as 2 cents USD.

I'd also guess

though I don't know for sure

that it's probably a lot easier to scale up manufacturing of tapes than it is hard drives.

EDIT: Does kind of make me wonder what the open-source options for tiered storage like that is. I've never really gone hunting, but it seems like there'd be a lot of commonality from place to place, and that for a lot of places that do it, it's not really their core competency (that is, they just want to do something that deals with storing and processing lots of data, not that they really care principally about data storage).

[–] TheFogan@programming.dev 3 points 2 days ago (1 children)

I mean there's probably a pretty large aspect of slow storage that's far more ideal for archiving/backup options, as LLMs are really only interested in the fastest of things.

[–] tal@lemmy.today 3 points 2 days ago* (last edited 2 days ago)

I would bet that a lot of the storage that AI companies are picking up isn't for the model itself, but for storing the huge amount of information that they want to use as their training corpus.

I'd bet that what they do is something like this:

  1. Download data and store in original form, non-destructively. This is probably not used incredibly frequently. When you see bots sucking down the whole Web, this is the sort of thing that is involved.

  2. Have some kind of filtered training corpus. This throws out a lot of stuff that is useless for training. This is generated from #1 by filtering software. It's probably smaller than #1. Probably a lot smaller.

  3. Probably some sort of scored index is generated at this stage to put an estimate on how useful or reliable the data in step #2 should be considered; I'd assume that this is an input into the training.

  4. The generated model, generated via training.

For the data in stage #1, I'd guess that AI companies might be able to use tapes. That being said, it might make sense to use faster storage if it accelerates the time to iterate on improving the filtering software.

But, yeah, for the later stages, tapes probably aren't gonna work.