I also found it useful to do some one-off batch processing of a big file without thinking of chunking and doing too complicated things. Imagine something like a 60GB file in a custom format that need to be deduplicated. It’s easier to just load it all up in memory and do run the algorithm 2-3x slower using a 50GB swapfile than spend hours to complicate the algorithm.
Yep. That’s why it can be nice to just have like an old 1TB drive completely used for swap. Things might get slow once it exceeds your physical memory, but at least it will eventually finish successfully – no out of memory crashes here!
I also found it useful to do some one-off batch processing of a big file without thinking of chunking and doing too complicated things. Imagine something like a 60GB file in a custom format that need to be deduplicated. It’s easier to just load it all up in memory and do run the algorithm 2-3x slower using a 50GB swapfile than spend hours to complicate the algorithm.
Yep. That’s why it can be nice to just have like an old 1TB drive completely used for swap. Things might get slow once it exceeds your physical memory, but at least it will eventually finish successfully – no out of memory crashes here!