I’m writting a simple bash script to do a recursive paste over >10000 files with two columns each.
The files looks like this:
And the code is this:
SPHERE=Sphere.matrix.txt rm -rf $SPHERE for i in `ls *.spheres | sort`; do if [ -f $SPHERE ]; then cut -f2 $i | paste $SPHERE - > $SPHERE.tmp mv $SPHERE.tmp $SPHERE else cat $i > $SPHERE fi done
it opens the first file and writes it to the output file. then it opens the second file, it takes the second colums and it pastes it to the output file, and so on…
the code works propperly, but it gets progresively slower because in each cycle it has to open and overwrite a bigger file.
if I could store the result of aech iteration in memory instead of in disk, I think that the performnace would be much faster.
could you please give me some guidance in this?
thank you so much!