Find duplicates in a folder and convert them to symlinks

Hi,
Duplicate files waste precious space on your computer’s hard drive.
Here a terminal approach with a script to vonvert them to symlinks:

first install the dependencies:

Code:
sudo apt-get install rdfind symlinks

Now create the blxremoveduplicates script:

Code:
#!/bin/bash
if [[ -d "$1" && ! -L "$1" ]] ; then
rdfind -makesymlinks true $1
symlinks -crs $1
else
echo "Enter a valid directory name!"
fi

USAGE : ./blxremoveduplicates folder_name

You can „compile“ it with shc and place it in the /usr/bin folder after that.
if you know that your directory contains broken symlinks use this script instead:

Code:
#!/bin/bash
if [[ -d "$1" && ! -L "$1" ]] ; then
find -L $1 -type l -delete
rdfind -makesymlinks true $1
symlinks -crs $1
else
echo "Enter a valid directory name!"
fi

Another method:

There are still good, mature applications for finding and removing duplicate files on Linux. FSlint offers a good graphical interface and should be available in most Linux distributions’ software repositories. You won’t find the shiniest interfaces, but you will find simple, functional software that doesn’t want to push junkware on you or sell you anything.

source: http://www.howtogeek.com/201167/how-to-find-and-remove-duplicate-files-on-any-operating-system/

You can install this with:

Code:
sudo apt-get install fslint

start it with: fslint-gui

Fslint is a bit easier to use but i do see that it fails very often and the method is nor so stable as my command line approach, but perhaps you`ll like fslint more, so i posted it here too.

Hope this tutorial it helped someone.

Experimental Compression method for folders

Do show you here a method to compress folders, that is not recommended for very very very very and very important backups, but might be interesting to decrease file size for transferring files etc. etc. :

first you need to install lrzip tar xz-utils and sed:

Code:
sudo apt-get install lrzip tar xz-utils sed

Now create a file named blxcompress.sh or any other name you want it to have with following content:

Code:
#/bin/bash
echo Compressing folder $1
tar cvf $1.tar $1
lrzip -n -o $1.tlrz $1.tar
xz -z $1.tlrz
rm $1.tar

Now you can compress a folder by typing ./blxcompress.sh folder_name

If you want to decompress create the following ./blxdecompress.sh script:

Code:
#/bin/bash
fname=`echo "$1" | sed 's/........$//'`
echo You compressed the folder $fname
cp $fname.tlrz.xz $fname-copy.tlrz.xz
xz -d $fname-copy.tlrz.xz
lrzip -d -o $fname-copy.tar $fname-copy.tlrz
tar -xvf $fname-copy.tar
rm $fname-copy.tar
rm $fname-copy.tlrz
echo folder $fname extracted sucessfully

Usage: ./blxdecompress.sh folder_name.tlrz.xz

This compression method may be better than normal xz compression. The reason is that it does search for repeating patterns in a distance of several hundreds of MB. Normal gzip and xz can`t do this.

lrz is a good compression format if you ahve files that contents repeat very often. This method does devide the process up in three steps and that is more stable than letting lrzip do all. lrzip gets really slow on my on if i try to use itas xz method. If i do use xz after lrzip it does work very well.

Tought it is ideal for compression of 1-4 GB files and if you have time for compression and decompression since its really slow but has sometimes nearly the same ratio as normal zpaq which is much slower.

You might „compile these scripts with shc and put them in /usr/bin if you like to do that!

regards bluedxca93

Image manipulating with lua/-imlib2

Gmic is a bit complicated, gluas has to many bugs and imagemagick is in C,
but you can`t write C programs and need to write your own simple image
filter?
Then the following instruction might be intertesting if you have ubuntu
installed:

Lua is implemented in pure ANSI C and compiles unmodified in all platforms
that have an ANSI C compiler. Lua also compiles cleanly as C++. source:
http://www.lua.org

How to install lua5.1 and the imlib2 plugin :
Code:

sudo apt-get remove lua5.2
sudo apt-get install lua5.1 luarocks
libimlib2-dev
sudo luarocks install lua-imlib2

Reference for easier writing your own image filter you `ll find here:

„_blank“>http://asbradbury.org/projects/lua-imlib2/doc/

However i didn`t understand somje things of it at the beginning so im now
posting a sample script:
Code:

-- simple selective blur
-- by bluedxca93
require("imlib2")
local im = imlib2.image.load("input.png")
local xlength = im:get_width()
local xheight = im:get_height()
for x=0, xlength-1 do
for y=0, xheight-1 do
level = 15
c0 = im:get_pixel(x, y)
c1 = im:get_pixel(x-1, y-1)
c2 = im:get_pixel(x, y-1)
c3 = im:get_pixel(x+1, y-1)
c4 = im:get_pixel(x-1, y)
c5 = im:get_pixel(x+1, y)
c6 = im:get_pixel(x-1, y+1)
c7 = im:get_pixel(x-1, y+1)
c8 = im:get_pixel(x-1, y+1)
c0r = (c0.red)
c1r = (c1.red)
c2r = (c2.red)
c3r = (c3.red)
c4r = (c4.red)
c5r = (c5.red)
c6r = (c6.red)
c7r = (c7.red)
c8r = (c8.red)
c0g = (c0.green)
c1g = (c1.green)
c2g = (c2.green)
c3g = (c3.green)
c4g = (c4.green)
c5g = (c5.green)
c6g = (c6.green)
c7g = (c7.green)
c8g = (c8.green)
c0b = (c0.blue)
c1b = (c1.blue)
c2b = (c2.blue)
c3b = (c3.blue)
c4b = (c4.blue)
c5b = (c5.blue)
c6b = (c6.blue)
c7b = (c7.blue)
c8b = (c8.blue)
if
c0r > c1r then cv1r = c0r – c1r else cv1r = c1r – c0r
end
if
c0r > c2r then cv2r = c0r – c2r else cv2r = c2r – c0r
end
if
c0r > c3r then cv3r = c0r – c3r else cv3r = c3r – c0r
end
if
c0r > c4r then cv4r = c0r – c4r else cv4r = c4r – c0r
end
if
c0r > c5r then cv5r = c0r – c5r else cv5r = c5r – c0r
end
if
c0r > c6r then cv6r = c0r – c6r else cv6r = c6r – c0r
end
if
c0r > c7r then cv7r = c0r – c7r else cv7r = c7r – c0r
end
if
c0r > c8r then cv8r = c0r – c8r else cv8r = c8r – c0r
end
if
c0g > c1g then cv1g = c0g – c1g else cv1g = c1g – c0g
end
if
c0g > c2g then cv2g = c0g – c2g else cv2g = c2g – c0g
end
if
c0g > c3g then cv3g = c0g – c3g else cv3g = c3g – c0g
end
if
c0g > c4g then cv4g = c0g – c4g else cv4g = c4g – c0g
end
if
c0g > c5g then cv5g = c0g – c5g else cv5g = c5g – c0g
end
if
c0g > c6g then cv6g = c0g – c6g else cv6g = c6g – c0g
end
if
c0g > c7g then cv7g = c0g – c7g else cv7g = c7g – c0g
end
if
c0g > c8g then cv8g = c0g – c8g else cv8g = c8g – c0g
end
if
c0b > c1b then cv1b = c0b – c1b else cv1b = c1b – c0b
end
if
c0b > c2b then cv2b = c0b – c2b else cv2b = c2b – c0b
end
if
c0b > c3b then cv3b = c0b – c3b else cv3b = c3b – c0b
end
if
c0b > c4b then cv4b = c0b – c4b else cv4b = c4b – c0b
end
if
c0b > c5b then cv5b = c0b – c5b else cv5b = c5b – c0b
end
if
c0b > c6b then cv6b = c0b – c6b else cv6b = c6b – c0b
end
if
c0b > c7b then cv7b = c0b – c7b else cv7b = c7b – c0b
end
if
c0b > c8b then cv8b = c0b – c8b else cv8b = c8b – c0b
end
if
level > cv1r and level > cv2r and level > cv3r and
level > cv4r and level > cv5r and level > cv6r and
level > cv7r and level > cv8r then
dr = ((c1r + c2r + c3r + c4r + c5r + c6r + c7r + c8r )/8 ) else
dr = c0r end
if
level > cv1g and level > cv2g and level > cv3g and
level > cv4g and level > cv5g and level > cv6g and
level > cv7g and level > cv8g then
dg = ((c1g + c2g + c3g + c4g + c5g + c6g + c7g + c8g )/8 ) else
dg = c0g end
if
level > cv1b and level > cv2b and level > cv3b and
level > cv4b and level > cv5b and level > cv6b and
level > cv7b and level > cv8b then
db = ((c1b + c2b + c3b + c4b + c5b + c6b + c7b + c8b )/8 ) else
db = c0b end
cf = imlib2.color.new( dr, dg, db)
im:draw_pixel(x, y, cf)
end
end
im:save(“output.png”)

file name: blxblur.lua
Usage lua blxblur.lua reads input.png and prints output.png

Writing filters is easy, but the required text, to do it is very long, but
with some simple find and replace command and a logical way to write them,
you don`t type that much. believe me. I thought the code is long, but it
schould help people to learn how to write their first image processing
filters and thats why i posted it in full length!.
It is human readbale code you do use for source coede and the lua
interpreter will help you to find errors more quickly than any other
meothod ive tried so far. Its ten times faster than using bash for doing it
but also five times slower than a imagemagick function, however quickly
enough to convert images in a reasonable amount of time.

regards bluedxca93