When going through a lot of raw data like log data, I found it to be the best way to manipulate this data is using a Unix / Linux based operating system. The Unix / Linux operating system has a lot of powerful build in tools, these tools can be used to manipulate the data the way you need it to be. In this post I’m going to record all the command’s I use to manipulate the data, I will try to update this post when I use a new command. As far as I know these commands are Distribution independent.
Feedback or useful commands are more than welcome, they can be placed int he comments below.
Date | Action |
---|---|
2013-06-15 | Initial setup |
2013-07-19 | Revision of the post |
2013-08-19 | Added new sed command |
2013-11-28 | Added new sed command |
2014-01-23 | Correcting typo’s |
2015-05-21 | Added tables, command headers and corrected tyo’s |
. | awk Command |
Command awk -F ';' '{print $1, $3}' <File>
This tool will help you get selected data out of a text based file.
Standard AWK will use a space as a delimiter.
With the option -F ‘ ‘ you can set the delimiter manually to anything you need, put it in between the ‘ ‘.
Print will send it on your screen
With $<nr> you can select witch columss you want
. | sort Command |
Command sort <File> sort -u <File>
This tool will group every same number/name together and place them in order of a to z / 0 to 9. The -u option will tell it to make every combination unique.
. | wc Command |
Command wc <File> wc -l <File>
With this tool you can count words/rows/ etc.
-l option will give you the amount of rows
. | Pipe outcome to a file |
command > <File>
This option sends the output to a file instead of showing it on screen
. | cat Command |
Command cat <file>
This tool will print the entire output of that file on screen
. | more Command |
Command more <File>
This command will print the entire output like the CAT command but only till it fills the screen, you can/need to hit next to go to the following full screen of text.
. | SED Command |
Command sed -e 's/OLD/NEW/g' <File>
Replaces an OLD word for a NEW word, but you can replace or search on a blank(nothing).
Command
sed -e 's/\s/,/g' <File>
Replaces the spaces in the file with a comma, i use this to export files to excel.
Command
sed -e 's/(.)//g' <File>
Replaces (.) with nothing, the dot means a random character.
. | grep Command |
Command grep <String> <File>
This let’s you search for what you are looking for in the specific file, as an example “Grep 10.0.0.1 network.log”.
Ziet er goed uit,
Veel handige tips.