When going through a lot of raw data like log data, I found it to be the best way to manipulate this data is using a Unix / Linux based operating system. The Unix / Linux operating system has a lot of powerful build in tools, these tools can be used to manipulate the data the way you need it to be. In this post I’m going to record all the command’s I use to manipulate the data, I will try to update this post when I use a new command. As far as I know these commands are Distribution independent.

Feedback or useful commands are more than welcome, they can be placed int he comments below.

 

Update tracking

Date	Action
2013-06-15	Initial setup
2013-07-19	Revision of the post
2013-08-19	Added new sed command
2013-11-28	Added new sed command
2014-01-23	Correcting typo’s
2015-05-21	Added tables, command headers and corrected tyo’s

 

.	awk Command

Command
awk -F ';' '{print $1, $3}' <File>

This tool will help you get selected data out of a text based file.

Standard AWK will use a space as a delimiter.
With the option -F ‘ ‘ you can set the delimiter manually to anything you need, put it in between the ‘ ‘.
Print will send it on your screen
With $<nr> you can select witch columss you want

 

.	sort Command

Command
sort <File>
sort -u <File>

This tool will group every same number/name together and place them in order of a to z / 0 to 9. The -u option will tell it to make every combination unique.

 

.	wc Command

Command
wc <File>
wc -l <File>

With this tool you can count words/rows/ etc.
-l option will give you the amount of rows

 

.	Pipe outcome to a file

command
> <File>

This option sends the output to a file instead of showing it on screen

 

.	cat Command

Command 
cat <file>

This tool will print the entire output of that file on screen

 

.	more Command

Command
more <File>

This command will print the entire output like the CAT command but only till it fills the screen, you can/need to hit next to go to the following full screen of text.

 

.	SED Command

Command
sed -e 's/OLD/NEW/g' <File>

Replaces an OLD word for a NEW word, but you can replace or search on a blank(nothing).

Command
sed -e 's/\s/,/g' <File>

Replaces the spaces in the file with a comma, i use this to export files to excel.

Command
sed -e 's/(.)//g' <File>

Replaces (.) with nothing, the dot means a random character.

 

.	grep Command

Command
grep <String> <File>

This let’s you search for what you are looking for in the specific file, as an example “Grep 10.0.0.1 network.log”.