Apache log one-liners using tail, awk, sort, etc.

Posted on by jess

Good bunch of samples with other examples found at: https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/

# top 20 URLs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS excluding POST data from the last 5000 hits
tail -5000 ./transfer.log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 IPs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLs requested from a certain ip from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 referrers from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
 
# top 20 user agents from the last 5000 hits
tail -5000 ./transfer.log | cut -d  -f12- | sort | uniq -c | sort -rn | head -20
 
# sum of data (in MB) transferred in the last 5000 hits
tail -5000 ./transfer.log | awk '{sum+=$10} END {print sum/1048576}'

Using HyperDB to separate and share user and user_meta between WordPress installations

Posted on by jess

I need to remember to keep this example for some testing. This should be a good start for sharing a user and user_meta between websites. I do know that user_meta tends to have very site-centric settings at times. Original article was located at: http://wordpress.aspcode.net/view/63538464303732726666099/how-to-use-hyperdb-to-separate-and-share-a-user-dataset-between-wordpress-installs

$wpdb->add_database(array( //Connect to Users Database
    'host'     => DB_HOST, // I am using the same host for my two DBs
    'user'     => DB_USER, // I am using the same username for my two DBs
    'password' => DB_PASSWORD, // I am using the same p/w for my two DBs
    'name'     => 'my_user_db_name', 
    'write'    => 0, // Change to 1 if you want your slave site's the power to update user data.
    'read'     => 1,
    'dataset'  => 'user',
    'timeout'  => 0.2,
));

$wpdb->add_database(array( // Main Database
    'host'     => DB_HOST,
    'user'     => DB_USER,
    'password' => DB_PASSWORD,
    'name'     => DB_NAME,
));

$wpdb->add_callback('user_callback');
function user_callback($query, $wpdb) {
    if ( $wpdb->base_prefix . 'users' == $wpdb->table || $wpdb->base_prefix . 'user_meta' == $wpdb->table) {
        return 'user'; 
    }
}

Create a new Git repo from and old repo

Posted on by jess

How to extend an old repository as a full copy in a new repository. This preserves the history of the old repository. Future changes will not affect the old repository, but will be committed to the new repository.

This originally came from the info found at: http://stackoverflow.com/questions/10963878/how-do-you-fork-your-own-project-on-github

// This makes the new repo as a checkout of the old repo to a new directory.
# git clone https://github.com/nicholls-state-university/nicholls-2012-core.git nicholls-2015-core
// Change directory to new repo area
# cd nicholls-2015-core
// Change the origin to the new repo. Remember to make the new repo area.
# git remote set-url origin https://github.com/nicholls-state-university/nicholls-2015-core.git
// Push commits to new area.
# git push origin master
// Push all changes to repo, just making sure.
# git push --all

Git local repositories

Posted on by jess

These are some quick examples and notes related to using git with local repositories. Using local repositories can be helpful maintaining file changes without committing to larger repository systems like Github. Instead of syncing with a remote repository, synchronization and changes are committed to the local repository and recorded.

First we create a new local folder and initialize it as blank Git repository.

# mkdir my-local-git
# cd my-local-git
# git init —bare

Then we just clone that to the location we want and work on it like any other git repository.

# git clone /where/is/my-local-git