Drupal 8 – Composer: updating, module installs, & external library support notes

This is a set of configuration steps used to setup the requirements for a Drupal 8 website I’m working with using Drupal 8. PHP-Composer with Bower and NPM integration is used for updating and installing Drupal core, modules, and libraries. Included are procedures, configurations, and websites with information.

Prerequisites – May be necessary on the composer system used for installation and updates.

  1. Use npm to install bower for package management on system. In this case it is installed globally with by using the -g option. If npm is not installed you will need to install that first, so use your operating system specific procedures to install npm.
    npm install -g bower

All command-line commands noted below should be run from the Drupal root folder where the site is downloaded to in the step #1.

  1. Downloaded initially prepared Drupal website with git. This site has been prepared by another IT office and uses a Git repository.
  2. Update composer.json file to support bower and npm as additional css and javascript package managers.

    Under in the repositories section, add asset-packagist.org as an additional repository.
    Look for “repositories” and add (commas as needed):
    {
    "type": "composer",
    "url": "https://asset-packagist.org"
    }

  3. Update site to current available Drupal version using composer. Use following commands:
    composer update drupal/core --with-dependencies
    drush updatedb

    – if Drush is working otherwise look at /update.php
    drush cache:rebuild
    – if Drush is working otherwise look in admin configuration > performance.
  4. Install Drupal modules needed for development and production work:

    composer --update-with-dependencies require 'drupal/ctools'
    # Activated: Chaos Tools
    composer --update-with-dependencies require 'drupal/rabbit_hole’
    # Activated: Rabbit Hole, Rabbit hole nodes, rabbit hole taxonomy
    composer --update-with-dependencies require 'drupal/ds’
    # Activated: Display Suite, Display Suite Extras, Display Suite switch views.
    composer --update-with-dependencies require 'drupal/field_group'
    # Activate: Field Groups module
    composer --update-with-dependencies require 'drupal/token'
    # Activate: token module
    composer --update-with-dependencies require 'drupal/redirect'
    # Activate: redirect module
    composer --update-with-dependencies require 'drupal/entity_reference_revisions'
    # Activate: Entity Reference Revisions module
    composer --update-with-dependencies require 'drupal/autogrow'
    # Activate: Autogrow module
    composer --update-with-dependencies require 'drupal/metatag'
    # Activate: Metatag module
    composer --update-with-dependencies require 'drupal/imce'
    # Activate: IMCE File Manager module
    composer --update-with-dependencies require 'drupal/simplesamlphp_auth'
    # Activate: SimpleSAMLphp Authentication module.
    composer --update-with-dependencies require 'drupal/seckit'
    # Activate: Security Kit module
    composer --update-with-dependencies require 'drupal/securelogin'
    # Activate: Secure Login module
    composer --update-with-dependencies require 'drupal/antibot'
    # Activate: Antibot module
    composer --update-with-dependencies require 'drupal/honeypot'
    # Activate: Honeypot module
    composer --update-with-dependencies require 'drupal/webform'
    # Activate Webform module
    composer --update-with-dependencies require 'drupal/pathauto'
    # Activate: Pathauto module
    composer --update-with-dependencies require 'drupal/diff'
    # Activate Diff module
    composer --update-with-dependencies require 'drupal/slick’
    # Activate: Slick and Slick UI modules. You must enable the Blazy, Media modules to install Slick.
  5. Installing external libraries with Composer as written about at:
    https://www.drupal.org/docs/8/modules/webform/webform-frequently-asked-questions/how-to-use-composer-to-install-libraries
    – This should help eliminate the warnings: Webform: External libraries and lay foundation for external libraries into drupal-root: /libraries. External libraries are needed for the Webform and Slick Carousel modules.
  6. From drupal root directory in terminal use:
    composer require wikimedia/composer-merge-plugin
    – installs the composer merge plugin to install separate libraries.
    – (untested!) May no longer be necessary after executing step #2.
    If you install these plugins, you may need to add the following to the composer.json file:
    Look for “config” and add:
    "fxp-asset": {
    "installer-paths": {
    "npm-asset-library":
    "libraries", "bower-asset-library": "libraries"
    }
    }
  7. It may also be necessary to have the installer extender composer plugin installed:
    composer global require oomphinc/composer-installers-extender
  8. Edit Drupal root composer.json file to support npm and bower assets:
    Look for “extra”, add:
    – Added options may need commas for syntax.
    "installer-types": [
    "bower-asset",
    "npm-asset"
    ],

    – Adds bower and npm support.

    "merge-plugin": {
       "include": [
           "modules/contrib/webform/composer.libraries.json"
       ]
    },

    – informs composer to look in referenced composer.libraries.json file for installs.

    Look for “installer-paths” and confirm these additions or changes.
    “installer-paths”: {
    "core": [
    "type:drupal-core"],
    "libraries/slick": [
    "bower-asset/slick-carousel",
    "npm-asset/slick-carousel"
    ],
    "libraries/{$name}": [
    "type:drupal-library",
    "type:bower-asset"
    ]

    – Add /library folder location. You may need to create this location drupal-root:/libraries
  9. From terminal in drupal root:
    composer update –lock
    – update lock, usually finishes install

    Follow up install checks:
    composer update --with-dependencies
    – to perform full updates

Helpful info: https://www.drupal.org/project/slick/issues/2855190
More notes: https://www.drupal.org/project/lightning/issues/3014759
Troubleshoot composer bower integration: https://www.drupal.org/project/lightning/issues/3014759
Composer & Bower integration, path configurations: https://www.drupal.org/project/slick/issues/2907371#comment-12882235
Slick Carousel requirements with bower integration notes – has set of command-line commands.: https://git.drupalcode.org/project/slick/blob/8.x-1.x/README.md
Installing External libraries requited by a Drupal module: https://www.drupal.org/docs/7/modules/libraries-api/installing-an-external-library-that-is-required-by-a-contributed-module
Helpful information about Composer and Bower integration including npm, bower, composer installations: https://gist.github.com/macagua/d85f8790dba9c0ad2745926937b2d240

Install slick libraries with bowser integration.

This is an example to install the slick-carousel library using composer with bower integration.

composer require bower-asset/slick-carousel

To install specific versions use:

composer require bower-asset/slick-carousel:dev-master
— Note: some external javascript requited use of dev-master versions.
— It will be necessary to update the drupal-root:/composer.json file, and you should also check the required areas.

To check versions of installed packages with composer use:
composer show

Github – create token to download slick assets. From command-line output of above. Can also be regenerated at github.com.

Drupal 8 content type, view, block, twig notes

Large general tutorial video making a directory with content types and views:
https://www.webwash.net/how-to-build-directory-using-drupal-8/

PathAuto module information. For helping to make more readable URLs
https://www.webwash.net/automatically-generate-url-aliases-using-pathauto-drupal-8/

Using Entity References in Views – Great overview of using entity references.
https://www.ostraining.com/blog/drupal/drupal-8-entity-reference-view/

Drupal 8 theme twig template file reference. Includes various naming conventions for pages, views, blocks, nodes, etc.
https://www.drupal.org/docs/8/theming/twig/twig-template-naming-conventions


NPM and Yeoman update/install notes for MacOS

Had to deal with some issues introduced by bad homebrew installations and possible old NPM installs. My guess is that I probably cheated/screwed up using a sudo install when I should have done something different.

This all started playing with the WebDevStudions WordPress plugin generator for yeoman.

Get a Plugin Kickstart with Yeoman & generator-plugin-wp!

This gist was very helpful – Fixing npm On Mac OS X for Homebrew Users

https://gist.github.com/DanHerbert/9520689

Another strange issue was caused by installing the generator-plugin-wp yoeman generator on a broken yeoman/npm system. This posted github issue helped. https://github.com/npm/npm/issues/10995 This needed a straightforward uninstall, cache clear, and reinstall. I may have reintialized my terminal sessions to drop out of sudo mode. It looked something like:

$ sudo npm remove -g yo generator-plugin-wp
$ npm cache clean
-- can't remember if I relaunched terminal to drop out of sudo mode.
$ npm install -g generator-plugin-wp

I’ll be testing this yeoman generator on the side for a while: https://github.com/WebDevStudios/generator-plugin-wp

Spying on a directory with auditd

Files start coming up missing for me on a server and I get freaked out looking for security holes, but sometimes users and other utilities are spiking the bunch bowl. You can get serious with watching files with other utilities, but I went back to good ole auditd.

A simple test to track stuff getting trashed from an upload folder:

auditctl -w /site-dir/wp-content/uploads/ -p wa -k upload_issue

A capital W will remove the rule:

auditctl -W /site-dir/wp-content/uploads/ -p wa -k upload_issue

Do a quick search for issues with ausearch.

ausearch -f wp-content/uploads

Now permanently add the rule on a redhat system by putting this line in /etc/audit/audit.rules. Just leave off the auditctl command.

 -w /site-dir/wp-content/uploads/ -p wa -k upload_issue

Of course you need to make sure your auditd process is running and using chkconfig, etc. Good ole check status like:

/etc/init.d/auditd status

Here are a few of the resources I used:

Please forgive the RedHat auth-walls…

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Defining_Audit_Rules_and_Controls.html
http://stackoverflow.com/questions/29519590/monitor-audit-file-delete-on-linux
http://www.cyberciti.biz/tips/linux-audit-files-to-see-who-made-changes-to-a-file.html

My Favorite Httrack commands

HTTrack is a website mirroring utility that can swamp your disks with mirror copies of the internet. I’ve had to use it several times to make off-line copies of websites for all sorts of weird reasons. You’ll find HTTrack at: www.httrack.com. You can get a full list of command line options at: https://www.httrack.com/html/fcguide.html. There is a spiffy web and Windows wizard interface for HTTrack, but I gave that up.

This is the recipe for the command line options I’ve been using to produce a browse-able offline version of accreditation documents. This command says “Make an offline mirror of these URLs, go up to 8 links deep on these sites and 2 links deep on other domains. Stay on the TLD (.edu) and do it as quickly as possible. Be warned as it currently stands this will fill up about 1.5GB of disk space ;P.

httrack http://www.nicholls.edu/sacscoc-2016/ http://www.nicholls.edu/catalog/2014-2015/html/ http://www.nicholls.edu/about/ -O /Users/nichweb/web-test -r8 -%e1 -%c16 -*c16 -B -l -%P -A200000

The great part is that the archive grows as URLs are added.

Apache log one-liners using tail, awk, sort, etc.

Good bunch of samples with other examples found at: https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/

# top 20 URLs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS excluding POST data from the last 5000 hits
tail -5000 ./transfer.log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 IPs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLs requested from a certain ip from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 referrers from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
 
# top 20 user agents from the last 5000 hits
tail -5000 ./transfer.log | cut -d\  -f12- | sort | uniq -c | sort -rn | head -20
 
# sum of data (in MB) transferred in the last 5000 hits
tail -5000 ./transfer.log | awk '{sum+=$10} END {print sum/1048576}'

Using HyperDB to separate and share user and user_meta between WordPress installations

I need to remember to keep this example for some testing. This should be a good start for sharing a user and user_meta between websites. I do know that user_meta tends to have very site-centric settings at times. Original article was located at: http://wordpress.aspcode.net/view/63538464303732726666099/how-to-use-hyperdb-to-separate-and-share-a-user-dataset-between-wordpress-installs

$wpdb->add_database(array( //Connect to Users Database
    'host'     => DB_HOST, // I am using the same host for my two DBs
    'user'     => DB_USER, // I am using the same username for my two DBs
    'password' => DB_PASSWORD, // I am using the same p/w for my two DBs
    'name'     => 'my_user_db_name', 
    'write'    => 0, // Change to 1 if you want your slave site's the power to update user data.
    'read'     => 1,
    'dataset'  => 'user',
    'timeout'  => 0.2,
));

$wpdb->add_database(array( // Main Database
    'host'     => DB_HOST,
    'user'     => DB_USER,
    'password' => DB_PASSWORD,
    'name'     => DB_NAME,
));

$wpdb->add_callback('user_callback');
function user_callback($query, $wpdb) {
    if ( $wpdb->base_prefix . 'users' == $wpdb->table || $wpdb->base_prefix . 'user_meta' == $wpdb->table) {
        return 'user'; 
    }
}

Create a new Git repo from and old repo

How to extend an old repository as a full copy in a new repository. This preserves the history of the old repository. Future changes will not affect the old repository, but will be committed to the new repository.

This originally came from the info found at: http://stackoverflow.com/questions/10963878/how-do-you-fork-your-own-project-on-github

// This makes the new repo as a checkout of the old repo to a new directory.
# git clone https://github.com/nicholls-state-university/nicholls-2012-core.git nicholls-2015-core
// Change directory to new repo area
# cd nicholls-2015-core
// Change the origin to the new repo. Remember to make the new repo area.
# git remote set-url origin https://github.com/nicholls-state-university/nicholls-2015-core.git
// Push commits to new area.
# git push origin master
// Push all changes to repo, just making sure.
# git push --all

Git local repositories

These are some quick examples and notes related to using git with local repositories. Using local repositories can be helpful maintaining file changes without committing to larger repository systems like Github. Instead of syncing with a remote repository, synchronization and changes are committed to the local repository and recorded.

First we create a new local folder and initialize it as blank Git repository.

# mkdir my-local-git
# cd my-local-git
# git init —bare

Then we just clone that to the location we want and work on it like any other git repository.

# git clone /where/is/my-local-git

OSX turn off or delay hibernate

Apple’s Mac OSX hibernate can be delayed indefinitely or turned off by issuing this terminal command.

sudo pmset -a hibernatemode 0

To get back to default normal hibernate mode type this terminal command.

sudo pmset -a hibernatemode 3

So the word is this won’t disable hibernate on battery failures. Haven’t tested, hope I won’t have to.