Semaphores to avoid critical task collision

This is a sample from my cakephp shell script that has some critical processing. It is ran every second and sometimes the script can take over a second to process. Thus, I needed a way to avoid the script from accessing the same critical business logic and manipulating data.

The semaphore works by acquiring the key with the given id: 1068. I have set the max number of semaphores to 1 for the key. All other shell scripts that are ran will autorelease when they cannot acquire the semaphore (hence another shell script is still processing).

/**
* Semaphore values
*/
private $sema_key = 1068;
private $sema_maxAcquire = 1;
private $sema_permissions = 0666;
private $sema_autoRelease = 1;

public function main() {
// Acquire our lock
   $semaphore = sem_get($this->sema_key, $this->sema_maxAcquire, $this->sema_permissions, $this->sema_autoRelease);
   if (!$semaphore) {
      throw new InternalErrorException('Error getting semaphore');
   }
   sem_acquire($semaphore);

   // DO LOGIC: CRITICAL TASK

   // Release the semaphore for other processes
   sem_release($semaphore);
}

Forking a running process and returning values

Using pcntl_fork() allows for forking of a running processing in PHP. In this example I fork my shell script and process data and return the results. This would be useful in a case where you had to make api calls to lets say 4 vendors: amazon, itunes, youtube, flickr. You would assign each child id a task to run. They each would then write there results to a shared memory location for further manipulation. I have written this test in CakePHP shell script.

class ForkShell extends AppShell {

    public function main() {

        $total = 0;
        $data = array();
        $result = array();
        $start = (float) array_sum(explode(' ', microtime()));

        echo "Parent PID: ".getmypid().PHP_EOL;

        $this->forkTest($data);

        // Manipulate the results from the fork
        foreach ($data as $key => $value) {
            $total = $total + $value['number'];
        }
        
        $result['sum_of_digits'] = $total;
        $result['data'] = $data;

        echo "Processing time: ".sprintf("%.4f", ((float) array_sum(explode(' ', microtime())) - $start))." seconds".PHP_EOL;
        print_r(Set::reverse($result));
    }

    public function forkTest(&$data = array()) {

        $pids = array();
        $parent_pid = getmypid();

        for ($i = 0; $i < 4; $i++) {
            if (getmypid() == $parent_pid) {
                $pids[] = pcntl_fork();
                echo "Forking child, now has ".count($pids)." elements".PHP_EOL;
            }
        }

        if (getmypid() == $parent_pid) {
	  		 
            // Process children results as they exit, but not before
            while (count($pids) > 0) {

                echo "Parent id: ".getmypid().PHP_EOL;

                // Wait for child to complete / maintains consistency
                $pid = pcntl_waitpid(-1, $status);
                // Open shared memory block (read only)
                $shm_id = shmop_open($pid, "a", 0, 0);
                // Read data based on chunk size to local variable
                $shm_data = unserialize(shmop_read($shm_id, 0, shmop_size($shm_id)));
                // Delete shared memory block
                shmop_delete($shm_id);
                // Close shared memory block
                shmop_close($shm_id);
                // Merge data to memory location of data
                $data = array_merge($data, $shm_data);
                /* Remove all PID entries created */
                foreach ($pids as $key => $tpid) {
                    if ($pid == $tpid) {
                        unset($pids[$key]);
                    }
                }
            }
            $pids = array();
            
        } else {

            echo "Child id: ".getmypid().PHP_EOL;

            // From here you could do API calls or other calculations and bring the results together
            $pdata = array();
            array_push($pdata, array(
                'child_id' => getmypid(),
                'number' => rand(5, 15)
            ));
            $data_str = serialize($pdata);

            // Open shared memory location
            $shm_id = shmop_open(getmypid(), "c", 0644, strlen($data_str));
            
            // attempt to write to shared memory
            if (!$shm_id) {
                exit("Couldn't create shared memory segment");
            } else if (shmop_write($shm_id, $data_str, 0) != strlen($data_str)) {
                exit("Couldn't write shared memory data");
            }

            exit(0);
        }
    }
}

I would not recommend running this script inside of CakePHP. It may cause issues with the caching.

Couchbase Datasource

I ran into a problem where I needed to utilize a NoSQL database. Since CakePHP is more fit for relational style database (MYSQL) I had to create a custom datasource. The following datasource allows models to have basic functionality of Couchbase Server.

Example Usage:

$keyName = "Jackson";
$cache = $this->YourModel->Get(array($keyName));
if (!$cache) {
   $data = array('Firstname' => 'Jessie',
                 'Lastname' => 'Jackson');
   $this->YourModel->Assign(array($keyName), json_encode($data));
}

Setup / Configuration:

To use the couchbaseSource you must specify it in your model. Do so by adding the following.
YourModel.php

public $useDbConfig = ‘bucketCB';

You can also programmatically change datasouces / database on the fly. For example, sometimes I want to use my default MYSQL datasource. This can be accomplished like so:

if($useNOSQL) {
   $this->Model->useDbConfig = 'bucketCB';
}else {
   $this->Model->useDbConfig = 'default'; // MYSQL
}

To use be able to use this datasource it must be added to cakePHP list of available datasources.
Database.php

public $bucketCB = array(
    'datasource' => 'CouchbaseSource',
    'username' => ‘bucketUsername,
    'password' => ‘bucketPassword’,
    'bucket' => ‘bucketName,
    'prefix' => ‘p’, // A _ is automatically prepended
    'expiry' => '1814400', // 3 Weeks
    'autoConnect' => true,
    'database' => NULL,
    'persistent' => false
);

app/Model/Datasource/CouchbaseSource.php

/**
 * Couchbase Datasource class
 * @Author Brandon Klimek
 * 
 */
class CouchbaseSource extends DataSource {

    /**
     * Description of datasource
     *
     * @var string
     */
    public $description = 'Couchbase DataSource';

    /**
     * Holds the object for the connected database
     *
     * @var object
     */
    public $conObject = NULL;

    /**
     * Holds the configuration settings that are passed in
     *
     * @var array
     */
    public $config = NULL;

    /**
     * The prefix of the couchbase keys
     *
     * @var string
     */
    public $prefix = NULL;

    /**
     * CouchDBSource Constructor
     *
     * @param array $config The configuration for the Datasource
     *
     * @return void
     * @link http://api.cakephp.org/class/data-source#method-DataSource__construct
     */
    public function __construct($config = array()) {

        // If no configuration is set we use the default
        $this->config = $config;

        // Setup the cache string that is used when building the string
        $this->prefix = (isset($this->config['prefix']) ? $this->config['prefix'] . "_" : "");

        if ($this->config['autoConnect']) {
            $this->connect();
        }
    }

    /**
     * Connect to the Datasource
     *
     * @return obj
     * @throws InternalErrorException
     */
    public function connect() {
        if ($this->conObject !== true) {
            try {
                $this->conObject = new Couchbase("127.0.0.1:8091", $this->config['username'], $this->config['password'], $this->config['bucket'], $this->config['persistent']);
            } catch (CouchbaseException $e) {
                throw new InternalErrorException(array('class' => $e->getMessage()));
            }
        }
        return $this->conObject;
    }

    /**
     * Handle queries to couchbase
     *
     * @param unknown_type $method
     * @param array() $param
     * @return array or false
     */
    public function query($method, $params) {

        // If not connected... reconnect!
        if ($this->conObject === NULL) {
            $this->connect();
        }

        $apiMethod = $this->__methodToClass($method);
        if (!method_exists($this, $apiMethod)) {
            throw new NotFoundException("Class '{$apiMethod}' was not found");
        } else {
            return call_user_func_array(array($this, $apiMethod), $params);
        }
    }

    /**
     * Translate method to className
     *
     * @param $method
     * @return string
     */
    private function __methodToClass($method) {
        return 'CB' . strtolower(Inflector::camelize($method));
    }

    /**
     * describe() tells the model your schema for ``Model::save()``.
     *
     * You may want a different schema for each model but still use a single
     * datasource. If this is your case then set a ``schema`` property on your
     * models and simply return ``$Model->schema`` here instead.
     */
    public function describe(&$Model) {
        return $this->description;
    }

    /////////////////////////////////////////////////
    // Query Methods
    /////////////////////////////////////////////////

    /**
     * Add a value with the specified key that does not already exist. Will fail if the key/value pair already exist.
     *
     * @return Contains the document ID or false if the operation failed
     */
    public function CBadd($key = NULL, $value = NULL, $expiry = NULL, $persisto = NULL, $replicateto = NULL) {
        return $this->conObject->add($key, $value, $expiry, $persisto, $replicateto);
    }

    /**
     * Append a value to an existing key
     *
     * @return scalar ( Binary object )
     */
    public function CBappend($key = NULL, $value = NULL, $expiry = NULL, $persisto = NULL, $replicateto = NULL) {
        return $this->conObject->append($key, $value, $expiry, $persisto, $replicateto);
    }

    /**
     * Compare and set a value providing the supplied CAS key matches
     *
     * @return scalar ( Binary object )
     */
    public function CBcas($casimoqie = NULL, $key = NULL, $value = NULL, $expiry = NULL) {
        return $this->conObject->cas($casimoqie, $key, $value, $expiry);
    }

    /**
     * Decrement the value of an existing numeric key. The Couchbase Server stores numbers as unsigned values. Therefore the lowest you can decrement is to zero.
     *
     * @return scalar ( Binary object )
     */
    public function CBdecrement($key = NULL, $offset = NULL) {
        return $this->conObject->decrement($key, $offset);
    }

    /**
     * Delete a key/value
     *
     * @return scalar ( Binary object )
     */
    public function CBdelete($key = NULL, $offset = NULL) {
        $this->conObject->delete($key, $offset);
    }

    /**
     * Wait until the durability of a document has been reached
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBkeyDurability($key = NULL, $casunique = NULL) {
        return $this->conObject->keyDurability($key, $casunique);
    }

    /**
     * Wait until the durability of a document has been reached
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBflush() {
        return $this->conObject->flush();
    }

    /**
     * Get a value and update the expiration time for a given key
     *
     * @return obj
     */
    public function CBgetAndTouch($key = NULL, $expiry = NULL) {
        return $this->conObject->getAndTouch($key, $expiry);
    }

    /**
     * Get a value and update the expiration time for a given key
     *
     * @return obj
     */
    public function CBgetAndTouchMulti($key = NULL, $expiry = NULL) {
        return $this->conObject->getAndTouchMult($key, $expiry);
    }

    /**
     * Fetch the next delayed result set document
     *
     * @return array ( Result list )
     */
    public function CBfetch($key = NULL, $keyn = NULL) {
        return $this->conObject->fetch($key, $keyn);
    }

    /**
     * Fetch all the delayed result set documents
     *
     * @return array ( Result list )
     */
    public function CBfetchAll($key = NULL, $keyn = NULL) {
        return $this->conObject->fetchAll($key, $keyn);
    }

    /**
     * Get one or more key values
     *
     * @return scalar ( Binary object )
     */
    public function CBget($key = NULL, $callback = NULL, $casunique = NULL) {

        if (is_array($key)) {
            $key = $this->buildCacheString($key);
        }
        return $this->conObject->get($key, $callback, $casunique);
    }

    /**
     * Get one or more key values
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBgetDelayed($keyn = NULL, $with_cas = NULL, $callback = NULL) {
        return $this->conObject->getDelayed($keyn, $with_cas, $callback);
    }

    /**
     * Get one or more key values
     *
     * @return array ( Array of documents )
     */
    public function CBgetMulti($keycollection = NULL, $casarray = NULL) {
        return $this->conObject->getMulti($keycollection, $casarray);
    }

    /**
     * Returns the version of the client library
     *
     * @return scalar ( Binary object )
     */
    public function CBgetClientVersion() {
        return $this->conObject->getClientVersion();
    }

    /**
     * Get the value for a key, lock the key from changes
     *
     * @return (none)
     */
    public function CBgetAndLock($key = NULL, $casarray = NULL, $getlexpiry = NULL) {
        return $this->conObject->getAndLock($key, $casarray, $getlexpiry);
    }

    /**
     * Get the value for a key, lock the key from changes
     *
     * @return (none)
     */
    public function CBgetAndLockMulti($keycollection = NULL, $casarray = NULL, $getlexpiry = NULL) {
        return $this->conObject->getAndLockMulti($keycollection, $casarray, $getlexpiry);
    }

    /**
     * Returns the number of replicas for the configured bucket
     *
     * @return scalar ( Number of replicas )
     */
    public function CBgetNumReplicas() {
        return $this->conObject->getNumReplicas();
    }

    /**
     * Retrieve an option
     *
     * @return mixed ( Different possible types )
     */
    public function CBgetOption($option) {
        return $this->conObject->getOption($option);
    }

    /**
     * Returns the versions of all servers in the server pool
     *
     * @return array ( List of things )
     */
    public function CBgetVersion() {
        return $this->conObject->getVersion();
    }

    /**
     * Execute a view request
     *
     * @return (none)
     */
    public function CBview($ddocname = NULL, $viewname = NULL, $viewoptions = array()) {
        return $this->conObject->view($ddocname, $viewname, $viewoptions);
    }

    /**
     * Generate a view request, but do not execute the query
     *
     * @return (none)
     */
    public function CBviewGenQuery($ddocname = NULL, $viewname = NULL, $viewoptions = NULL) {
        return $this->conObject->viewGenQuery($ddocname, $viewname, $viewoptions);
    }

    /**
     * Increment the value of an existing numeric key. Couchbase Server stores numbers as unsigned numbers, therefore if 
     * you try to increment an existing negative number, it will cause an integer overflow and return a non-logical numeric 
     * result. If a key does not exist, this method will initialize it with the zero or a specified value.
     *
     * @return scalar ( Binary object )
     */
    public function CBincrement($key = NULL, $offset = NULL, $create = NULL, $expiry = NULL, $initial = NULL) {
        return $this->conObject->increment($key, $offset, $create, $expiry, $initial);
    }

    /**
     * Get the durability of a document
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBobserve($key = NULL, $casunique = NULL, $observeddetails = NULL) {
        return $this->conObject->observe($key, $casunique, $observeddetails);
    }

    /**
     * Get the durability of a document
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBobserveMulti($keycollection = NULL, $observeddetails = NULL) {
        return $this->conObject->observeMulti($keycollection, $observeddetails);
    }

    /**
     * Prepend a value to an existing key
     *
     * @return scalar ( Binary object )
     */
    public function CBprepend($key = NULL, $value = NULL, $expiry = NULL, $casunique = NULL, $persistto = NULL, $replicateto = NULL) {
        return $this->conObject->prepend($key, $value, $expiry, $casunique, $persistto, $replicateto);
    }

    /**
     * Update an existing key with a new value
     *
     * @return scalar ( Binary object )
     */
    public function CBreplace($key = NULL, $value = NULL, $expiry = NULL, $casunique = NULL, $persistto = NULL, $replicateto = NULL) {
        return $this->conObject->replace($key, $value, $expiry, $casunique, $persistto, $replicateto);
    }

    /**
     * Store a value using the specified key, whether the key already exists or not. Will overwrite a value if the given key/value already exists.
     *
     * @return scalar ( Binary object )
     */
    public function CBassign($key = NULL, $value = NULL, $expiry = NULL, $casunique = NULL, $persistto = NULL, $replicateto = NULL) {

        if (is_array($key)) {
            $key = $this->buildCacheString($key);
        }

        $time = (($expiry == NULL) ? $this->config['expiry'] : $expiry);

        try {
            $this->conObject->set($key, $value, $time, $casunique, $persistto, $replicateto);
        } catch (Exception $e) {
            return false;
        }
        return true;
    }

    /**
     * Set multiple key/value items at once
     *
     * @return scalar ( Binary object )
     */
    public function CBassignMulti($kvarray = NULL, $expiry = NULL) {
        return $this->conObject->setMulti($kvarray, $expiry);
    }

    /**
     * Specify an option
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBsetOption($option = NULL, $mixed = NULL) {
        return $this->conObject->setOption($option, $mixed);
    }

    /**
     * Get the database statistics
     *
     * @return array ( List of things )
     */
    public function CBgetStats() {
        return $this->conObject->getStats();
    }

    /**
     * Update the expiry time of an item
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBtouch($key = NULL, $expiry = NULL) {
        return $this->conObject->touch($key, $expiry);
    }

    /**
     * Change the expiration time for multiple documents
     *
     * @return boolean ( Boolean (true/false) )
     */
    public function CBtouchMulti($keyarray = NULL, $expiry = NULL) {
        return $this->conObject->touchMulti($keyarray, $expiry);
    }

    /**
     * Private buildCacheString
     *
     * Takes in a array and then splits it into a string and underscores
     *
     * @param array resetSplit
     * @return string
     */
    private function buildCacheString($restSplit = array()) {

        $count = count($restSplit);
        $string = "";
        foreach ($restSplit as $key => $method) {
            $string .= Inflector::slug($method);
            if ($count - 1 != $key) {
                $string .= "_";
            }
        }
        return strtolower($this->prefix . $string);
    }

}

Converting a json object to an array

Using restful web services relies on json results. You need your json object to be decoded and then converted to a array. Using the set library from CakePHP we can accomplish this. In this example I send a message to Android push gateway, then attempt to read the result.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://android.googleapis.com/gcm/send');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'Authorization: key= ENTER_YOUR_KEY',
    'Content-Type: application/json'
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(array(
   'registration_ids' => $registation_ids,
   'data' => array("m" => $message),
)));

$result = curl_exec($ch);
if ($result === FALSE) {
   // do error handling
   debug('curl request failed');
}

// The json decoding / converting line
$data = Set::reverse(json_decode($jsonresult));

The data variable now contains an array of information ready to parse.

Memcached optimization

Swapping out CakePHP’s default file logging for memcache can greatly improve performance. In my case I use couchbase server to manage NoSQL. From there administrative panel you can see every time your app has a page refresh.

Memcached optimization with CakePHP 2.x

To connect CakePHP with memcached place this in your app/Config/core.php

Cache::config('default', array(
	'engine' => 'Memcache',
	'prefix' => Inflector::slug(APP_DIR) . '_',
	'servers' => array(
			'127.0.0.1:11211' // localhost, default port 11211
		), //[optional]
	'duration' => 7200,
	'serialize' => false,
	'persistent' => true, // [optional] set this to false for non-persistent connections
	'compress' => false,
	'probability' => 100 
	)
);

Your application will now benefit from using memcache instead of a standard logging system. In the event that you make major changes to you application you may need to “flush” the memcache bucket to clear the cached schema stored by CakePHP.

Running shells as repeating cronjobs

It is common to use shells to handle background tasks on your server. Sometimes that task needs to be executed as quickly as possible. For example, you have a queue of tasks that need to process as soon as entered.

CakePHP can be configured on Unix servers to run a modified shell script via crontab.

Begin by modifying the defaultShell that comes with cake. I have created a new script call secondShell.

#!/bin/bash
TERM=dumb
export TERM
cmd="cake"
while [ $# -ne 0 ]; do
   if [ "$1" = "-cli" ] || [ "$1" = "-console" ]; then
      PATH=$PATH:$2
   shift
   else
      cmd="${cmd} $1"
   fi
   shift
done
COUNT=0
while [ $COUNT -le 30 ]; do
   $cmd
   let COUNT=COUNT+1
   sleep 2;
done

The modifications make this shell run ever 2 seconds for 30 occurrences (2 x 30 = 60 seconds)

To run this shell place it in the crontab by using command “crontab -e” in terminal.

*/1     *       *       *       *       /(your path)/app/Lib/Shells/secondShell YourShell -cli /usr/bin -console /path/to/your/app/Console -app /(your path)/app >/dev/null 2>&1

The */1 tells the scheduler to run the CakePHP Shell script every minute. Learn more about cron job scheduling

Wildcard Subdomain Prerouting

Ever have a need to have wildcard subdomains on your CakePHP application? This is the solution to that problem. Assume all subdomains have there own template folder, and the subdomains need to be able to be created and modified on the fly.

Create a subdomain model, as well as a table in your database called Subdomains with the following schema

-----------------
MYSQL Subdomain Schema
-----------------
sub_id         int (11)
sub_name       varchar (255)
sub_template   varchar (255)
sub_active     tinyint(1)

Initially, the URL is used to extract the subdomain and the host name from the clients entry point. The contents are then assigned to an array in a custom variable using the internal cache. The bootstrap is called within the dispatcher stage of the MVC architecture as shown below.

CakePHP MVC Structure

Bootstrap.php

preg_match('/^(?:www\.)?(?:(.+)\.)?(.+\..+)$/i', env('HTTP_HOST'), $urlmatches); 
Configure::write('SubdomainHTTP', array('subdomain' => empty($urlmatches[1]) ? false : $urlmatches[1], 'hostURL' => empty($urlmatches[2]) ? false : $urlmatches[2]));

The subdomain is now passed through the application and can be accessed by configure::read. A custom library called SubdomainRoute writes values to the cache and the router then reads the results. If a template is set after the SubdomainRoute is initialized it assumes the client is accessing our application on a valid subdomain found in the database.

Routes.php

App::uses('SubdomainRoute', 'Routes');

if (Configure::read('SubdomainHTTP.subdomain')) {
    $subdomainRouting = new SubdomainRoute();
}

if (Configure::read('Subdomain.sub_template’) != NULL) {
    Router::connect('/', array('controller' => ‘pages’, 'action' => ‘index’));
}

The subdomain is determined by accessing the mysql subdomain table. The results are stored and later used for routing (above), as well as assigning a template folder for a particular subdomain (below).

app/Lib/Routes/SubdomainRoute.php

App::uses('Subdomain', 'Model');

class SubdomainRoute extends CakeRoute {

    /**
     * Name of the subdomain to use
     * 
     * @var
     */
    private $subdomain = NULL;

    /**
     * Overrides the routes constructor not to use templates
     */
    public function __construct() {

        $this->subdomain = Configure::read('SubdomainHTTP.subdomain');

        if ($this->subdomain != false) {
            $this->setSubdomain();
        }
    }

    /**
     * Determine subdomain (assoc_id), set to global var
     *
     * @return null
     */
    private function setSubdomain() {

        $Subdomain = new Subdomain();
        $subdomain = $Subdomain->find("first", array(
            'conditions' => array('Subdomain.sub_name' => $this->subdomain, 'Subdomain.sub_active' => '1'),
            'recursive' => -1,
            'fields' => array('Subdomain.sub_name', 'Subdomain.sub_template')
                )
        );

        if (!isset($subdomain['Subdomain']['sub_bus_id'])) {
            throw new BadRequestException('The subdomain specified does not exist.');
        }

        Configure::write('Subdomain', array(
            'sub_name' => $subdomain['Subdomain']['sub_name'],
            'sub_template' => $subdomain['Subdomain']['sub_template']
                )
        );
    }
}

The final step is to tell cake which template the subdomain is associated too. This assumes that you have created the templates folder in the directory.

AppController.php

private function setTheme() {

    $subdomainDetails = Configure::read('Subdomain');
    if (isset($subdomainDetails['sub_template'])) {
        return $this->theme = ucfirst($subdomainDetails['sub_template']);
    }
    return $this->theme = “default”;
}

Sixthpoint

web development blog

CakePHP