User Tools

Site Tools


stewfix:prior_to_2105

DEPRECATED

This is promoted to a module as of May 2015! Please see stewfix instead

Leave these notes as background material.

In the early days I had to run many fixes ;)

stewfix

A set of scripts to make writing “quick fix it!” programs easier.

Handles authentication, logging, paging and all that housekeeping poop, so you can focus on what needs doing.

Has methods for traversing the database, the file system, accessing foreign databases and so forth.

Installing

It is not a module. I am too lazy to package it. Somebody else do it please.

There are three scripts. I put them in ./tools/stewfix off the webroot.

stewfix.header.php

<?php
/*
 * stewfix header

 * (c) 2013 steward at phpfox, stewfoxdev.com
 *
 *
 */

if(!defined('DENY_IP'))
{
    define('DENY_IP',true);
}
if(!defined('DENY_SUPERADMIN'))
{
    define('DENY_SUPERADMIN',true);
}


if(defined('DENY_IP') AND DENY_IP)
{
    switch($_SERVER['REMOTE_ADDR'])
    {
        case '127.0.0.1':               // localhost
        case '75.157.241.9':          // add your IPs
            break;
        default:
                die('IP not authorized: '.$_SERVER['REMOTE_ADDR']);
    }
}

/* Make sure we are running PHP5 */
if (version_compare(phpversion(), '5', '<') === true)
{
    exit('phpFox 2 or higher requires PHP 5 or newer.');
}

//ob_start();  usually gets in the way of tools. we want to see output immediately

/**
 * Key to include phpFox
 *
 */
define('PHPFOX', true);

/**
 * Directory Separator
 *
 */
define('PHPFOX_DS', DIRECTORY_SEPARATOR);

/**
 *  phpFox Root Directory
 *  Assume we are in tools/stewfix
 */
define('PHPFOX_DIR', dirname(dirname(dirname(__FILE__))) . PHPFOX_DS);

define('PHPFOX_START_TIME', array_sum(explode(' ', microtime())));

// Require phpFox Init
require(PHPFOX_DIR . 'include' . PHPFOX_DS . 'init.inc.php');

/*********************************************
 * Additional checks here to ensure this is only run by admin, staff, or some user
 *
if(defined('DENY_SUPERADMIN') AND DENY_SUPERADMIN)
{
    if (phpFox::getUserId() <2 )
    {
        die('UserId not authorized');
    }
}
**/

This script sets up the fox runtime environment.

If you move the script, you will need to add or remove “dirname” calls for the phpFox Root Directory.

The script checks for your IP. That's an important security measure to ensure ONLY you can run the job.

At the end you may also check for a particular user id.

stewfix.class.php

Here is the generic class. It provides a framework for the final utility to be run.

<?
/*
 * stewfix class

 * (c) 2013 steward at phpfox, stewfoxdev.com
 *
 *
 */

require_once('stewfix.header.php');


class stewfix
{
    protected $sTable,$iOffset,$iLimit,$bPreview;

    static $logtag;
    static $logfp;

    public function __construct($args=NULL)
    {
        $oreq = Phpfox::getLib('request');

        $this->iOffset  = $oreq->get('offset',0);
        $this->iLimit   = $oreq->get('limit', 20);
        $this->sTable   = $oreq->get('table','');
        $this->bPreview = $oreq->get('preview',true);
        $this->iErrors  = $oreq->get('errs',0);
        $this->iCount   = 0;
        self::$logtag   ='';
    }

    
    public function line($sFileName,$sLine)
    {
        $sFileName = STEWFIX_LOG_DIR . $sFileName; 
        $fp=fopen($sFileName,'a') OR DIE('Fail to open line '.$sFileName);
        @chmod($sFileName,0775);
        fputs($fp,$sLine."\n");
        fflush($fp);
        fclose($fp);
    }
    
    
    public function getLogFileName()
    {
        return STEWFIXLOG_FILENAME;
    }

    public function log_open()
    {
        // Open file for appending
        self::$logfp = self::$logfp=fopen($this->getLogFileName(),'a') OR DIE('Fail to open log '.$this->getLogFileName());

        @chmod($this->getLogFileName(),0775);
    }

    public function pageComplete()
    {
        // Did we do a full batch?
        return ($this->iCount == $this->iLimit);
    }
    public function incrementOffset()
    {
        // Get another batch
        $this->iOffset += $this->iLimit;
    }


    // Output list of variables (type and value)
    public function log()
    {
        $aArgs=func_get_args();

        $this->log_open();

        // Current timestamp
        $tm=date('Y-m-d H:i:s');

        $aLines=array();
        if(is_array($aArgs))
        {
            for($i=0;$i<count($aArgs);$i++)
            {
                $aLines[] = var_export($aArgs[$i],true);
            }
        }
        else
        {
            $aLines[]=var_export($aArgs,true);
        }
        fputs(self::$logfp,"\n\n$tm\n".implode("\n",$aLines)."\n");
        fflush(self::$logfp);
        fclose(self::$logfp);
    }


    /*
        For a filename with more than one dot, this will
        take only the last segment as the extension
    */
    static function parts($filePath)
    {
        $fileParts = pathinfo($filePath);
        return array(
            'dir' => $fileParts['dirname'],                                         // no trailing slash
            'base' => $fileParts['filename'],                                       // fred
            'ext' => (isset($fileParts['extension'])?$fileParts['extension']:''),   // txt
            'name' => $fileParts['basename']                                        // fred.txt
        );
    }
    static function getFileName($filePath)
    {
        $parts=static::parts($filePath);
        return $parts['name'];
    }
    static function getFileExt($filePath)
    {
        $parts=static::parts($filePath);
        return $parts['ext'];
    }
    static function change_ext($fn, $newext)
    {
        $f = self::parts($fn);
        if ( $f['dir']=='.' AND substr($fn,0,1) !='.' )
        {
            return $f['base'] . '.' . $newext;
        }
        if ( $f['dir']=='/' AND substr($fn,0,1) ==DIRECTORY_SEPARATOR )
        {
            $f['dir']='';
        }
        return $f['dir'].DIRECTORY_SEPARATOR.$f['base'].'.'.$newext;
    }


    public function run()
    {
        $aRows = $this->getRows();

        if(count($aRows)==0)
        {
            echo '<hr>Done!';
            return;
        }

        $this->iCount=0;
        foreach($aRows AS $row)
        {
            ++$this->iCount;
            if (! $this->processRow($row) )
            {
                die('<hr>Abort');
            }
        }

        // Did do a full batch?
        if ($this->pageComplete())
        {
            // Get another batch
            $this->incrementOffset();
            $this->goNextPage();
        }

        // optional post processing
        $this->afterRun();

        // count < limit we are done
        echo '<hr>Done!';
        return;
    }

    public function getNextPageUrl($script)
    {
        $url = 'http://' . Phpfox::getParam('core.host') . '/tools/stewfix/';
        $url .= $script. '?';
        $url .= 'offset='.$this->iOffset;
        $url .= '&preview='.$this->bPreview;
        $url .= '&errs='.$this->iErrors;
        return $url;
    }
    public function goNextPage()
    {
        // We can use hidden variables here to pass into the next page
        // But it seems more useful to be able to watch the url
        // So wait until we have a front end that can report progress before we go hidden.
        $url = $this->getNextPageUrl('');

        echo '
            <form method="POST" name="form" id="form" action="'.$url.'">
            </form>
            <script language=javascript>document.form.submit();</script>
            ';
        exit;
    }

    // A database select
    public function getRows()
    {
        return array();
    }
    // Do something with each row
    public function processRows()
    {
        return array();
    }

    public function afterRun()
    {
        return;
    }
}

FIX.php

The generic fix tool is the last script.

Here is a skeleton copy. Start with this file and rename it.

<?php
/*
 * FIX
 * (c) 2013 steward at phpfox, stewfoxdev.
 */

require_once('stewfix.class.php');

class fix extends stewfix
{

    public function __construct()
    {
        parent::__construct();
        $this->iLimit=200;
    }

    public function getRows()
    {
        $aRows = Phpfox::getLib('database')->select('user_name')
            ->from(Phpfox::getT('user'))
            ->limit($this->iOffset,$this->iLimit)
            ->order('user_id')
            ->execute('getSlaveRows');
        return $aRows;
    }

    public function processRow($row)
    {
        echo '<br> '.$row['user_name'];

        return true;
    }

    public function getNextPageUrl($script)
    {
        return parent::getNextPageUrl(basename(__FILE__));
    }

}

$dbfix=new fix();
$dbfix->run();

If you are following along so far, you have installed all three scripts. If you now point your browser at ./tools/stewfix/FIX.php you should see it cycle through your user list, 200 per page. It just echoes the user name to the screen.

BODYMASS.php

Using the scripts above, all we have left to write is the meat of our processing.

Here is a simple example. In this case I am reading two custom fields and creating a third.

I copy FIX.php to BODYMASS.php and (a) fuddle with the query, then (b) add the processing logic.

In this case I am creating a new user custom field called BMI (Body Mass Index) from two existing fields (height and weight).

<?php
/*
 * BODYMASS
 * (c) 2013 steward at phpfox, stewfoxdev.com
 *
 *  Create BMI (Body Mass Index) from height and weight fields
 */

require_once('stewfix.class.php');

class bodymass extends stewfix
{

    public function __construct()
    {
        parent::__construct();
        $this->iLimit=200;
    }

    public function getRows()
    {
        $aRows = Phpfox::getLib('database')->select('user_id, cf_height,cf_weight')
            ->from(Phpfox::getT('user_custom'))
            ->limit($this->iOffset,$this->iLimit)
            ->order('user_id')
            ->execute('getSlaveRows');
        return $aRows;
    }

    public function processRow($row)
    {
        echo '<br> '.$row['user_id'] . ' ' . $row['cf_height'] . ' ' . $row['cf_weight'];       ;


        // Extract height in cm
        $h = str_replace('user.cf_option_','', $row['cf_height']);
        preg_match('/([\d]+)_([\d]+)m/', $h, $aMatches);
        $m  = isset($aMatches[1]) ? $aMatches[1] : 0;
        $cm = isset($aMatches[2]) ? $aMatches[2] : 0;
        
        $h = ($m * 100) + ($cm * 10);  // height in cm

        if($h < 1)
        {
            return true;
        }

        // Extract weight in kilos
        $w = str_replace('user.cf_option_','', $row['cf_weight']);

        $w = (int) $w; // weight in kilos


        // 0.007716049382716 == 77.2
        $bmi = (int) (10000 * ($w / ($h * $h)));

        echo "  <b>H $h W $w  BMI $bmi</b>";

        $aFields=array(
            'cf_tall'=>$h,
            'cf_wide'=>$w,
            'cf_bmi'=> $bmi
        );

        Phpfox::getLib('database')->update(Phpfox::getT('user_custom'), $aFields, 'user_id='.$row['user_id']);

        return true;
    }

    public function getNextPageUrl($script)
    {
        return parent::getNextPageUrl(basename(__FILE__));
    }

}

$dbfix=new bodymass();
$dbfix->run();

Discussion

No time this morning, obviously this page needs to be fleshed out with a ton of discussion.

Kill inactive users

Removing inactive users is a case of deleting records wile paging through the table.

In this case, paging parameters must be adjusted to compensate. The explanation gets wordy and I'd need a test case to be sure I say the right things.

I do not have time to test this morning, so here is the idea, from a different fix script:

/**
 *      ALTERNATE PAGING TO USE WHEN JOB MAY HAVE ERRORS
 *      SOMETHING SIMILAR TO THIS MUST ALSO BE USED WHEN DELETING


// As we are changing the value that we query on, we always start the query at offset zero.
// But if we had errors, those same rows will keep coming up and eventually we'll hit the limit and be cycling endlessly.
// So make our offset the count minus the errors
// EG read 0,20; process 20; read 0,20; process 18; read 2,20; process 5; read 7,20; etc
// In the last read, 7 is the total errors encountered (2 on page 2, 5 on page 3)
// For this to work, there must be an ORDER clause in the query
public function incrementOffset()
{
// Get another batch
$this->iOffset = $this->iErrors;
}
 ***/

Conceptually not difficult. The Devil is in the details. Always run test cases on backup copies.

It amounts to adding a method to incrementOffset(), and adjusting the normal logic to compensate for deleted or cases.

Given that needs to be worked out, the script so far is:

<?php
/*
 * KILL-INACTIVE
 * steward 16 nov 14
 *
 *  Nuke inactive members.
 */

require_once('stewfix.class.php');


class killInactive extends stewfix
{

    public $tmLastOn;

    /*
     * Override limits etc here
     */
    public function __construct()
    {
        parent::__construct();

        // How many per page
        $this->iLimit=500;

        // Members who have not logged in for a year
        $this->tmLastOn = time() - (365 *24 *60 *60) ;
    }


    /*
     * The main query called once per page
     */
    public function getRows()
    {
        //stewlog::sql(true);

        $aRows = Phpfox::getLib('database')->select('u.user_id, u.user_name, u.email,u.joined,u.last_login')
            ->from(Phpfox::getT('user'),'u')
            ->join(PhpFox::getT('user_activity'), 'ua', 'ua.user_id = u.user_id' )
            ->where(
                'u.profile_page_id =0
                AND u.last_login < \'' . $this->tmLastOn .'\'
                AND u.user_group_id NOT IN (5,7,1)
                AND ua.activity_total =0'
            )
            ->limit($this->iOffset,$this->iLimit)
            ->order('u.last_login')
            ->execute('getSlaveRows');
        return $aRows;

    }


    /*
     * Process one row of the main query
     * Return true to continue, false to halt the job.
     */
    public function processRow($row)
    {
        echo '<br> '.$row['user_name'] . ' last on '.date ('Y-m-d', $row['last_login']);


        // Delete user UNTESTED
        /***
        $iId = $row['user_id'];
        Phpfox::getService('user.auth')->setUserId($iId);
        Phpfox::massCallback('onDeleteUser', $iId);
        Phpfox::getService('user.auth')->setUserId(null);
        ****/


        return true;
    }

    /*
     * Normal paging
     */
    public function getNextPageUrl($script)
    {

        // When writing the query, useful to stop at first page
        //die('<hr>End of first page');


        return parent::getNextPageUrl(basename(__FILE__));
    }

}

$dbfix=new killInactive();
$dbfix->run();

What else?

Here is a listing of my stewfix folder. Seems I have written quite a few fast fix programs for my system in the recent past.

ADMIN-NOTES-TO-PAYMENTS.php	USERORPHAN.php
ALLJPG-PHOTO.php		USERRELATED.php
ALLJPG-USER.php			VIDTHUMBS.php
BODYMASS.php			VIDVOTES.php
DIR2TABLE.php			stew.util.php
EXPIREALL.php			stewfix.class.php
GALLERY16.php			stewfix.header.php
GET-FACEB-MEMBERS.php		stewfix.jpg.class.php
GET-FACEB-TRANS.php		tconnect.php
KILLCURLY.php			tcurl.php
MAILTHREAD.php			tmail.php
PAYMENTS.php			tsecure.php
PHOTOCHECK.php			user_related_activity.php
PHOTODIR.php			user_related_count.php
USER-KILL-INACTIVE.php		user_related_custom.php
USER150.php			user_related_custom_value.php
USERCOPY.php			user_related_field.php
USERDELETECOMMENTS.php		user_related_space.php
USERFIX09.php			userwashere.php
USERINTEGRITY.php

cron.php		dbfixPRIVACY.php	getwaitlist.php
dbfixALLJPG.php		dbfixRATE.php		images.php
dbfixBLOGC.php		dbfixRATELIKE.php	importvbforum.php
dbfixCMV.php		dbfixTHUMB.php		jwtest.php
dbfixGALLERY.php	dbfixUSER.php		memcachetest.php
dbfixGROUPLIKE.php	dbfixUSERPHOTO.php	pattycurl.php
dbfixMAIL.php		dbfixUSRGRP.php		spitdbxml.php
dbfixNOPROBUMS.php	dbfixVID.php		stewfix
dbfixORPHANUSER.php	dbfixVIDTHUMB.php	tsecure.php
dbfixPAGELIKE.php	dbfixYOUTUBETHUMB.php	unitUSERPHOTO.php
dbfixPHOTO.php		findhack.php		webwolf
dbfixPHOTO50.php	findpics.php		xlat.php
dbfixPHOTOKILL.php	flowplayer
dbfixPHOTOPROFILE.php	genSASS.php

It's very handy for one-off fixes to my system !

Anything that can be driven by the database…

stewfix/prior_to_2105.txt · Last modified: 2015/04/22 17:24 by steward