Hello! Welcome to my humble web presence. I'm Mark Hamstra, 29 years young, wearer of many hats at modmore, resident of Leeuwarden, the Netherlands. Most of my time is spent building and maintaining awesome extras and tools at modmore, but I also love gaming, cooking, and my dogs. More about me.

This site is where I share thoughts, cool projects and other oddities related to MODX, xPDO and ExtJS. I also write about MODX regularly over at MODX.today. Sometimes I post three blogs in a week, sometimes there's nothing new here in a year. Read a random article.


When did you last check the size of your modx_session database table? Was it huge, like gigabytes huge? If so, you're not alone. 

To understand the problem, you need a little bit of background.

How do sessions work in PHP?

Typically, the standard PHP session handler will create a session when requested, and store it as a simple file somewhere on the system. The path it writes session to is configured with the session.save_path configuration in the php.ini, and can point anywhere. When that option is empty, it writes to the temp directory on the server. 

Creating and loading sessions is simple enough, but the next thing the session handler does is clean up sessions. This is called garbage control, or gc. This removes sessions beyond their expiration time, to make sure it doesn't keep growing indefinitely and takes up your vital disk space. 

Garbage control doesn't have to run on every request. If your session/cookie life time is configured to a week and you're not too picky about the exact timing they are removed, then sessions only really need to be checked once a day. Cleaning up sessions can take a little time and resources, so PHP is by default configured to only do that once every 100 requests.  

How do sessions work in MODX?

MODX registers a custom session handler that takes care of storing, retrieving, and cleaning up sessions. It writes this to one of its own database tables (modx_session), rather than files. This allows MODX a little more control over the flow of sessions. 

It is also possible to instruct MODX to use standard PHP sessions, and there's also an extra available to write sessions to Redis. But the vast majority of sites will simply be using the default, writing to the database.

So, why does MODX not clean up its session table?

MODX awaits the signal from PHP that it's time to clean up sessions. This relies on 2 configuration options in PHP:

  • session.gc_probability
  • session.gc_divisor

You can find the official documentation for those options here.

Usually the probability is set to 1, and divisor to a value like 100 or 1000. That means that in approximately 1/100 requests, the garbage control will run. 

When MODX does not seem to be cleaning up its table, it's usually because of an attempt to improve the performance of the garbage collection, by bypassing PHP and off-loading it to a cron job that runs outside of the request/response cycle. 

Those environments assume PHP writes its sessions to a standard location in the filesystem, and clean up that directory based on the timestamp on the file. The session.gc_probability option is then set to 0, to tell PHP to never run its own session garbage collection.

That works great - if your sessions are written to the standard location. Which MODX doesn't. 

How common is this?

Based on data from SiteDash, which checks the size and status of your session table automatically, it's pretty common indeed. Out of a sample of 1727 sites, 27% seem to be affected by this.

How can I fix this?

Re-enable session.gc_probability. Set it to 1, and make sure session.gc_divisor is also set properly for your traffic.

Depending on your host and if you have access to a server control panel, you may be able of changing it yourself. In other cases, contact your host and ask them how it should be changed. 

For a recent extra, I needed to get some arbitrary data into a package, in such a way that it's available for both the setup.options.php and the resolver - without duplicating that data. Specifically, it's a big array containing definitions for a theme with assets and elements that needed to be manually created/updated only if the user choose to do so.

After some time, I found a way to do that using the package attributes. And in this article I'll show you how to add that to a standard build.

Define the data

First, define the data. I created the file _build/data/theme.inc.php to return an array, but you can place it where it makes most sense. The file will only be accessed when building the package, so does not have to be in the core or assets folder (although it could be, if that makes sense for your use case).

<?php

$def = [
    // a whole bunch of data and elements
];

return $def;

Add the data to the package attributes

The package attributes is a special part of the package, which gets stored in the package manifest rather than a vehicle. It's used to hold some standard information: package name, changelog, readme, and license, among others.

In a standard build script the code to set the package attributes looks something like this:

<?php
// ... lots of other code ...
$builder->setPackageAttributes([
    'license' => file_get_contents($sources['docs'] . 'license.txt'),
    'readme' => file_get_contents($sources['docs'] . 'readme.txt'),
    'changelog' => file_get_contents($sources['docs'] . 'changelog.txt'),
    'setup-options' => [
        'source' => $sources['build'] . 'setup.options.php',
    ],
]);

It turns out though - package attributes are not limited to those standard items. Any attribute will be stored into the package manifest.

Let's take advantage of that by adding our own attribute containing (in this case) a theme-definition from our file:

<?php
$builder->setPackageAttributes([
    'license' => file_get_contents($sources['docs'] . 'license.txt'),
    'readme' => file_get_contents($sources['docs'] . 'readme.txt'),
    'changelog' => file_get_contents($sources['docs'] . 'changelog.txt'),
    'setup-options' => [
        'source' => $sources['build'] . 'setup.options.php',
    ],
    'theme-definition' => json_encode(include __DIR__ . '/data/theme.inc.php'),
]);

As the theme definition returns an array, we're simply include-ing it. I decided to encode it as JSON, but I don't think you have to do that - the package manifest is serialised so should also support arbitrary arrays.

If you were to build a package at this point, that would include the theme-definition, but it's not being used yet.

Accessing package attributes in setup.options.php

In the _build/setup.options.php file, which is used to build the interface for the setup options shown when installing a package, the package attributes are available in $options['attributes'].

For example, to retrieve the theme-definition, the code would look like this:

<?php
$def = array_key_exists('theme-definition', $options['attributes']) 
    ? json_decode($options['attributes']['theme-definition'], true)
    : [];

if (empty($def) || !is_array($def)) {
    return 'Failed to load theme definition: ' . json_encode($options, JSON_PRETTY_PRINT);
}

foreach ($def as $definition) {
    // ... render the option ...
}

Now you can build a dynamic interface based on your data definition. We return an error to the setup options panel if we can't find the attribute.

Access data in resolvers

Building the interface is step one - accessing the same information in a resolver is step two.

In resolvers, the package attributes are in $options.

<?php

$def = array_key_exists('theme-definition', $options) 
    ? json_decode($options['theme-definition'], true) 
    : [];

if (empty($def) || !is_array($def)) {
    $modx->log(modX::LOG_LEVEL_ERROR, 'Failed to load theme definition');
    return false;
}

The selected values in the setup options window are also available in $options. So if you created a setup option named "create_template", you can check that like so:

<?php

$def = array_key_exists('theme-definition', $options) 
    ? json_decode($options['theme-definition'], true) 
    : [];

if (empty($def) || !is_array($def)) {
    $modx->log(modX::LOG_LEVEL_ERROR, 'Failed to load theme definition');
    return false;
}

if (array_key_exists('create_template', $options) && $options['create_template']) {
    foreach ($def['templates'] as $template) {
        // ... create the template or something ...
    }
}

Especially for use cases like themes, or where you have some dynamic data you want to manually create/update in a resolver instead of as a vehicle, this can be a useful technique to have under your belt.

Patreon is a community membership service that lets you pledge monthly donations, at a price you set yourself, to creators.

Back in December 2017 I first created a Patreon account to support Vasily "bezumkin" Naumkin with his work on the MODX core. Shortly after that I added pledges to the creators of PHPunit, FlySystem, and Wait But Why (which has nothing to do with programming, but is just one of my favourite blogs on the internet). At a later point in 2018 I also pledged to Homebrew.

They're all small amounts, more a token of my support and gratitude than anything else. The largest part was for Vasily ($25, until July), and the others were $3-5 each for a total of $38.

There are no tangible benefits, other than making sure that the software doesn't just go away by rewarding the creator. I routinely spend more money on stupid things I don't really need (like a huge foam enter button), while what these creators share with the world is much more valuable, so that's a really good deal.

They are also recurring, automatically charged monthly, which starts to add up over time.

In 2018 my personal Patreon donations totalled at $348. Still not enough to pay anyone's bills or full time employment, but probably more than I would've donated to these projects if they only accepted one-off donations, and the numbers do get more meaningful for the creator when you consider the effect of more people contributing. Previously these creators would fund their work through client work, sacrifice free time after a day job, rely on one-time donations, or juggle things in another way, while with Patreon they're offered a predictable (extra) income directly attributable to the work they share.


After a chat about the work that goes into my open source projects and in particular the work that goes into the MODX core, I decided to set up my own Patreon yesterday.

I am fortunate enough that I have a business that's running well and pays the bills, but I still constantly have to prioritise my time and energy. When the to do list explodes, or energy gets low (yay, burnout), I have to choose what things to work on. The reality of running a business is that things that make money are more important than things that don't, and as a result it's usually the open source work that gets snoozed first even though I strongly believe that the work I do on the MODX project should be an important part of my day-to-day work.

So that's where Patreon comes in.

By supporting me on Patreon, you're supporting my work for the MODX core and open source extras. Every contribution shows that these hours upon hours of work are worth something to you, and that will motivate me to keep up and make more time available to keep making the CMS you use better, one PR at a time.


If you're interested in supporting others in the MODX community, check out Joshua L├╝ckers' Patreon, who is the most recent person to become a MODX integrator and has been putting in lots of hours as well.

I'm not aware of any other MODXers with a Patreon page at the moment (Vasily shut his down in July), but if you find any, or have any other Patreons you support, leave a comment below :)

For SiteDash I built a worker queue, based on MySQL, to handle processing tasks asynchronously. There's a central database and separate worker servers inside the same private network that poll for new tasks to execute. These worker servers run PHP, using xPDO 3, to perform the tasks the application server has scheduled.

One problem that would occasionally pop up is that the worker servers would lose connection with the database. The database is on a different server in the network, so that could come from rebooting the database server, a deploy or backup causing high load, network glitch, or just.. gremlins.

Obviously, the worker servers need to talk to the database to be useful, so I started looking at a way to 1) detect the connection was dropped and 2) automatically reconnect if that happens. It turns out to be fairly straightforward (once you know how!).

First, I implemented a check to see if the connection is alive. It does that by checking if a prepared statement (query) could be prepared.

<?php
while (true) {
    $q = 'query that is irrelevant here';
    $stmt = $xpdo->query($q);
    if ($stmt) {
        $stmt->execute();
    }
    else {
        reconnect();
    }
    // Execute task, if any
    sleep(1);
}

function reconnect() {
    global $xpdo;
    $xpdo->connection->pdo = null;
    return $xpdo->connect(null, array(xPDO::OPT_CONN_MUTABLE => true));
}

The workers run in an infinite loop, one loop per second, so this check happens every second. When the statement can't be prepared it's treated as a dropped connection, and we call the reconnect method to restore the connection.

The reconnect happens by unsetting the PDO instance on the xPDOConnection instance. Without that, xPDO thinks it still has a connection, and will continue to fail. Because we don't unset the xPDOConnection instance, we can just call $xpdo->connect() without providing the database connection details again.

With this check in place, the loop can still get stuck in a useless state if there's a reason it can't reconnect. That can have some unintended side effects and makes it harder to detect a problem that needs manual interference, so I also implemented another check.

Every 10 loops, another query is sent to the database with a specific expected response; a simple SELECT <string>. The idea is the same as the check above, see if the statement can't be prepared or doesn't return the expected result, and if so, do something.

Here's what that roughly looks like:

<?php
$wid = 'Worker1';
$loops = 0;
while (true) {
    $loops++;
    
    $q = 'query that is irrelevant here';
    $stmt = $xpdo->query($q);
    if ($stmt) {
        $stmt->execute();
    }
    else {
        reconnect();
    }
    // Execute task, if any
    sleep(1);
    
    // Every 10 loops, check if the connection is alive
    if (($loops % 10) === 0) {
        $alive = $xpdo->query('SELECT ' . $xpdo->quote($wid));
        if (!$alive || $wid !== $alive->fetchColumn()) {
            break;
        }
    }
}

function reconnect() {
    global $xpdo;
    $xpdo->connection->pdo = null;
    return $xpdo->connect(null, array(xPDO::OPT_CONN_MUTABLE => true));
}

In this case, we're not calling the reconnect() method. Instead, we're breaking out of the loop. This way the PHP process can end gracefully, instead of pretending to be churning along properly. When the process ends, supervisord is used to automatically restart it. When a new process is unable of connecting, the logs and monitoring get a lot louder than when a worker silently keeps running, so this system is working nicely.

Now, this obviously isn't the entire worker code for SiteDash. Over time it has grown into 300 lines (not counting the tasks themselves) of worker logging, automatic restarting when a deployment happened, analytics, ability to gracefully kill a process, and dealing with unexpected scenarios like a database connection getting dropped.

Overall this system has managed to keep the processes running quite nicely. There were some issues where certain tasks would cause a worker to get stuck, which have now been resolved, and currently the biggest limiting factor for the worker uptime is deployments. The workers need to restart after a deployment to make sure there is no old code in memory, and I have been fairly busy with adding features to SiteDash (like remote MODX upgrades last week!).

It's also been fun and interesting to try to get a good insight into these background processes and tweaking the monitoring to notify about unexpected events, without triggering too many false negatives. A challenge I'd like to work on in the future is automatically scaling the number of workers if the queue goes over a certain threshold, but for now I can manually launch a couple of extra processes quite quickly if things take too long.

Some fun numbers:

  • Overall, since launching workers as indefinitely running processes, the average worker process was alive for 12,5 hours
  • Since fixing the last known glitch where a process could get stuck executing certain tasks, on October 29th, the average worker stayed online for 2 days, 12 hours and 52 minutes.
  • The longest running workers started on November 1st and stayed up for 12 days, 5 hours and 40 minutes before being restarted due to a deployment.

I'm writing this post on the brand new MODX 2.7, released earlier this week. Among new features like a trash manager, form customisation for create and update in the same set, purging old packages and lots more, there's one more change you're likely to start experiencing soon: deprecated features.

Shortly after updating to 2.7, your MODX error log will start showing new messages saying various things are deprecated. Some examples of what you might encounter include:

  • modAction support is deprecated since version 2.3.0. Support for modAction has been replaced with routing based on a namespace and action name. Please update the extra with the namespace <namespace> to the routing based system.
  • Flat file processor support is deprecated since version 2.7.0.
  • Flat file processor support, used for action mgr/resources/getlist with path path/to/action.php, is deprecated since version 2.7.0.
  • modRestClient::__construct is deprecated since version 2.3.0. Use the modRest classes instead.
  • Old modTemplateVar getRender inputmethod

While those logs typically go into the MODX error log, it's also possible to see them when installing packages (especially the modAction and modRestClient messages).

What does it mean?

Basically, a feature is used which is on the shortlist to be removed from MODX in a future release. In many cases, that future release may be MODX 3.0, but that's not necessarily set in stone for everything.

The primary goal is to inform you, as someone responsible for a site, what might break in the future. You don't have to freak out right away, but it is useful to take a good look at your log after using your site and manager, to see what's at risk. Perhaps there are third party extras that will need to be updated, or you could also have custom systems running that need some tweaks.

It's better to find out now, so you can plan ahead, then when 3.0 comes out and your site does break.

How do I know what needs fixing?

Hopefully, the messages contain enough detail that it's clear where the problem is. That's not always the case, unfortunately, such as with the flat file processor message (which should be clearer in 2.7.1) and the rather vague "Old modTemplateVar getRender inputmethod" that ended up being a false positive in most cases.

My expectation is the you'll see the "modAction support since version 2.3.0 is deprecated" and "Flat file processor support is deprecated since version 2.7.0" messages the most.

In the modAction message, you'll see the namespace mentioned which should correspond with one of your extras or custom components.

Once 2.7.1 comes out, the "Flat file processor support" (which means a processor doesn't use the modProcessor class) should become rarer as it wont get triggered on every manager request anymore, and will also include the actual processor file that was called. That should pinpoint exactly what extra (or core feature, that's not ruled out!) is at fault and needs some work.

I'm a developer, how do I fix my extra?

Here are some resources to get started:

I'll also be working on my extras (both free and premium ones) to replace deprecated features as much as possible. A few of my extras still use modAction and modRestClient is also in use in various places. If I find the time I'll try to write more about dealing with specific deprecated features.

My log is now huge! How do I possible parse through all of this?

Now that you've asked... give SiteDash a try! Connect your site, and use its error log analyser to quickly summarise large error logs. It will group messages together for you, and try to explain what it finds.

(As of this week, it can also remotely upgrade sites for you, so if you're not on 2.7 yet, that's cool to try!)

Is it possible to disable the deprecated notices?

Yes, there's a new setting called log_deprecated that you can change. When disabled, you'll no longer get any deprecated notices in your logs. I would however recommend leaving it on at least for a little while to see what features your site uses that need to be updated. Report those logs to the developers of extras you use to make sure they're aware of what needs to be addressed. That gives both you and them the time to fix them well ahead of MODX 3.

Is the proper term "deprecation" or "depreciation"?

Deprecation, or deprecated.

If something it depreciated (with the extra i), it means the monetary value of a thing has decreased over time (like a car), but to deprecate something means to discourage its use as it will become obsolete.