You are here:Home»KB»Web Design»CMS»Joomla»Joomla Release System and Configuring an Update Server
Sunday, 10 November 2019 10:25

Joomla Release System and Configuring an Update Server

Written by

NB: This document is not finished so if you are reading this please bear with me. Thes instructions are what i am using/will use for quantumwarp.com

The Joomla Update Enviroment Explained

This might of been done in the past but I want to explain in clear terms on how the Joomla Update Enviroment works, the implementations I have found and then how I implemented my own Joomla Update Server so i can get on with what I really wanted to do in the first place which is to develop joomla extensions. There is some documentation but no real practical gold standard setup where someone says do this, use this software. Akeeba Release System is what a lot of people use but this is a difficult program to master. At the bottom of this page you will see a whole raft of resources which i used to put this article together so if there is something I have missed you might find help in one of those links and then let me know. These instructions will serve as template for setting up your own Joomla Update Server from scratch, How to implement this in your extensions and will be what I use here on quantumwarp.com and my extensions. I might also use this for non-joomla extensions if possible.

This article will cover topics such as:

  • Overview of the Joomla Update Environment
  • Configure and update server on your website for your Joomla extensions.
  • Manage free and paid for extention updates.
  • Manage non-joomla updates
  • Setup your product pages witha good adapatable SEF structure
  • Publish your extensions to the JED

The Different Areas of the Joomla Update Environment

I will now break down the update enviroment into their different constituent parts that you might need. Your installation should do all or some of these depending on your configuration. Also you might have one piece of software do all of these or a combination of softwares acting as one. A Joomla Update Server can be a combination of the following items but a lot of people just think the server just hands out the XML update streams, end users dont, they expect the file aswell

  • Joomla Update Server
    • Joomla! XML extension update streams
      • XML file that informs Joomla of updates to the software that registered this stream in Joomla.
      • You can use two options for your server type; collection or extension.
      • The collection server type allows developers to define an extension's manifest to pull updates from a collection. This type of server can be used if the developer wants to define all of their extension's updates in a single file (not recommended) or if their extension has multiple sub-extensions which are not distributed or updated at the same time (such as a package extension type). Ideal if your extension is a package and you want to allow users to update the package components individually, also allows you to have different update XML files for different versions of Joomla.
      • The extension server type allows developers to define an extension's manifest to pull updates from a single extension's manifest. All collection manifests eventually point to this XML file. All updates in this file must be defined after an <updates> tag at the beginning of the file. The below example is the update definition for the Joomla! 3.9.6 release:
      • I have also seen where XML files still have the eupdate entires for olders version of the extension being downloaded.
      • Multiple servers can be defined within the <updateservers> tag.
      • Deploying an Update Server - Joomla! Documentation - This will explain in more detail about this file
    • JED Remote XML files
      • XML file for automatically updating an extension listing in the Joomla! Extensions Directory. This file is read every 24 hours by the JED. The information is not pushed to JED.
      • Setting up update streams · akeeba/release-system Wiki · GitHub - This is closest to official doucmentation for the JED update stream and information of the other stream types.
    • Master XML Update stream - This has a single XML that has the update streams for all of your extensions in one place. Not sure when to use this. Similiar to a Collection but different useage case.
    • Download Manager - JDownloads, PhocaDownload, Straight file download (not recommended)
    • Subscription Manager - PayPlans: Will have all of the license keys and payments for subscriptions. PayPlans might also offer its own update server for commerical apps.
  • Client Joomla Installation
    • Joomla Update Mechanism - This is the part of Joomla that handles getting the updates in your Joomla installation. The code is native to the platform.
    • Joomla Extension
      • Manifest XML - This hold all of your extensions configurations options. Either named updates.xml or named after the extension rsfirewall.xml, dont use any other name.
      • Defined Update Server - This setting must be configured in your extensions manifest XML file. It is possibel to list more than one update server in your XML file.
      • License key - This is needed if you need to authenticate the download rights of the license Key (optional)

Joomla Database Tables (Client) - What do they do?

These are the tables in the clients Joomla installation, not the update server.

  • jos_updates - This is the local cache for available updates. When you check for updates this table is populated with available updates and is probably why it sometimes seems slow to load after clicking the 'Find Updates' button, conversely when you empty the update cache by clicking on 'Clear Cache' this table gets emptied. You might not see all of them listed in the Joomla admin because some seem to be language specific and most people only use one language.
  • jos_update_sites - This table hold all of the register update streams which point to Joomla Update servers. these entries appear in the Joomla Admin Extension Update page (Admin --> Extensions --> Manage --> Update)
  • jos_update_sites_extensions - This table registers multiple extensions against a single update stream which allows multiple extensiosn to share a single update stream and a single update license when needed. i.e. RSFirewall might be a combination of a main component and separate plugins. The developer might want to only update the system plugin and not the component. these updates would appear as separate available updates in the joomla admin. This is particularily useful for joomla packages, why re-install everything if you only change one item. this saves bandwidth and things going wrong.

Methods for Commercial Extension Updates

So far I have discovered there are several methods so far for handling commercial software updating. These are an overview as there are probably different ways of implementing these.

  1. Standard update stream XML - Manipulated Update Stream
    • Use the function onInstallerBeforePackageDownload() to modify the URL of the download package in the udpate stream to added in host/hash/code to authorised the download.
    • This function runs everytime the update process is run and requires a seperate installer extension but this can also be part of the main extension. (RSFirewall)
    • This method was a way of sending API keys for Joomla versions less than 3.2 where the update streams needed to end in .xml , this now no longer a problem. It is also an alternative way to send host domain and other information but you could get the host information from the request header.
  2. Standard update stream XML - Update Stream With API Key
    • The API key is added into the update stream stored into the database when the API key is added in the extension. (EasyBlog)
    • This link (URL) which can be used with Joomla!'s "Install from URL" feature to install subscription-based extensions. (if multi domain installation allowed)
  3. Standard update stream XML - No download link
    • You can implement the Update Servers in a similar way to how Paid Extensions manage it - you can implement the Update Servers to inform the users about the available updates and display a message saying that the update is available through contacting them via your website. The key point here is to inform users about the updates - the decision to provide them automatically or not is your decision. This means the user has to login in to download the extension but can still be notified. The method does not need API keys etc..

You can use a different update stream for commercial and free extensions but you can potential use these methods to allow the supply of both types using the same update stream. The difference is just in the request, the commercial ones have extra variables being sent.

RSFirewall Commercial Extension Update Environment Example

NB: `jos_` is the fictional database prefix I am using.

In this worked example I follow how RSFirewall manages its licensing information. In particular this is an example of how the update url is manipulated upon 'update this extension' submission, this will make more sense after following this through.

Installing the extension and how the update server is registered in Joomla

  • After you install RSFirewall there will be a new entry added that has been added into the Joomla  Extension Update sites. This is the extension's update stream and has been read from the file rsfirewall.xml found in the root of the zip file you just used to install RSFirewall. The installation adds an entry into jos_update_sites to create this record.
  • After installation there will also be a record in jos_update_sites_extensions where the extension is registered against the update stream. In the case of RSFirewall there is only one entry becasue the developer manages the RSFirewall updates as a single extension.
  • The code in the rsfirewall.xml in particular that was used to add this Update Server is shown below. This is the Extension's Update Stream.
    <updateservers>
    <server type="extension" priority="1" name="RSFirewall!">https://www.rsjoomla.com/updates/com_rsfirewall/Component/com_rsfirewall.xml</server>
</updateservers>

What happens when you update this extension

I have not entered a license key as this is optional and because I am exploring this 'Shared XML/Manipulate Update Stream' method. i.e. you get a different version dependent on the license key or lack off. The different outcomes are explored later.

  • Goto (Admin --> Extensions --> Manage --> Update)
  • Click 'Find Updates'. Joomla will now dowload the com_update.xml file from RSJoomla. The code looks as shown below:
    <updates>
<update>
	<name>RSFirewall! 2.12.1</name>
	<description></description>
	<element>com_rsfirewall</element>
	<type>component</type>
	<folder></folder>
	<version>2.12.1</version>
	<infourl title="RSFirewall! 2.12.1">https://www.rsjoomla.com/support/documentation/rsfirewall-user-guide/changelog.html</infourl>
	<downloads>
		<downloadurl type="full" format="">https://www.rsjoomla.com/updates/com_rsfirewall/Component/com_rsfirewall.zip</downloadurl>
	</downloads>
	<tags>
		<tag>stable</tag>
	</tags>
	<maintainer>RSJoomla!</maintainer>
	<maintainerurl>https://www.rsjoomla.com/</maintainerurl>
	<section>Updates</section>
	<targetplatform name="joomla" version=".*" />
	<client>administrator</client>
</update>
</updates>
  • Joomla will parse the com_update.xml file, extract the latest version number and then compare this to the version number of installed RSFirewall extension to see if the one offered at RSJoomla is newer. When a new version is found you will see an entry like below shown in the Joomla admin
  • Joomla now checks to see if RSFirewall has registered any extra code to be run before the update package is downloaded. RSFirewall has, is in the file plugins/installer/rsfirewall/rsfirewall.php and is as follows :
    <?php
/**
* @package RSFirewall!
* @copyright (C) 2015 www.rsjoomla.com
* @license GPL, http://www.gnu.org/copyleft/gpl.html
*/

defined('_JEXEC') or die;

class plgInstallerRsfirewall extends JPlugin
{
	public function onInstallerBeforePackageDownload(&$url, &$headers)
	{
		$uri 	= JUri::getInstance($url);
		$parts 	= explode('/', $uri->getPath());
		
		if ($uri->getHost() == 'www.rsjoomla.com' && in_array('com_rsfirewall', $parts)) {
			if (!file_exists(JPATH_ADMINISTRATOR.'/components/com_rsfirewall/helpers/config.php')) {
				return;
			}
			
			if (!file_exists(JPATH_ADMINISTRATOR.'/components/com_rsfirewall/helpers/version.php')) {
				return;
			}
			
			// Load our config
			require_once JPATH_ADMINISTRATOR.'/components/com_rsfirewall/helpers/config.php';
			
			// Load our version
			require_once JPATH_ADMINISTRATOR.'/components/com_rsfirewall/helpers/version.php';
			
			// Load language
			JFactory::getLanguage()->load('plg_installer_rsfirewall');
			
			// Get the version
			$version = new RSFirewallVersion;
			
			// Get the update code
			$code = RSFirewallConfig::getInstance()->get('code');
			
			// No code added
			if (!strlen($code)) {
				JFactory::getApplication()->enqueueMessage(JText::_('PLG_INSTALLER_RSFIREWALL_MISSING_UPDATE_CODE'), 'warning');
				return;
			}
			
			// Code length is incorrect
			if (strlen($code) != 20) {
				JFactory::getApplication()->enqueueMessage(JText::_('PLG_INSTALLER_RSFIREWALL_INCORRECT_CODE'), 'warning');
				return;
			}
			
			// Compute the update hash			
			$uri->setVar('hash', md5($code.$version->key));
			$uri->setVar('domain', JUri::getInstance()->getHost());
			$uri->setVar('code', $code);
			$url = $uri->toString();
		}
	}
}
  • The class name plgInstallerRsFirewall indicates this is registered with the Joomla installer process and the extends JPlugin just means it can use Joomla functions related to plugins.
  • Once you understand what the function onInstallerBeforePackageDownload() does and why, it is quite useful. If you read the class name you can clearly see that this function (if exists for the respective extension) is run by Joomla before the download package is requested but after the update stream has been read out of the database. Remember the download package is the zip file defined in com_rsfirewall.xml, in this case:
    https://www.rsjoomla.com/updates/com_rsfirewall/Component/com_rsfirewall.zip
  • Reading the code of onInstallerBeforePackageDownload() I figured out that this function:
    • Is passed the download package URL which has been extracted from the update stream com_rsfirewall.xml earlier in the update process.
    • Loads various libraries and things from Joomla and RSFirewall (not going into that here)
    • It then checks if the license key is set. If it is not set then the code returns a warning saying no key set.
    • If there is a key, this code then checks to make sure it is the right length, if not returns an error message.
    • If the two tests above pass then the code builds some variables hash/domain/code and adds these to the download package URL as GET variables
    • This modfied URL is passed back
  • Joomla now grabs the download packge using the modified (or not) URL as if it was the original URL. The modified URL will look as follows: 
    "https://www.rsjoomla.com/updates/com_rsfirewall/Component/com_rsfirewall.zip?hash=f0588314d3680f6c15c71c7FF394129d&domain=localhost&code=xxxxxxxxxxxxxxxxxxxx"

    hash = a MD5 hash of the RSFirewall Version (local) and the license key
    domain = domain that RSFirewall is installed on
    code = license key

Notes on this process

  • Using onInstallerBeforePackageDownload() allows invisible manipulation of the update stream which facilitates:
    • Being able to check for updates without a license key. Clients can see how out of data their version is even if they dont have a valid key.
    • Being able to validate a subscription when a valid license key is installed and download a package if allowed.
    • This code can allow the sending of host and version information. This can be useful for statisictics for the vendor.
    • A single remote update stream (com_rsfirewall.xml) for both free and commercial software. It is only when you access the download URL specifed in the XML that the hash/domain/code supplied come into play.
    • This code might grab all download package and run the test on very URL to see if it is pointing at www.rsjoomla.com and has com_rsfirewall in the url.

Various Error Messages

The remote server responds with various error messages in text. I do not know if this is where Joomla gets the reponse messages, styles them as errors, and them displays them at the top of the updates page, it probably is, or if it is specific to RSFirewall becasue it has installed some extra code.

  • If you have the entered the wrong you get a failed message at the top of the update page 
    Failed to download package. Download it and install manually from https://www.rsjoomla.com/updates/com_rsfirewall/Component/com_rsfirewall.zip.
    The text response from the RSJoomla Update Server is "The update code was not found in our database."

  • if you click try and download directly from the download package URL. basic HTML (no body etc..)
    This extension cannot be downloaded directly; this is a commercial product and the only way to download it is either through your RSJoomla! account or automatically by Joomla! updates once you supply your license code in the extension's configuration.<br /> More info <a href='https://www.rsjoomla.com/support/documentation/general-faq/where-do-i-find-my-license-code-.html'>here</a>.

I found this code in action from RSFirewall by:

  • search for the error message that came up after I replace the update code with a dodgy one
    • Your update code appears to be incorrect. Please make sure you've copied it correctly.
  • which returned 
    PLG_INSTALLER_RSFIREWALL_INCORRECT_CODE="Your update code appears to be incorrect. Please make sure you've copied it correctly. <strong><a href="/_QQ_"https://www.rsjoomla.com/support/documentation/general-faq/where-do-i-find-my-license-code-.html"_QQ_" target="_QQ_"_blank"_QQ_">More information</a></strong>"
  • I then searched for 
    PLG_INSTALLER_RSFIREWALL_INCORRECT_CODE
  • which returned 
    D:\websites\htdocs\quantumwarp.com\plugins\installer\rsfirewall\rsfirewall.php
  • The function responsible is onInstallerBeforePackageDownload() is within plugins/installer/rsfirewall/rsfirewall.php, Not all extensions use this code and seems to of been added by a seperate exentions which is part of RSFirewall. If you want more information have a look in the RSFirewall Zip package.

 

 

Resources

These are articles that I have found that all relate to the Joomla Release System and will help me undertand how to implement and use it.

Official Documentation (JRS)

Other Documentation

Implementing Code Examples

  • Updating the update stream in the Database (Client)
    • How to add Joomla Updater to commercial component - So the purpose of this article is to explain to commercial extension developers how they can leverage the Joomla Updater in their components, but still limit updates only to customers who have a current subscription.
    • Joomla Update System requirement (From Scratch) | Smarty Blog - This article is written for those developers who face issues with update server requirements. Wont go deep into theory part but here is what I did to implement those features in my Free and paid extensions. i will cover some details that you wont find there.
  • Alter the update stream on the Update Server (Vendor)
    • Joomla Update System implementation for paid extensions - Digigreg -Implementing this feature for free extensions is very easy, instead Joomla Update System for paid extensions is a little bit complicated. As the official Joomla! site has not published a standard way to achieve this result, all developers use own method. This uses VirtueMart.
  • Add addtional data to #__update_sites_extensions (Client?/Vendor?)
  • Login with a Hash / API Key enabled downloads
    • How to Use a Hash in the URL to Authenticate Logins to a Joomla Website | itoctopus - This is a simple example to get me started
    • I might not need to login but rather construct database lookups to manually see if the user can download an update rather than relying on loging in and usr the native Joomla authorisation.
    • Suitable  code might be in Akeeba Subscriptions.
    • Create or Update a component to convert HASHes to login details. Does Joomla store all of a user's groups in one table?
    • I could hash the username with a secret key (not Joomla's becasue you could use the username to get the secret key) to make a hash and then this could be decodes to the user name and then could check what grousp the user belongs to, checks the clients domain, the required software and then provide the download if allowed.
    • Maybe a separate table with random keys against users
    • Make a separate helper plugin that works with Akeeba Release System
    • ARS does have a system similiar tot his but it is not verified agaist usergroups
    • ARS Download ID system warrants a look. It grabs the variables from the URL.
      • The answer to that question is the Download ID. The Download ID is an MD5 hash derived from a user's numeric ID,
        username, encrypted password and email address which uniquely identified the user to ARS. Appending the download
        URL with the dlid=download_id query string parameter will allow ARS to authenticate the user and, if he has
        access to the download, proceed with the download.

Release Systems

These are extensions that handle the XML and the actual download of the software. These systems dont have inbuilt subscription systems and the functionality varies.

Update Server (XML Only)

Create your XML files using a component rather than doing them manually.

Package Checking

This is related, you need to make sure your plugins are compliant with Joomla's coding practice including using 'Tabs' which I hate.

Update Notifiers

Again these are along the same line as this article, so where better to put them to complete the circle. These Joomla extensions will notfy you of updates to your Joomla installation and it's extensions with various options. The inbuilt Joomla update notification system plugin runs every couple of hours and is the first thing I turn off. When developing multiple websites you dont want mutiple emails from multiple websites weveryday telling you the same thing. The Joomla notification plugin only notfies you of a Joomla platform update and nto the extensions.

 

Current Work

List the basic mechanisim in my words

To Do list

  • update server (XML) for free ones: do i need to upgrade to modern standards? which component is good to go?, do i do this for free and paid?
  • subscription for paid extensions (and how this relates to nonppaid xml files)
  • product page: old versions displayed in a module, tabs, similiar to the joomla JED, should i use custom fields?
  • internal message system ie for kunena, can I refator the one already in joomla? is this message system present in joomla 4?
  • extension checker - needs updating
  • Should i use Akeeba Release system?
  • Should i use a combination of a Joomla article, JDownloads and a component to build the XML file
  • Using a component allows me to track the usage of my software.
  • Joomla uses Github RAW to serve its XML files
  • I could create the XML files manually
  • What SEF structure would i give to these different softawres, I have to consider I have joomla and non-joomla software that will use this system

My Update System

  • Potential names:  QWUpdateServer or QWUpdate  (but consider QWUpdateChecker) = camelcase the name and add QW at the beginning, no more than 2 words

com_updateserver Notes

  • update server has been archived.
  • outputted xml is all on one line
  • this only seem to create single XML file (i.e. master XML)
  • cannot use for non-joomla currently
  • not all fields are present
  • some fields are context sensitive (not a bad thing for joomla extensions, good for consistency)
  • an option to upload the file is there to be downloaded from the actually software
  • no option to manually put in your own url for the file

Update Server Options

  • Manually create XML files
    • Pro
      • Easy to do
      • Can handle Joomla and Non-Joomla easily
      • Repo Structure is easy to do
      • Free
    • Con
      • Cannot directly control download rates (could use JDownloads)
      • Cannot directly  log downloads (could use JDownloads)
      • Cannot apply subscriptions
  • Akeeba Release System
    • Pro
      • The component is already made
      • Free
      • Logs downloads
      • Basic dowload control
    • Con
      • Very difficult to use
      • Only made for Akeeba and not the end user.
      • Subscriptions are only available through Akeeba subscriptions
  • Update Server (by Igor Berdichevskiy)
    • Pro
      • Free
      • Easy to use
      • can control the repo layout with menus
    • Con
      • basic functionality
      • no longer updated
  • Make my own
    • Pro
      • Can make it handle only XML
      • I can add joomla usergroups
      • I control the software
      • Free
      • Can integrate it with JDownloads or other extension.
      • Can integrate with PayPlans.
    • Con
      • Will take time to make

Adding Subscription Integration

These are my basic notes I will use for reference:

Developing my own user authentication using a License Key in the update stream (added dynamically by onInstallerBeforePackageDownload() )

  • Use a hash to verify that the user exists and then load that user's object
    • The update stream request will have the key in it
      1. The key could be a secure hash of the username and 2 secret keys to prevent dercyption hash = md5($secretkey1.md5($secretkey2.md5($username)). This will allow simple verification.
      2. You can also have a table for my 'Update Component' that stores authorised domain information which will be check agast the request header.
    • The user has now been identified from the key
    • Without validation load the User Object
    • use the consequential 'Authenticated' user's usergroup rights to dowload assets, if permissions allow, download asset.
    • I am not sure how this will acutally play with jDownloads and ARS. An update stream error response is only text. Will ARS return this. JDownlaods and the relevant ARS Release must have the same usergroup configured for this to work otherwise the response might be from JDownloads mightbreak things?
  • Use a hash to verify that the user exists and then check the database for permissions
    • The update stream request will have the key in it
      1. The key could be a secure hash of the username and 2 secret keys to prevent dercyption hash = md5($secretkey1.md5($secretkey2.md5($username)). This will allow simple verification.
      2. You can also have a table for my 'Update Component' that stores authorised domain information which will be check agast the request header.
    • The user has now been identified from the key. The user is not be logged in.
    • get his usergroups he belongs to by loading them from the database directly.
    • validate he can download the requested asset by checking to see if he is a member of the correct group and response accordingly.

NB:

  • A license key generate by payplans would be better which would link to a particluar software/license from which this can then be checked either at payplans or by the groups the user belongs to
  • My component would have these 2 primary functions
    • A function to run through all users in a subscription group to make sure they have a unique ID. This can be used for automation.
    • A function to add an indiviual user with a unique ID. This will be used when a user is manually added.

Akeeba Modification Notes

  • Overrride a stream for QWcrm
  • Find out where the code is blocking me using XML for a Non-Joomla platform (add a bypass for the QWcrm) but make sure the JED and other streama are not affected.
  • I could modify the INI view to present an XML stream
  • Can I use the Download ID system for license keys?
  • Add another stream that is designed for QWcrm.

Building your own XML

  • https://docs.joomla.org/Deploying_an_Update_Server - The full spec for the update.xml is here and the documentation is easy to follow.
  • libraries/src/Updater/Update.php - The location for the Update class responsible for parsing the update.xml in Joomla 4.
  • libraries\src\Changelog\Changelog.php - The location for the Changelog class responsible for parsingthe changelog.xml in Joomla 4.
  • Different <targetplatform> version masks
    • version="3.[123456789]|10"
Read 1047 times Last modified on Thursday, 28 November 2019 15:50
Published in Joomla
back to top