0
7-easy-steps-to-satis-typo3-private-packages-for-composer

When we launched our T3Terminal.com venture, One of the first super demands from our customers was “Support Composer” for premium TYPO3 products too.

Because as the TYPO3 People just love the Composer, the community also has a dedicated composer-server https://composer.typo3.org/

And, you know, we always learn lots from customer’s demand and feedback.

T3Terminal’s free TYPO3 products already have composer support with most popular Packagist.org But for a premium package, it was still manual installation.

Team T3Terminal took the challenge and was super excited to make TYPO3 Satis Private packages for Composer.

One more thing, manier times you would have noticed that packagist.org is very-slow when you install and update your TYPO3 packages through composer. We recommend setting up your private Composer-server to have productive work.

We had a growing problem: every time we were pushing something on GIT, the CI would take more then five minutes to deploy a new version.

git-command-composer

Before you begin, If you don’t know what a composer is? Then Google “PHP Composer” ;) and you will love to explore it. Because that’s how modern PHP developers work nowadays!

What’s the problem?

Two PHP developers Nils and Jordi have developed this cool PHP application-level package manager solution to change the way - how developers install PHP packages eg., TYPO3 Extensions - Call Composer.

Packagist.org is the main repository to connect your TYPO3 packages.

But here is the real problem, Packagist has only featured to publish your OpenSource public TYPO3 packages. It contains thousands of PHP packages.

What about your TYPO3 private packages? You may have private packages stored at your private Git repositories like Github, Gitlab, Bitbucket etc.

How can you provide your private TYPO3 packages and packages your customer or teams? That’s a reason, you are still here to know how? And of course, If you are facing performance issues with packagist.org ;)

What are the easy-ways to setup your TYPO3 Private Packages?

Yet, there are only two ways to install and configure your TYPO3 Private Packages.

packagist-composer-repository

Option 1: SaaS Solution by Packagist.com

This option is awesome, If you have a good budget for your private projects :) It’s been developed and maintained by the same team of Packagist.org, We have tried the trial version and looks promising with great features like;

  1. Easily integrate with your favourite Git tools like Github, Gitlab and Bitbucket
  2. Very user-friendly user-interface to host your private repositories
  3. Much faster and reliable for deployment
  4. Access and restriction options

After all, It’s made by the creators of the Composer!

Option 2: Self-Hosted Solution with Satis

Unfortunately, there is only one good option available to setup your TYPO3 Satis. That's the goal of this blog to let you know, how to setup Satis.

composer-t3terminal

What is Satis?

composer-satis-image

Well, Satis is an OpenSource project for static composer repository generator. Here is the source https://github.com/composer/satis also it’s been mentioned at the official guide at Getcomposer.org

Let’s proceed with step-by-step guide to everything like setup TYPO3, create webhooks, security & authentication etc., as follows.

Just to give you an impression, Once you install and configure whole Satis setup, then you will have the final setup as follows.

composer-folder-structure

Architecture of Packagist

architecture-of-packagist

Architecture of TYPO3 Private Composer Server

architecture-of-typo3-private-composer-server
  • Step 1. Installation of Satis

Pre-requisites

  1. We suggest creating one subdomain like https://composer.yourdomain.com/
  2. Composer >= 1.9
  3. PHP >= 7.2

Install Satis

Go to your favourite command line and enter the following command

composer create-project composer/satis --stability=dev --keep-vcs
your-favourite-command-line-composer

That’s it, “satis” folder has been initiated which contains all dependency modules like Symfony, composer etc., at your root folder.

  • Step 2. Configure satis.json

At your root folder, Create satis.json and let’s configure your private repositories as follows.

{
            "name": "Repository Name",
            "homepage": "https://composer.yourdomain.com",
            "archive": {
                "directory": "dist",
                "absolute-directory" : "/path/to/save/packages/",
                "format": "zip",
                "skip-dev": true
            },
            "require-all": true,
            "require-dependencies": true,
            "require-dev-dependencies": false,
            "output-html": true,
            "repositories": [
                {
                    "name": "vendor/package1", "type": "vcs", "url": "https://gitlab.com/vendor/package1.git" 
                    },
                {
                    "name": "vendor/package1", "type": "vcs", "url": "https://github.com/vendor/package2.git" 
                    },
                { 
                    "name": "georgringer/news", "type": "vcs", "url": "https://github.com/georgringer/news"
                }
            ],
            "require": {
                "vendor/package1": "*",
                "vendor/package2": "*"
            }
        }

Most of the setup is self-explained, although Let me try to write quick important configuration options of satis.json

  1. absolute-directory: Set the current root folder to have a “dist” folder within your domain.
  2. require-dependencies: Satis will automatically download dependent other repositories.
  3. output-html: Keep set true, it will generate nice formatted index.html
  4. repositories: Configure your all private as well as public repositories using Git.
  5. require: From here, Satis will generate packages.
  • Step 3. Generate Satis For Composer Packages

Now you are all set to build satis packages at your composer-server. Run the following command.

Syntax: php bin/satis build <configuration file> <build-dir>
Example: php satis/bin/satis build satis.json ./

Explain the command:

  1. php: Your php bin path (generally php globally works)
  2. satis bin: As we have satis folder, so define satis/bin/satis
  3. build: is a command to generate packages
  4. satis.json: As we have configured satis.json at the root
  5. packages path: The path to generate package listing (with index.html)

If you have configured your private repositories of Git tools, It should ask you for authentication as username and password.

It’s just first-time you need to add user/pass, it will save credentials at composer’s auth.json for future use.

auth-json-future-use

Limitation of 60 call/hr

Oh!! Are you facing limit error calls from your Git access like?

limitation-of-60-call-hr

No worries, Just jump to the URL mentioned in the description and generate Token from your Git tool to remove the 60 calls/hr limitation.

Generate of Github/Gitlab token

generate-of-github-and-gitlab-token

Once successfully run the build command, It will generate packages.json + dist folder with all the packages are downloaded in .zip files. Take a look at the structure of the “dist” folder.

Also, Go to your https://composer.yourdomain.com/index.html You should be able to have a nice interface with all your generated packages information.

private-composer-repository
  • Step 4. Test-drive with Your TYPO3 Private Packages with Composer

Now, It’s time to have a test-drive at your TYPO3 composer-based project.

1st Configure your composer.json

"repositories": [
            {
                "type": "composer",
                "url": "https://composer.yourdomain.com/"
            },
            {
                "packagist.org": false
            },
            {
                "composer.typo3.org": false
            }
      ],

2nd Run composer install command

composer require vendor/package-name ^1

Hurray!

Your satis packages should download from the latest extension from your Composer server.

If you are facing any issues, then check out more details by passing -vvv to your command.

  • Step 5. Webhook between your TYPO3 Private Repositories & Satis

Of course, Story is not over yet, You will need to work on a couple of more tasks ;)

Now you should make a bridge to keep connected your TYPO3 private packages between Git & Satis. Whenever your team releases something at Git tool, then it should be automatically grabbed by your Satis composer server.

That’s why You will need to prepare the so-called “webhook”. For that, let’s create one PHP-script.

webhook.php

<?php
            // Set Composer home (also if not work then set to .htaccess)
            putenv("COMPOSER_HOME=/your/home/.composer");

            // Initiate Symfony
            require_once __DIR__.'/satis/vendor/autoload.php';

            use Symfony\Component\Process\Process;

            // Basic-Configuration
            $config = array(
                'bin' => '/your/path/satis/bin/satis',
                'json' => '/your/path/satis.json',
                'webroot' => '/your/path/',
                'user' => null,
                'secretykey' => 'set-your-secret-key',
            );

            // Validation Testing
            if (!file_exists($config['bin'])) {
                die('The Satis bin could not be found.');
            }

            if (!file_exists($config['json'])) {
                die('The satis.json file could not be found.');
            }

            if (!file_exists($config['webroot'])) {
                die('The webroot directory could not be found.');
            }

            // Switch to user
            if (null !== $config['user']) {
                $command = sprintf('sudo -u %s -i', $config['user']);
                exec($command);
            }

            // Option 1: For direct update from Browser can use the package get parameter to pass the repo url to update. Eg., https://composer.yourdomain.com/?package=giturl
                if(!empty($_GET['package'])){
                $command = sprintf('%s build %s %s %s', $config['bin'], '--repository-url '.htmlentities($_GET['package']), $config['json'], $config['webroot']);
            }

            // Option 2: Gitlab Webhook
            else if(!empty($_SERVER["HTTP_X_GITLAB_TOKEN"])){

                // Validate Secret Tocken
                if ($_SERVER["HTTP_X_GITLAB_TOKEN"] == $config['secretykey']) {
                        $input = file_get_contents("php://input");
                        $json = json_decode($input);
                        if (!is_object($json)) {
                        die('No valid JSON');
                    }
                    if ($json->object_kind == "push" || $json->object_kind == "tag_push") {
                        $repoUrl = $json->repository->git_http_url;
                        $command = sprintf('%s build %s %s %s', $config['bin'], '--repository-url '.$repoUrl, $config['json'], $config['webroot']);
                    }
                    else {
                        die("Invalid Push Event from Gitlab!");
                    }
                }
                else {
                    die("Invalid Gitlab Secret Key!");
                }
            }

            // Option 2: Github Webhook
            else if(!empty($_SERVER["HTTP_X_HUB_SIGNATURE"])){

                // Validate Secret Tocken
                $signature = $_SERVER['HTTP_X_HUB_SIGNATURE'];

                if ($signature) {
                    $hash = "sha1=".hash_hmac('sha1', file_get_contents("php://input"), $config['secretykey']);
                    if (strcmp($signature, $hash) == 0) {
                        $payload = json_decode($_POST['payload']);
                        if (!is_object($payload)) {
                            die('No valid JSON');
                        }
                        $repoUrl = htmlentities($payload->repository->clone_url);
                        $command = sprintf('%s build %s %s %s', $config['bin'], '--repository-url '.$repoUrl, $config['json'], $config['webroot']);
                    }
                    else {
                        die("Invalid Github Secret Key!");
                    }
                }
                else {
                    die("Invalid Github Secret Key!");
                }
            }

            // Prepare Default Shell-Command eg., For daily-cron
            else {
                $command = sprintf('%s build %s %s', $config['bin'], $config['json'], $config['webroot']);
            }

            // Final command to execute
            if (!empty($command)) {
                $command = "/usr/bin/php7.2 ".$command;

                // Execute final command to build Composer-server
                $exitCode = exec($command);

                // Error-handeling
                $process = new Process($command);
                $exitCode = $process->run(function ($type, $buffer) {
                    if ('err' === $type) {
                        echo '<br><strong>Error:</strong> '.$buffer;
                        error_log($buffer);
                    } else {
                        echo '.';
                    }
                });

                $returnMsg = '<br><br>Satis mirror : '.($exitCode === 0 ? 'Successful rebuild '.(!empty($_GET['package']) ? ' of package '.htmlentities($_GET['package']).'.' : ' of entire index!') : ' An error occured! : '.print_r($exitCode, true));
            }
            else {
                $returnMsg = "<br>No command found!";
            }

            echo "<br>" . $returnMsg . "\n";
        ?>

Uuf! Bit long-code, but very self-explained & no worries, it’s been well-tested ;)

Now, Let’s set webhook.php to your favourite Git tools. So, whenever you release push or release tag from your Git then it will inform your Satis Composer-server to generate new TYPO3 packages at your private server.

How to use Webhook to generate TYPO3 Private Packages?

  • Option 1: Update packages through Web-URL

You can run webhook.php at your web-browser eg.,

https://composer.yourdomain.com/webhook.php?package=https://github.com/vendor/package.git

Webhook has been designed in a way to only get a particular repository to your Satis composer server.

  • Option 2: Integrate Webhook to your Git tool

You can easily configure this webhook at your favourite Git tool like Github, Gitlab, Bitbucket etc., Here are a few examples.

Webook at Github

  • Step 1. Go to your Repositories > Settings > Webhooks
  • Step 2. Click on “Add webhook”
add-webhook
  • Step 3. Add your webhook.php URL, Secret key etc., And save form.
add-your-workhook-php-url-secret-key-etc

Webhook at Gitlab

  • Step 1. Go to your Gitlab Repositories > Settings > Integration
  • Step 2. Set your webhook.php URL, Secret key and submit webhook.
webhook-at-gitlab
  • Step 3. Edit your webhook, You can test-drive by Submitting “Test”, and able to checkout request and response.
  • Step 6. Setup Cron to Crawl Automatic TYPO3 Private Packages

We suggest setting CRON to regularly crawl your TYPO3 private packages from each of your each Git repositories.

Let’s see an example,

Daily 12 o’clock run the cron

* * * * * usr/bin/php7 /var/www/your/path/satis/bin build /var/www/your/path/satis.json /var/www/your/path/

  • Step 7. Secure Your TYPO3 Private Packages at Composer-Server

I know, your first question may be how to secure your packages at Composer server?

Fortunately, there are many ways to protect and give access/authentication to your customer or team as follows.

Tips

Make sure to setup robots.txt to disable SEO crawling.

Here are lists of security options available as follows.

  • Option 1. SSH (requires the SSH2 PECL extension)

{
            "repositories": [{
                "type": "composer",
                "url": "ssh2.sftp://yourdomain.com",
                "options": {
                    "ssh2": {
                    "username": "composer",
                    "pubkey_file": "/home/composer/.ssh/id_rsa.pub",
                    "privkey_file": "/home/composer/.ssh/id_rsa"
                    }
                }
            }]
      }
  • Option 2. SSL/TLS (HTTPS) using a client certificate

{
        "repositories": [{
            "type": "composer",
            "url": "https://yourdomain.com",
                "options": {
                "ssl": {
                "local_cert": "/home/composer/.ssl/composer.pem"
                }
            }
        }]
      }
  • Option 3. Custom HTTP Header field for token authentication

{
        "repositories": [{
            "type": "composer",
            "url": "https://yourdomain.com",
            "options": {
                "http": {
                    "header": [
                    "API-TOKEN: YOUR-API-TOKEN"
                    ]
                }
            }
        }]
      }

Now imagine, you have a customer who buys your great TYPO3 package from you. He wants to install a composer-based TYPO3 extension. As composer supports, http-based authentication,, so practically that’s suite the solution, Here is the step-by-step guide to setup security and password-protected TYPO3 private packages for a composer.

  • Step 1. Setup .htaccess

# Let’s allow webhook.php to access for your Git-tools
      <FilesMatch "webhook.php">
            SetEnvIf Request_URI "webhook.php$" allow
            Order allow,deny
            Allow from env=allow
            Satisfy any
      </FilesMatch>

      # Let’s set authentication      
      AuthUserFile /path/to/.htpasswd
      AuthType Basic
      AuthName "My Composer Restriction"
      Require valid-user
  • Step 2.Create .htpasswd

Add your username and password, You can use cool-tools like

https://www.web2generators.com/apache-tools/htpasswd-generator

That’s it, Now let’s try to access your server at your TYPO3 composer-based installation.

Test-drive Authentication at Your TYPO3 Composer

  • Step 1. Configure your composer.json

Make sure to configure composer.json with your composer-server instead of packagist.org, Checkout above test-drive topic.

  • Step 2. Ask for Authentication

So, a composer will look at your server to download the latest TYPO3 private packages, and it should be asking for Authentication.

ask-for-authentication

Individual Package’s Authentication Access

Hmm, so you are looking for authentication to each TYPO3 package separately, no worries, You can easily configure endless possibilities with your .Htaccess and Server too.

.htaccess

 # Let’s allow webhook.php to access for your Git-tools
      <FilesMatch "webhook.php">
          SetEnvIf Request_URI "webhook.php$" allow
          Order allow,deny
          Allow from env=allow
          Satisfy any
      <FilesMatch>


      # Configure your package1
      <FilesMatch "^(your-package1-prefix.*)\.(zip)$">
          AuthName "Access of My Composer"
          AuthUserFile /access/.htpasswd_your_package1
          AuthType basic
          Require valid-user
      <FilesMatch>

      # Configure your package1
       <FilesMatch "^(your-package2-prefix.*)\.(zip)$">
            AuthName "Access of composer.T3Terminal.com"
            AuthUserFile /access/.htpasswd_your_package2
            AuthType basic
            Require valid-user
       <FilesMatch>

Notes

To get a hint to what exactly to set in FileMatch, Just go to your /dist/vendor/package to the end zip file. You can set a prefix match from that zip file.

Helpful links

  1. https://getcomposer.org/doc/articles/handling-private-packages-with-satis.md
  2. https://github.com/composer/satis
  3. http://packagist.org/
  4. https://packagist.com/
  5. https://composer.typo3.org/

Wrap-up!

  1. For your composer, If you want Private packages access or better performance compared to packagist.org, Then just go with great Satis solution.
  2. You can realize now, To create your TYPO3 Private Packages is quite easy, Just try all above Step 1 to 7 to Install and configure Satis composer-server.
  3. All steps are well-tested, so we are sure you will not get any big trouble.

We hope you like this easy guideline to setup Satis, Do you know any other ways to create TYPO3 Private Packages for Composer. Feel free to write at the comment box, in-case if you have any troubles or issues.

Please share this article in your TYPO3 networks.

Have a Happy time with a great Composer :)

Post comments

Leave A Reply

Your email address will not be published.