While conducting an assessment for a client earlier this year we encountered the Plixer Scrutinizer application in use on the internal network. Having never seen this particular application before, a quick search provided the following description:

Plixer Scrutinizer is a network monitoring and analysis appliance that collects, interprets, and contextualizes data from every digital exchange and transaction to deliver insightful network intelligence and security reports.

The product documentation also provided deployment guides for multiple virtual machine platforms, including KVM with a link to download an image (https://docs.plixer.com/projects/plixer-scrutinizer-docs/en/latest/deployment_guides/deploy_virtual/virtual_kvm.html).

Extracting the file system from the KVM QCOW disk can be done a few ways. I chose to utilize the nbd module from qemu-utils, the generic process for doing this is as follows:

# apt-get install qemu-utils
# modprobe nbd max_part=16
# qemu-nbd -c /dev/nbd0 /path/to/image.qcow2

With the new device setup, the partition table can be dumped to identify the disk layout:

# fdisk -l /dev/nbd0
Disk /dev/nbd0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000a89ae

Device      Boot   Start       End   Sectors Size Id Type
/dev/nbd0p1 *       2048   2099199   2097152   1G 83 Linux
/dev/nbd0p2      2099200 209715199 207616000  99G 8e Linux LVM

The disk image contains two partitions, the first is for system boot and contains the bootloader, kernel, initial file system, while the second contains the system's root file system. The second partition type is Linux LVM, meaning it cannot be mounted directly and requires LVM utilities to access. The first step is to activate the LVM target using the pvscan command:

# pvscan --cache /dev/nbd0p2
  pvscan[1340564] PV /dev/nbd0p2 online.

With the LVM partition activated, the physical volumes can be listed using pvdisplay:

# pvdisplay /dev/nbd0p2
  --- Physical volume ---
  PV Name               /dev/nbd0p2
  VG Name               vg_scrut
  PV Size               <99.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              25343
  Free PE               0
  Allocated PE          25343
  PV UUID               qgr177-hDNb-efLX-Y8AB-lPuE-jUvU-ejn2t0

The output shows that the Volume Group (VG) is vg_scrut, vgdisplay can then be used to list the volumes within the VG:

# lvdisplay /dev/vg_scrut
  --- Logical volume ---
  LV Path                /dev/vg_scrut/lv_swap
  LV Name                lv_swap
  VG Name                vg_scrut
  LV UUID                glfyh1-2iiy-K2Ki-h6ii-exyR-Lqda-0qETJy
  LV Write Access        read/write
  LV Creation host, time localhost, 2022-03-16 17:53:56 +0000
  LV Status              available
  # open                 0
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/vg_scrut/lv_root
  LV Name                lv_root
  VG Name                vg_scrut
  LV UUID                uatqDs-i3wS-yHVw-4qe1-hLuD-vfwR-nIBkMe
  LV Write Access        read/write
  LV Creation host, time localhost, 2022-03-16 17:53:56 +0000
  LV Status              available
  # open                 0
  LV Size                20.00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/vg_scrut/lv_db
  LV Name                lv_db
  VG Name                vg_scrut
  LV UUID                ArDzWb-ncPf-1mgJ-TD1u-2Dg1-NKEh-zI42kS
  LV Write Access        read/write
  LV Creation host, time localhost, 2022-03-16 17:53:57 +0000
  LV Status              available
  # open                 0
  LV Size                <75.00 GiB
  Current LE             19199
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

In this case we are looking for the root file system which is contained within lv_root. This partition can be mounted directly using the LV Path value:

# mount  /dev/vg_scrut/lv_root tmp
# ll tmp
total 88
dr-xr-xr-x. 19 root  root   4096 Apr 21  2022 ./
drwxrwxr-x   3 chris chris  4096 Oct 19 18:18 ../
lrwxrwxrwx.  1 root  root      7 Mar 16  2022 bin -> usr/bin/
drwxr-xr-x.  2 root  root   4096 Mar 16  2022 boot/
drwxr-xr-x.  2 root  root   4096 Mar 16  2022 dev/
drwxr-xr-x. 85 root  root   4096 Apr 21  2022 etc/
drwxr-xr-x.  5 root  root   4096 Apr 21  2022 home/
lrwxrwxrwx.  1 root  root      7 Mar 16  2022 lib -> usr/lib/
lrwxrwxrwx.  1 root  root      9 Mar 16  2022 lib64 -> usr/lib64/
drwx------.  2 root  root  16384 Mar 16  2022 lost+found/
drwxr-xr-x.  2 root  root   4096 Apr 11  2018 media/
drwxr-xr-x.  2 root  root   4096 Apr 11  2018 mnt/
drwxr-xr-x.  4 root  root   4096 Apr 21  2022 opt/
drwxr-xr-x.  2 chris chris  4096 Apr 21  2022 plxr_spool/
drwxr-xr-x.  2 root  root   4096 Mar 16  2022 proc/
dr-xr-x---.  4 root  root   4096 Apr 21  2022 root/
drwxr-xr-x.  2 root  root   4096 Mar 16  2022 run/
lrwxrwxrwx.  1 root  root      8 Mar 16  2022 sbin -> usr/sbin/
drwxr-xr-x.  2 root  root   4096 Apr 11  2018 srv/
drwxr-xr-x.  2 root  root   4096 Mar 16  2022 sys/
drwxrwxrwt.  7 root  root   4096 Apr 21  2022 tmp/
drwxr-xr-x. 14 root  root   4096 Apr 21  2022 usr/
drwxr-xr-x. 20 root  root   4096 Apr 21  2022 var/

With the root file system mounted it is now possible to inspect the application content in hopes of identifying vulnerabilities that can be used on the target within the client environment. Initial inspection of the system identified that the application is utilizing Apache with FastCGI. This was identified by reviewing the configuration file /home/scrutinizer/files/conf/httpd-plixer.conf:

# This will hold all the configurations for apache that Plixer makes.
# We will no longer be editing the default httpd.conf file.
...
## FASTCGI SETUP ##
ErrorLogFormat "[%t] [%l] %F: %E: %M"
FcgidIOTimeout 600
FcgidBusyTimeout 600
FcgidMaxProcesses 100
FcgidIdleTimeout 1800
FcgidProcessLifeTime 1800
FcgidMaxRequestLen 52428800
FcgidMinProcessesPerClass 5
FcgidMaxProcessesPerClass 100
FcgidInitialEnv PGDATABASE plixer
FcgidInitialEnv PGHOST localhost
FcgidInitialEnv PGUSER plixer
FcgidInitialEnv PGSSLKEY timber_badger:/usr/share/httpd/.postgresql/postgresql.key
AddType application/x-httpd-fcgi .fcgi
...
...
Alias /fcgi "/home/plixer/scrutinizer/html/fcgi"
<Directory "/home/plixer/scrutinizer/html/fcgi">
      RewriteEngine Off
      Options +ExecCGI
      AllowOverride None
      Order allow,deny
      Allow from all
</Directory>

Within the directory specified inside the Apache configuration file, a single 12mb file was found (scrut_fcgi.fcgi). The file contents can be seen in the following excerpt:

#!/opt/perl-5.34.0/bin/perl
#line 2 "/opt/perl/bin/par.pl"
eval 'exec /usr/bin/perl  -S $0 ${1+"$@"}'
    if 0; # not running under some shell

package __par_pl;

# --- This script must not use any modules at compile time ---
# use strict;
...
...
CORE::exit($1) if ($::__ERROR =~/^_TK_EXIT_\((\d+)\)/);
die $::__ERROR if $::__ERROR;

1;

#line 1006

 __END__
PK<BINARY CONTENT>

This application is written in Perl using the Perl Archive Toolkit (PAR) (https://metacpan.org/pod/PAR) as well as the PAR Crypto filter (https://metacpan.org/pod/PAR::Filter::Crypto).

In practice, this file uses Perl to extract the zip contents attached at the bottom of the file, unpacking to a directory in /tmp/. For instance, the application is extracted to /tmp/par-726f6f74 in the following example:

$ ll /tmp/par-726f6f74/cache-0f9488d5891e440457464a09412b8fd4a393c4a3
total 24
drwxr-xr-x 3 root root 4096 Oct 27 21:03 ./
drwxr-xr-x 3 root root 4096 Oct 27 20:57 ../
-rw-r--r-- 1 root root  178 Oct 26 21:03 _CANARY_.txt
-rw-r--r-- 1 root root 3322 Oct 27 21:03 d4787e12.pl
-rw-r--r-- 1 root root  657 Oct 27 21:03 e52e8794.pl
drwxr-xr-x 4 root root 4096 Oct 27 21:03 inc/
-rw-r--r-- 1 root root    0 Oct 27 21:03 inc.lock

The actual application contents are encrypted using the use Filter::Crypto::Decrypt module:

package main;
#line 1 "script/scrut_fcgi.pl"
use Filter::Crypto::Decrypt;
460aecfc30146bb6acd3f326e386638f66ba2f653bc6b.......

The module responsible for decrypting the application ships within the archive and can be found inside the inc directory:

$ ll /tmp/par-726f6f74/cache-0f9488d5891e440457464a09412b8fd4a393c4a3/inc/lib/auto/Filter/Crypto/Decrypt/
total 28
-r-xr-xr-x 1 root root 24728 May  9 18:09 Decrypt.so

While the source of the Perl module for the Crypto filter is available, I decided to take the approach of analyzing the extracted binary statically, as we often encounter instances where we are forced to analyze binary content that applies encryption and/or obfuscation (practice makes progress).

Within the shared object the function FilterCrypto_FilterDecrypt handles decryption by passing a hardcoded key filter_crypto_pswd into PKCS5_PBKDF2_HMAC_SHA1 with a known 'random' salt value to recreate a known unique password for each call:

EVP_CIPHER_CTX_init(ctx_1);
    if ( EVP_CipherInit_ex(ctx_1, aes_256_cbc, 0LL, 0LL, 0LL, enc) )
    {
      if ( EVP_CIPHER_CTX_set_key_length(ctx_1, 32LL) )
      {
        if ( PKCS5_PBKDF2_HMAC_SHA1(&filter_crypto_pswd, 32LL, in_pass, in_salt, 2048LL, 32LL) == 1 )
        {
          out_buf = 0LL;
          if ( EVP_CipherInit_ex(ctx_1, 0LL, 0LL, hmac_key, iv, enc) )

The hardcoded key material filter_crypto_pswd is stored within the library at offset 0x3A20:

.rodata:0000000000003A20 filter_crypto_pswd db 4Bh, 44h, 0B4h, 75h, 7Eh, 0EEh, 9, 1Dh, 0E6h, 72h, 0FDh; 0
.rodata:0000000000003A20                                         ; DATA XREF: FilterCrypto_FilterDecrypt+6B2↑o
.rodata:0000000000003A2B                 db 85h, 0EAh, 73h, 0B9h, 19h, 7Fh, 0F9h, 84h, 2Ah, 9Eh; 0Bh
.rodata:0000000000003A35                 db 0B3h, 5Ch, 0BBh, 38h, 80h, 9Eh, 49h, 0E7h, 13h, 0E2h; 15h
.rodata:0000000000003A3F                 db 4Eh                  ; 1Fh
.rodata:0000000000003A40 rng_seed        dq 405FC00000000000h    ; DATA XREF: FilterCrypto_PRNGInit+A0↑r

There are a few ways to proceed to retrieve the encrypted content, the documentation page for the module explicitly calls out the shortcomings (https://metacpan.org/pod/Filter::Crypto#WARNING):

None of the above checks are infallible, however, because unless the source code decryption filter module is statically 
linked against the Perl executable then users can always replace the Perl executable being used to run the script with 
their own version, perhaps hacked in such a way as to work around the above checks, and thus with debugging/deparsing 
capabilities enabled. Such a hacked version of the Perl executable can certainly be produced since Perl is open source 
itself.

Looking at how the library works internally; the easiest solution was to hook the SSL import calls using LD_PRELOAD. The LD_PRELOAD environment variable allows users to specify additional shared libraries to be loaded before others, enabling the override of function calls in those later-loaded libraries with custom implementations provided in the LD_PRELOAD libraries. The following example code implements a simple shared object that will print the key material as it is used as well as the decrypted Perl code:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <string.h>
#include <syslog.h>
#include <stdio.h>

// gcc evphook.c -o evphook.so -fPIC -shared -ldl -lcrypto

int key_len = 0;
void printHexString(const char* str) {
    int i;
    // Iterate over each character in the string
    for (i=0; i<key_len; i++) {
        // Print the hexadecimal representation of the character
        printf("%02X ", (unsigned char)str[i]);
    }
    printf("\n");
}

//function prototype -  int EVP_CipherUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl);
int EVP_CipherUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl) {
        int (*original_target)(EVP_CIPHER_CTX *ctx, unsigned char *out,int *outl, const unsigned char *in, int inl);
        int ret;

        original_target = dlsym(RTLD_NEXT, "EVP_CipherUpdate");
        ret = original_target(ctx,out,outl,in,inl);
        printf("%s",out);
        return ret;
}

//function prototype -  int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc);
int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc) {
    int (*original_target)(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *type,ENGINE *impl, const unsigned char *key, const unsigned char *iv, int enc);
    *(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CipherInit_ex");  
    if(key != '\x00'){
            printf("### Decrypt Init:\n#### Key: ");
            printHexString(key);
            printf("#### IV: ");
            printHexString(iv);
    }
    return((*original_target)(ctx,type,impl,key,iv,enc));
}

//function prototype -  int EVP_CIPHER_CTX_set_key_length(EVP_CIPHER_CTX *x, int keylen);
int EVP_CIPHER_CTX_set_key_length(EVP_CIPHER_CTX *x, int keylen) {
    int (*original_target)(EVP_CIPHER_CTX *x, int keylen);
        key_len = keylen;
        *(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CIPHER_CTX_set_key_length");
        return((*original_target)(x,keylen));
}

//function prototype -  int EVP_CipherFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl);
int EVP_CipherFinal_ex(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl) {
        int (*original_target)(EVP_CIPHER_CTX *ctx, unsigned char *outm, int *outl);
       int ret;

        *(void **)(&original_target) = dlsym(RTLD_NEXT, "EVP_CipherFinal_ex");
        ret = original_target(ctx,outm,outl);
        printf(" %s\n##### CipherFinal\n",outm);
        return ret;
}

The compiled shared object is loaded using the LD_PRELOAD environment variable to hook the defined calls and output the decrypted application content:

# LD_PRELOAD="/home/plixer/evphook.so" perl /home/plixer/scrutinizer/html/fcgi/scrut_fcgi.fcgi
### Decrypt Init:
#### Key: 5B 1F 31 FC 73 F8 C5 5F E2 52 DA A2 3C 76 EA DC 0E AB 3A A9 9F 73 C1 E3 49 32 73 D5 17 2F D1 FC
#### IV: AC D3 F3 26 E3 86 63 8F 66 BA 2F 65 3B C6 BA 93 00 FB C2 01 00 00 00 00 61 02 00 00 00 00 00 00
#!/usr/bin/perl
#START #UTF-8#
# http://www.perl.com/pub/2012/04/perlunicook-standard-preamble.html #UTF-8#
use utf8;                       # so literals and identifiers can be in UTF-8 #UTF-8#
use v5.16;                      # or later to get "unicode_strings" feature #UTF-8#
use strict;                     # quote strings, declare variables #UTF-8#
use warnings;                   # on by default #UTF-8#
use warnings qw(FATAL utf8);    # fatalize encoding glitches #UTF-8#
use open qw(:std :utf8);        # undeclared streams in UTF-8 #UTF-8#

#END #UTF-8#

# sanitize known environment variables.
use Plixer::Util::Taint qw( untaint_environment );

BEGIN {
# Bug 24156 - force LANG=en_US.UTF-8 in Scrutinizer
$ENV{LANG} = 'en_US.UTF-8';
untaint_environment();
}

With access to the decrypted application content further testing identified multiple vulnerabilities, which resulted in unauthenticated users being able to compromise the application server and pivot further into the environment. The details of the vulnerabilities can be found in our public disclosure repository:

https://github.com/atredispartners/advisories/blob/master/ATREDIS-2023-0001.md

It is worth noting that Plixer made the disclosure process effortless and were communicative during the process, it was refreshing to work with a vendor who was accepting of our report and prioritized the remediation process.