Selfnet Blog

Feb 04, 2019

How a spam mail led to CVE-2018-18898

We at Selfnet use Best Practical's open-source ticketing system Request Tracker (RT) to handle emails sent to support@selfnet.de. For some reason, I happen to be one of the admins of our RT instance. One day, people using the RT complained that that its web interface has become unresponsive. Logging into the RT server, I noticed that the RT server processes was running at 100% CPU load. Attaching strace to it showed no activity, meaning that it's spinning CPU cycles one something internal not interacting with the OS. Doing what every lazy admin does, I restarted the RT service hoping that this was a single event caused by cosmic rays or some other glitch.

As you might have already guessed, that wasn't the case and the RT locked up again. Time for some deeper investigation. So I created a core dump of the RT processes and inspected it with gdb to figure out what RT was getting stuck at:

$ gdb perl core.7780
(gdb) bt
#0  0x000055b414603727 in S_regcppush (my_perl=my_perl@entry=0x55b416538010, rex=rex@entry=0x55b41f71d300, parenfloor=<optimized out>, maxopenparen=<optimized out>) at regexec.c:328
#1  0x000055b414609e0a in S_regmatch (prog=<optimized out>, startpos=<optimized out>, reginfo=<optimized out>, my_perl=0x55b41f8acc58) at regexec.c:7361
#2  S_regtry (my_perl=my_perl@entry=0x55b416538010, reginfo=reginfo@entry=0x7ffe4a279360, startposp=startposp@entry=0x7ffe4a279350) at regexec.c:3628
#3  0x000055b4146152b8 in Perl_regexec_flags (my_perl=0x55b416538010, rx=0x55b41f6c5bc0, 
    stringarg=0x55b41f88a77a "\"... \"[REDACTED].\" <[REDACTED]@[REDACTED].com.br>, \"[REDACTED]\" <[REDACTED]@[REDACTED].com.br>, [REDACTED] <[REDACTED]@gmail.com>, \"[REDACTED] I"..., strend=0x55b41f88c69f "", 
    strbeg=0x55b41f88a77a "\"... \"[REDACTED].\" <[REDACTED]@[REDACTED].com.br>, \"[REDACTED]\" <[REDACTED]@[REDACTED].com.br>, [REDACTED] <[REDACTED]@gmail.com>, \"[REDACTED] I"..., minend=<optimized out>, sv=0x55b4193b5d80, data=0x0, flags=1) at regexec.c:3158
#4  0x000055b4145a2453 in Perl_pp_subst (my_perl=0x55b416538010) at pp_hot.c:2980
#5  0x000055b41459c7a6 in Perl_runops_standard (my_perl=0x55b416538010) at run.c:41
#6  0x000055b414522775 in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2483
#7  perl_run (my_perl=0x55b416538010) at perl.c:2411
#8  0x000055b4144fb9fd in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:116

Seems like RT is busy parsing Email addresses using regular expressions, nothing out of the ordinary given that RT is written in Perl. To figure out how exactly RT got there, we'll need to know which file and which line is currently being executed. A bit of googling brought up this answer on stackoverflow, explaining how to access the Perl interpreter's internal state. With this in mind we can get to what we want to know.

(gdb) p my_perl->Icurcop->cop_file 
$2 = 0x55b4193baf00 "/usr/share/perl5/Email/Address/List.pm"
(gdb) p my_perl->Icurcop->cop_line
$3 = 310

Looking into that file, we now know for sure, that RT is busy parsing email addresses using some kind of complex regex.

if ( $line =~ s/^($CRE{'mailbox'})($RE{cfws}*)(?=,|;|$)//o
    || ($line =~ s/^($CRE{'obs-mailbox'})($RE{cfws}*)(?=,|;|$)//o and $obsolete = 1)

Since I don't know Perl and have absolutely no intention of learning it, I was more or less lost at this point. To at least make our RT instance usable again, I asked the admin of our mail server if there was an email stuck in the queue that contained the email address from the stack trace. After confirming that this was the case and the email was just spam they deleted the email from the queue gave me a copy for further examination.

I could have stopped here since our RT was now back in service, but I wasn't exactly happy with a spam mail bringing down our ticket system. Knowing that Email::Address::List is used to extract email addresses from incoming mails, I hacked a simple Perl script that parses the offending email using Email::Address::List. As to be expected, this script didn't finish in timely manner. Some random editing and checking if parsing still takes excessively long yielded this concise test case:

use strict;
use Email::Address::List;

my $header = <<'END';
"AAAA" <a@a.com.br>, "...
"A A A A A A A." <a@a.com.br>, "A A A A A" <aa@a.com.br>,
END

my @list = Email::Address::List->parse($header);
foreach my $e ( @list ) {
  if ($e->{'type'} eq 'mailbox') {
    print "an address: ", $e->{'value'}->format ,"\n";
  }
  else {
    print $e->{'type'}, "\n"
  }
}

Running this script with Email::Address::List < 0.06 takes 3 seconds, much too long given the size of the input. Doing some small edits, such removing "... from the first line brings parsing speed back to instant. I sent that test case to Alex Vandiver since his name was next to copy of the Email::Address::List module on CPAN. He was able to reproduce this issue and coordinated a new release of that module:

metacpan.org/release/BPS/Email-Address-List-0.06

Changes for version 0.06 - 2019-01-02

Changes to address CVE-2018-18898 which could allow DDoS-type attacks. Thanks to Lukas Kramer for reporting the issue and Alex Vandiver for contributing fixes. Fix pathological backtracking for unkown regex Fix pathological backtracking in obs-phrase(i.e. obs-display-name) Fix pathological backtracking in cfws, quoted strings

And that is the story on how being curious can lead to greater good.

Random notes: In the process of writing that blog post I needed to recreate the gdb output from when the issue occurred. I still had the coredump around, but Perl has been upgraded, so I couldn't just run $ gdb perl core.7780 to examine the core dump. After extracting the proper version of Perl including matching debug symbols, I wasn't able to convince gdb to use the extracted debug symbols. Luckily, there's eu-unstrip from elfutils that - as the name implies - does the inverse of strip, thus combining a stripped executable and debug symbols into a an executable with debug symbols that's usable by gdb.