How a spam mail led to CVE-2018-18898
We at Selfnet use Best Practical's open-source ticketing system Request
Tracker (RT) to handle emails sent to support@selfnet.de. For some
reason, I happen to be one of the admins of our RT instance. One day, people
using the RT complained that that its web interface has become
unresponsive. Logging into the RT server, I noticed that the RT server
processes was running at 100% CPU load. Attaching strace to it
showed no activity, meaning that it's
spinning CPU cycles one something internal not interacting with the OS. Doing what every lazy admin
does, I restarted the RT service hoping that this was a single event
caused by cosmic rays or some other glitch.
As you might have already guessed, that wasn't the case and the RT locked up again. Time for some deeper investigation. So I created a core dump of the RT processes and inspected it with gdb to figure out what RT was getting stuck at:
$ gdb perl core.7780
(gdb) bt
#0 0x000055b414603727 in S_regcppush (my_perl=my_perl@entry=0x55b416538010, rex=rex@entry=0x55b41f71d300, parenfloor=<optimized out>, maxopenparen=<optimized out>) at regexec.c:328
#1 0x000055b414609e0a in S_regmatch (prog=<optimized out>, startpos=<optimized out>, reginfo=<optimized out>, my_perl=0x55b41f8acc58) at regexec.c:7361
#2 S_regtry (my_perl=my_perl@entry=0x55b416538010, reginfo=reginfo@entry=0x7ffe4a279360, startposp=startposp@entry=0x7ffe4a279350) at regexec.c:3628
#3 0x000055b4146152b8 in Perl_regexec_flags (my_perl=0x55b416538010, rx=0x55b41f6c5bc0,
stringarg=0x55b41f88a77a "\"... \"[REDACTED].\" <[REDACTED]@[REDACTED].com.br>, \"[REDACTED]\" <[REDACTED]@[REDACTED].com.br>, [REDACTED] <[REDACTED]@gmail.com>, \"[REDACTED] I"..., strend=0x55b41f88c69f "",
strbeg=0x55b41f88a77a "\"... \"[REDACTED].\" <[REDACTED]@[REDACTED].com.br>, \"[REDACTED]\" <[REDACTED]@[REDACTED].com.br>, [REDACTED] <[REDACTED]@gmail.com>, \"[REDACTED] I"..., minend=<optimized out>, sv=0x55b4193b5d80, data=0x0, flags=1) at regexec.c:3158
#4 0x000055b4145a2453 in Perl_pp_subst (my_perl=0x55b416538010) at pp_hot.c:2980
#5 0x000055b41459c7a6 in Perl_runops_standard (my_perl=0x55b416538010) at run.c:41
#6 0x000055b414522775 in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2483
#7 perl_run (my_perl=0x55b416538010) at perl.c:2411
#8 0x000055b4144fb9fd in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:116
Seems like RT is busy parsing Email addresses using regular expressions, nothing out of the ordinary given that RT is written in Perl. To figure out how exactly RT got there, we'll need to know which file and which line is currently being executed. A bit of googling brought up this answer on stackoverflow, explaining how to access the Perl interpreter's internal state. With this in mind we can get to what we want to know.
(gdb) p my_perl->Icurcop->cop_file
$2 = 0x55b4193baf00 "/usr/share/perl5/Email/Address/List.pm"
(gdb) p my_perl->Icurcop->cop_line
$3 = 310
Looking into that file, we now know for sure, that RT is busy parsing email addresses using some kind of complex regex.
if ( $line =~ s/^($CRE{'mailbox'})($RE{cfws}*)(?=,|;|$)//o
|| ($line =~ s/^($CRE{'obs-mailbox'})($RE{cfws}*)(?=,|;|$)//o and $obsolete = 1)
Since I don't know Perl and have absolutely no intention of learning it, I was more or less lost at this point. To at least make our RT instance usable again, I asked the admin of our mail server if there was an email stuck in the queue that contained the email address from the stack trace. After confirming that this was the case and the email was just spam they deleted the email from the queue gave me a copy for further examination.
I could have stopped here since our RT was now
back in service, but I wasn't exactly happy with a spam mail bringing
down our ticket system. Knowing that Email::Address::List is used to
extract email addresses from incoming mails, I hacked a simple Perl
script that parses the offending email using Email::Address::List. As
to be expected, this script didn't finish in timely manner. Some random
editing and checking if parsing still takes excessively long yielded
this concise test case:
use strict;
use Email::Address::List;
my $header = <<'END';
"AAAA" <a@a.com.br>, "...
"A A A A A A A." <a@a.com.br>, "A A A A A" <aa@a.com.br>,
END
my @list = Email::Address::List->parse($header);
foreach my $e ( @list ) {
if ($e->{'type'} eq 'mailbox') {
print "an address: ", $e->{'value'}->format ,"\n";
}
else {
print $e->{'type'}, "\n"
}
}
Running this script with Email::Address::List < 0.06 takes 3 seconds,
much too long given the size of the input. Doing some small edits,
such removing "... from the first line brings parsing speed back to
instant. I sent that test case to Alex Vandiver since his name was next
to copy of the Email::Address::List module on CPAN. He was able to reproduce
this issue and coordinated a new release of that module:
metacpan.org/release/BPS/Email-Address-List-0.06
Changes for version 0.06 - 2019-01-02
Changes to address CVE-2018-18898 which could allow DDoS-type attacks. Thanks to Lukas Kramer for reporting the issue and Alex Vandiver for contributing fixes. Fix pathological backtracking for unkown regex Fix pathological backtracking in obs-phrase(i.e. obs-display-name) Fix pathological backtracking in cfws, quoted strings
And that is the story on how being curious can lead to greater good.
Random notes:
In the process of writing that blog post I needed to recreate the gdb
output from when the issue occurred. I still had the coredump around,
but Perl has been upgraded, so I couldn't just run $ gdb perl
core.7780 to examine the core dump. After extracting the proper
version of Perl including matching debug symbols, I wasn't able to
convince gdb to use the extracted debug symbols. Luckily, there's
eu-unstrip from elfutils that - as the name implies - does the
inverse of strip, thus combining a stripped executable and debug
symbols into a an executable with debug symbols that's usable by gdb.