Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

epanos depends on older/non-current versions of pycparser #1

Closed
nihilus opened this issue Jun 15, 2015 · 39 comments
Closed

epanos depends on older/non-current versions of pycparser #1

nihilus opened this issue Jun 15, 2015 · 39 comments
Labels

Comments

@nihilus
Copy link

nihilus commented Jun 15, 2015

Seems like you got the indentation wrong when you checked in the code... I cannot get it running and I am not a python wizard.

@drvink
Copy link
Owner

drvink commented Jun 16, 2015

Can you provide a backtrace or more details about the problem you're having? (I can guarantee you that indentation isn't the problem here, as others have been able to successfully run the code from this repo!)

@nihilus
Copy link
Author

nihilus commented Jun 16, 2015

I will do a check on my XP VM when I get home...

@drvink
Copy link
Owner

drvink commented Jun 20, 2015

By the way, just in case you're expecting this to work on any MIPS binary, you'll probably be disappointed (but if you want an example file that will successfully decompile, I can provide one).

@nihilus
Copy link
Author

nihilus commented Jun 20, 2015

@techbliss Do you recall the errors when trying to run epanos?

@techbliss
Copy link

Well its due to cpp binary.
it cant find it even when in path, i tried several ways to add path /folders inviroment setting and more etc in cpp.py for locating the binary,Without luck.
it dosent say, but i asume its cpp.exe from minigw.
So throws a Path error no matter what i do.

@drvink
Copy link
Owner

drvink commented Jun 20, 2015

Just as a sanity check, first make sure pycparser, flufl.enum, and pyc-fmtstr-parser are installed.

Can you try using the C preprocessor binary from clang and fake stdlib headers from pycparser? Don't forget to set the paths in epanos.py.

When I wrote epanos, I was using the C preprocessor binary from lcc that pycparser used to bundle, so if you have problems with the clang one, the lcc one should definitely work.

@techbliss
Copy link

Yes pycparser, flufl.enum, and pyc-fmtstr-parser are installed.correctly, i can call them from ida python shell.
Ill give it a shot tomorrow when i have access to my laptop.

@drvink
Copy link
Owner

drvink commented Jun 20, 2015

OK, let me know how it goes. :)

@nihilus
Copy link
Author

nihilus commented Jun 20, 2015

I figured out I had to fix this as well: nihilus@526f093 for some reason unknown :-/ I am using Python 2.7.6

@techbliss
Copy link

traceback without C preprocessor binary from clang and fake stdlib headers from pycparser?
I think i set cpp path correct in epanos.py

error

#include <stdio.h>
#include <string.h>
#include <math.h>
#include <assert.h>
#include <time.h>
#include <GL/gl.h>
#include <GL/glu.h>

<EPANOS_cpp>:34:8: before: _ZN4_q128instanceE
Traceback (most recent call last):
  File "C:\Program Files (x86)\IDA 6.8\python\idaapi.py", line 601, in IDAPython_ExecScript
    execfile(script, g)
  File "C:/Program Files (x86)/IDA 6.8/plugins/epanos/epanos.py", line 87, in <module>
    print run(decompile=True)
  File "C:/Program Files (x86)/IDA 6.8/plugins/epanos/epanos.py", line 68, in run
    return decomp.run(externs, text_fns, cpp_filter, cpp_all, decompile)
  File "C:/Program Files (x86)/IDA 6.8/plugins/epanos\decomp\c\gen.py", line 110, in run
    data_txt = data.get_data(data_segs, cpp_filter)
  File "C:/Program Files (x86)/IDA 6.8/plugins/epanos\decomp\data.py", line 123, in get_data
    cdecl.get_decls(declstrs, cpp_in).decls[utils.cpp_decomp_tag])
  File "C:/Program Files (x86)/IDA 6.8/plugins/epanos\decomp\c\decl.py", line 99, in get_decls
    return process_ast(parser.parse(decls, filename=utils.cpp_decomp_tag),
  File "C:\Python27\lib\site-packages\pycparser\c_parser.py", line 139, in parse
    debug=debuglevel)
  File "C:\Python27\lib\site-packages\pycparser\ply\yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "C:\Python27\lib\site-packages\pycparser\ply\yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "C:\Python27\lib\site-packages\pycparser\c_parser.py", line 1673, in p_error
    column=self.clex.find_tok_column(p)))
  File "C:\Python27\lib\site-packages\pycparser\plyparser.py", line 55, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
ParseError: <EPANOS_cpp>:34:8: before: _ZN4_q128instanceE

@drvink
Copy link
Owner

drvink commented Jun 20, 2015

It's been a while since I've looked at the code, but I'm pretty sure if it's getting to that point, then the C preprocessor is at least working, so that's good.

As for getting it to work on your target, you're almost certainly going to have to modify the code and do some manual annotations on your database. Here's an example to help get you started. To use it:

  1. Open ep in IDA as a big-endian (mipsb) file and let IDA analyze the file.
  2. Run the attached ep.idc script.
  3. Run the attached fptrs.py script.
  4. Save your database.
  5. Run epanos.py and say "yes" to the prompt.

The decompilation result should show up in the message pane.

@techbliss
Copy link

still getting a error (another one)i will look at it tommorow

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include <assert.h>
#include <time.h>
#include <GL/gl.h>
#include <GL/glu.h>

'tuple' object has no attribute 'decls'
Traceback (most recent call last):
File "C:\Program Files (x86)\IDA 6.8\python\idaapi.py", line 601, in IDAPython_ExecScript
execfile(script, g)
File "C:/Program Files (x86)/IDA 6.8/plugins/epanos/epanos.py", line 87, in
print run(decompile=True)
File "C:/Program Files (x86)/IDA 6.8/plugins/epanos/epanos.py", line 68, in run
return decomp.run(externs, text_fns, cpp_filter, cpp_all, decompile)
File "C:/Program Files (x86)/IDA 6.8/plugins/epanos\decomp\c\gen.py", line 110, in run
data_txt = data.get_data(data_segs, cpp_filter)
File "C:/Program Files (x86)/IDA 6.8/plugins/epanos\decomp\data.py", line 123, in get_data
cdecl.get_decls(declstrs, cpp_in).decls[utils.cpp_decomp_tag])
AttributeError: 'tuple' object has no attribute 'decls'

@drvink
Copy link
Owner

drvink commented Jul 25, 2015

Hi @techbliss, were you able to get this working? Please let me know if you're still having trouble.

@nihilus
Copy link
Author

nihilus commented Jul 26, 2015

@techbliss Wake up!

@techbliss
Copy link

sorry for late reply.
No i haven't been able to get past the tuple error that i commented on the 20 Jun.
.
I think Tuple error may have to do with cpp arguments.
cpp_args in cpp.py

cpp_args:
            Set this to the command line arguments to cpp. Be careful with
            quotes - it's best to pass a raw string (r'') here. For example:
            r'-I../utils/fake_libc_include'
            If several arguments are required, pass a list of strings.

Can you explain more about the arguments.

@nihilus
Copy link
Author

nihilus commented Jul 30, 2015

Me neither... The tuple error seems to be a show-stopper :-(

@nihilus
Copy link
Author

nihilus commented Jul 30, 2015

@drvink However feel free to from my MIPS-fork of snowman (http://www.gitgub.com/nihilus/snowman) and help me implement the ISA :-)

@drvink
Copy link
Owner

drvink commented Aug 3, 2015

I finally had a chance to sit down and take a look at this. I figured it out--sorry it took so long. :)

If you use cpp.exe from LCC, it should just work. Alternatively, if you want to use clang as your preprocessor, apply this patch (edit the paths accordingly, of course):

diff --git a/epanos/epanos.py b/epanos/epanos.py
index 7a09efb..076601e 100644
--- a/epanos/epanos.py
+++ b/epanos/epanos.py
@@ -22,13 +22,13 @@ except:

 def run(decompile=True):
     # change cpp path/args as needed
-    pycparser_dir = r'%s\local\pycparser\utils' % os.environ['HOMEPATH']
+    pycparser_dir = r'C:\Python27\Lib\site-packages\pycparser'
     decomp_dir = os.path.dirname(os.path.abspath(__file__))
-    cpppath = r'%s\cpp.exe' % pycparser_dir
-    cppargs = [
+    cpppath = r'C:\LLVM\bin\clang.exe'
+    cppargs = [r'-E'] + [
         r'-I%s' % path for path in
-        (r'%s\fake_libc_include' % pycparser_dir,
-         decomp_dir)]
+        (r'C:\path\to\fake_libc_include',
+         decomp_dir)] + ['-']
     decomp.set_cpp(cpppath, cppargs)

     ## Desired functions (missing functions will cause a warning if encountered

cpp.exe from gcc should also work. I've tried all three and I get the same decompilation results regardless of which I use.

Please let me know if this works for you!

@nihilus
Copy link
Author

nihilus commented Aug 3, 2015

Wont work... I included llc cpp in my fork and it doesnt work.

Check at my fork where Ive included the dependencies etc.

@techbliss
Copy link

can you specify a little more about

r'C:\path\to\fake_libc_include'

@drvink
Copy link
Owner

drvink commented Aug 3, 2015

@nihilus OK, will look in a moment.

@techbliss Clone eliben/pycparser and change that path to the location of the fake_libc_include directory in the cloned repo.

@techbliss
Copy link

when runned the fptrs.py.It sets the new_fptrs.
then in epanos.py it get the new_fptrs

fptrs

    externs = frozenset(
        ['expm1', 'drand48', 'fmodf', 'srand48', 'glMatrixMode',
         'glScalef', 'glPopMatrix', 'scanf', 'glColor3f', 'sscanf',
         'strcmp', 'strcspn', 'malloc', 'strlen', 'free', 'strncmp',
         'glVertex2f', 'printf', 'glTranslatef', 'glPushMatrix', 'glRotatef',
         'gettimeofday', 'exit', 'glMultMatrixf', 'glBegin',
         'glEnd', 'glViewport', 'glLoadIdentity', 'gluPerspective', 'glClear',
         'floor', 'glFinish', 'time', 'glClearColor', 'glShadeModel'])
    text_fns = frozenset(
        ['value__GfN21', 'hls_to_rgb__GfN21PfN24', 'exprand__Gf',
         'addToSeq__GP7animSeqP11animCommand', 'readAnimation__Gv',
         'stopAnimation__Gv', 'killSeq__GP7animSeq',
         'processCommand__GP11animCommand', 'tasteQueue__Gv',
         'setacttargets__Gv', 'twixt__GiPff', 'foldtwixt__GiPffT3',
         'drawshape__GiT1', 'drawit__Gv', 'animateacts__Gv',
         'createBlankActAnim__Gv', 'createActTable__Gv', 'reshape__GiT1',
         'display__Gv', 'main'])

But when running the fptrs.py nothing happends, or at least nothing at all shows after running.
Could it be, that when no new strings are made, its hard to find them in the tuble.
Would if fptrs.py failed.

anyway to modify fptrs.py to check if the script actuelly worked.

@drvink
Copy link
Owner

drvink commented Aug 3, 2015

fptrs.py just sets the targets of a bunch of jalr $t9 instances that IDA couldn't resolve. If you want to confirm it's working, go to address 0x10005280 (just for an example), then run the script. ; glVertex2f should have appeared to the right of the instruction. (You're running fptrs.py from the zip file I linked earlier, right? The one in the epanos distribution isn't the one you want to run.)

@techbliss
Copy link

yes im am using the one you supplied.

before

la      $t9, glBegin
jalr    $t9 ; glBegin
li      $a0, 9           # mode
la      $s0, glVertex2f
la      $s1, square
move    $t9, $s0
lwc1    $f12, (square - 0x10008E50)($s1)  # x
jalr    $t9 ; glVertex2f           #0x10005280
lwc1    $f13, (flt_10008E54 - 0x10008E50)($s1)
move    $t9, $s0
lwc1    $f12, (flt_10008E58 - 0x10008E50)($s1)  # x
jalr    $t9 ; glVertex2f
lwc1    $f13, (flt_10008E5C - 0x10008E50)($s1)
move    $t9, $s0
lwc1    $f12, (flt_10008E60 - 0x10008E50)($s1)  # x
jalr    $t9 ; glVertex2f
lwc1    $f13, (flt_10008E64 - 0x10008E50)($s1)
move    $t9, $s0
lwc1    $f12, (flt_10008E68 - 0x10008E50)($s1)  # x
jalr    $t9 ; glVertex2f
lwc1    $f13, (flt_10008E6C - 0x10008E50)($s1)
la      $t9, glEnd
jalr    $t9 ; glEnd
nop
ld      $ra, 0x20+var_8($sp)

after ftprs.py

                la      $t9, glBegin
                jalr    $t9 ; glBegin
                li      $a0, 9           # x
                la      $s0, glVertex2f
                la      $s1, square
                move    $t9, $s0
                lwc1    $f12, (square - square)($s1)  # x
                jalr    $t9 ; glVertex2f     #0x10005280
                lwc1    $f13, (square+4 - square)($s1)  # y
                move    $t9, $s0
                lwc1    $f12, (square+8 - square)($s1)  # x
                jalr    $t9 ; glVertex2f
                lwc1    $f13, (square+0xC - square)($s1)  # y
                move    $t9, $s0
                lwc1    $f12, (square+0x10 - square)($s1)  # x
                jalr    $t9 ; glVertex2f
                lwc1    $f13, (square+0x14 - square)($s1)  # y
                move    $t9, $s0
                lwc1    $f12, (square+0x18 - square)($s1)  # x
                jalr    $t9 ; glVertex2f
                lwc1    $f13, (square+0x1C - square)($s1)  # y
                la      $t9, glEnd
                jalr    $t9 ; glEnd
                nop
                ld      $ra, 0x20+var_8($sp)

there is a little difference but to be honest i am not sure if the changes are correct.
still getting the error.
i used clang this time

plugins/epanos\decomp\data.py", line 127, in get_data decls = izip(data, cdecl.get_decls(declstrs, cpp_in).decls[utils.cpp_decomp_tag]) AttributeError: 'tuple' object has no attribute 'decls'

@drvink
Copy link
Owner

drvink commented Aug 3, 2015

This is harder than writing the decompiler was!

  • What version of Python are you using? I'm on 2.7.9, Windows 32-bit, from python.org (not a cygwin or MSYS2 Python).
  • How did you install pycparser? (I installed it via pip, but I copied the fake libc headers directory out of the git repo.)
  • Can you paste your ~/.gitconfig?
  • Are you using my repo or @nihilus's?

I'll try installing IDA in a VM and see if I can get the decompiler to run under that just to ensure there's not something special about my environment that I set up ages ago.

@techbliss
Copy link

Using your repo.
Using win x86 python 2.7.10
Installed pycparser via pip also,even tried Manuel install.
Also copying the fake header out.
@nihilus also using same setup.
Maybe issue on windows.

@drvink
Copy link
Owner

drvink commented Aug 6, 2015

I was able to successfully run the decompiler on a completely fresh Windows 7 VM just now. The trick is that you'll need an older version of pycparser: it worked for me with pip install pycparser==2.10, but the current version (2.14) did not work.

@nihilus
Copy link
Author

nihilus commented Aug 6, 2015

@drvink Ah, thx for investigating this. :-) 👍

@drvink
Copy link
Owner

drvink commented Aug 6, 2015

@nihilus Were you able to get it to run?

@nihilus
Copy link
Author

nihilus commented Aug 6, 2015

I will give it a try tonight. Been to occupied with implementing MIPS-support for Snowman.

@drvink
Copy link
Owner

drvink commented Aug 10, 2015

@nihilus @techbliss Had a chance to try with pycparser 2.10? It's great to hear you're working on an actual MIPS decompiler, by the way--the work I did here was rather specialized, not a very general-purpose tool.

@nihilus
Copy link
Author

nihilus commented Aug 10, 2015

@drvink That made things work... However I see epanos is unable to handle little-endian files. A pity :-/

@drvink
Copy link
Owner

drvink commented Aug 10, 2015

Yes. I'm afraid it's endian-ignorant, as it was basically a PoC for single binary where there were no endianness issues. I hope it may give you some ideas toward static analysis/recompilation, but it's very far from a complete solution. Endianness is probably one of the easier things you could fix in it, but honestly, extending epanos is not what I would recommend. However, I encourage you to learn from two of its design choices which have been proven in both research and industry code: mapping of types to functions and data, and mapping between instructions and types. Types are key!

You may find LLVM's MIPS TableGen files to be especially helpful in your development (and similar to epanos's instruction categorization). MIPS's instructions are fairly naturally and intuitively related to C and the way compilers familiar to any reverser will translate code. :)

@techbliss
Copy link

@drvink wasent epanos also more of proof of concept.Rather than making it decompile both big and little endian.Still great work.
Also it did jump start you @nihilus to work on your own decompiler.

https://github.com/yegord/snowman/issues/14 longest issue i have seen in a long time. :)
To be honest i am really not any of a MIPS expert , but ida plugin and python got me interested.
Also i am damn stubborn when i cant get it to run.

@nihilus
Copy link
Author

nihilus commented Aug 10, 2015

You mean that the ISA is orthogonal? ;-P
The biggest issue with endian are instructions like SWL/SWR/LWL/LWR...
Well as I am an admin of http://boomerang.sf.net I've got some clues about how to design a decompiler. :-)

@drvink
Copy link
Owner

drvink commented Aug 10, 2015

I had quite a lot of fun writing it, and I hope you're having fun as well. Another thing to keep in mind: I don't know if your goal involves human readability, but as epanos shows, the decompiled code can be damn ugly if you only want to feed it back to a compiler. :)

If epanos is of any inspiration or help to you, I'd appreciate a citation in your work!

@drvink
Copy link
Owner

drvink commented Aug 10, 2015

Also, if you were both able to get epanos to run, may I close this issue?

@techbliss
Copy link

yes you can close issue, thx for the help.

@drvink
Copy link
Owner

drvink commented Aug 10, 2015

Cheers, guys, and good luck!

@drvink drvink closed this as completed Aug 10, 2015
@drvink drvink added the bug label Aug 10, 2015
@drvink drvink changed the title Cannot run - indentation is wrong. epanos depends on older/non-current versions of pycparser Sep 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants