Extract Rust specific strings from binaries #791 #836

Arker123 · 2023-07-17T19:05:10Z

This pull request introduces a new feature to extract Rust specific strings from binaries.
Solves: #791

Arker123 · 2023-07-17T19:05:52Z

There are still several tasks pending, such as updating references from source code, comments, etc. Therefore, I'm marking this pull request as a draft for now.

mr-tz · 2023-07-18T07:10:07Z

On first glance this appears to be identical or very close to the Go extract script. Can we just reuse that code or functions from it, so we don't repeat the same code twice?

Arker123 · 2023-07-28T07:33:40Z

On first glance this appears to be identical or very close to the Go extract script. Can we just reuse that code or functions from it, so we don't repeat the same code twice?

Agreed! Let's consolidate the code and reuse the functions from the Go extract script in a separate PR after merging this one. We can create a file something like language/utils.py to house the shared functions. Thanks for the suggestion!

Arker123 · 2023-07-28T07:38:53Z

The current algorithm is based on pull request #791 (comment), and it's progressing well.
TODO: Extract references from the .text segment as well and improve code style and comments.

On a positive note, the algorithm appears to be splitting the strings quite effectively, as shown in the attached image.

floss/language/rust/extract.py

Arker123 · 2023-08-05T06:39:17Z

TODO: Refactor comments and type hints

Arker123 · 2023-08-05T09:06:42Z

The current outcome appears promising.

floss/language/rust/coverage.py

floss/language/rust/extract.py

.github/mypy/mypy.ini

floss/language/rust/coverage.py

floss/language/rust/extract.py

Co-authored-by: Moritz <[email protected]>

Arker123 · 2023-08-22T07:38:29Z

Or we can consider adding a new field "offset_end" to the StaticString class instead of managing both ref_data and Static_string separately like in latest commit? I'd like to know your opinion.

floss/language/rust/extract.py

floss/results.py

floss/language/rust/extract.py

* simplify rdata section extraction * add initial Rust extract tests * simplify rust string extraction

mr-tz

sorry, I think I didn't properly format my changes (in the PR to the fork)

Arker123 · 2023-08-22T19:08:11Z

No problem at all! Thanks for the help! 😄

mr-tz · 2023-08-23T08:48:52Z

floss/language/rust/extract.py

+    xrefs_lea = find_lea_xrefs(pe)
+    xrefs_push = find_push_xrefs(pe)
+    xrefs_mov = find_mov_xrefs(pe)


do we want to do all of these all the time (for both architectures or specific to 32/64 bit?)

Push and mov operations are only specific to the i386 architecture, and they have been handled in the latest commit. Thanks

mr-tz · 2023-08-23T08:49:25Z

floss/language/rust/extract.py

can you please share some current coverage output for 32 and 64 bit samples?

Currently, both are at approximately 90% coverage.
coverage32.txt
coverage64.txt

can you repeat this for a few random samples and just share the stats here?

On random samples from VT:-
result.txt
Average:- 94.5%
Low:- 88%
High:- 99%

great, that looks pretty promising!

floss/language/go/coverage.py

Co-authored-by: Moritz <[email protected]>

mr-tz · 2023-08-23T09:18:57Z

floss/language/utils.py

+        code = section.get_data()
+
+        if pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_AMD64"]:
+            xrefs: Iterable[VA] = []  # no push instructions on amd64


can we be sure string references would never be pushed? probably, but can you add some context/references around this? the comment is a little misleading currently

I've carefully checked amd64 binary files, and as of now, I haven't come across any mov or push instructions related to string references. However, considering the complexity of the Instruction set, I can't rule out the possibility of some rare cases escaping my notice. To ensure completeness, I've included this in the latest commit.

mr-tz · 2023-08-23T09:20:21Z

floss/language/utils.py

+        code = section.get_data()
+
+        if pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_AMD64"]:
+            xrefs: Iterable[VA] = []  # no mov instructions on amd64


same comment as above

…into rust-strings

mr-tz · 2023-08-23T11:36:04Z

floss/language/utils.py

+    # It is 0 in case of Rust extraction because we want to extract all strings from binary file
+    # while in case of Go extraction, we want to extract only large strings (len > 2800) from binary file


let's only leave the respective comments where this is called, so please remove here

Initial implementation of Rust specific strings

a75661e

Arker123 added 2 commits July 28, 2023 12:59

New algorithm

e7c7595

code style

8a394bb

williballenthin reviewed Aug 1, 2023

View reviewed changes

floss/language/rust/extract.py Outdated Show resolved Hide resolved

floss/language/rust/extract.py Outdated Show resolved Hide resolved

floss/language/rust/extract.py Outdated Show resolved Hide resolved

Arker123 added 6 commits August 5, 2023 11:35

Implemented separation of references from .text segment

e9ca68e

Added rust coverage script

45978ea

Introduced shared functions into language/utils.py

4cbffaf

Refractored Go and Rust extraction files

f128d19

Removed unused functions

80dce99

Modularized code into separate functions

13c8920

Arker123 added 3 commits August 5, 2023 13:23

Merge remote-tracking branch 'origin/master' into rust-strings

27958fb

Refractored comments and type hints

e074722

Tweaks

dbf7ad1

Arker123 marked this pull request as ready for review August 5, 2023 09:00

mr-tz requested changes Aug 7, 2023

View reviewed changes

mr-tz reviewed Aug 7, 2023

View reviewed changes

floss/language/rust/extract.py Outdated Show resolved Hide resolved

Arker123 added 4 commits August 11, 2023 18:01

Update coverage Script

bbd3d53

Tweaks

4839543

Minor fixes

3ebd075

code style

226486e

mr-tz requested changes Aug 14, 2023

View reviewed changes

Arker123 and others added 2 commits August 14, 2023 14:59

Apply suggestions from code review

c46410e

Co-authored-by: Moritz <[email protected]>

Tweaks

8fabe4b

Further Improvised Design

9fe75c7

Tweaks

a67f9f2

mr-tz reviewed Aug 22, 2023

View reviewed changes

floss/language/rust/extract.py Outdated Show resolved Hide resolved

floss/language/rust/extract.py Outdated Show resolved Hide resolved

Design Tweaks

07a7558

mr-tz reviewed Aug 22, 2023

View reviewed changes

floss/results.py Outdated Show resolved Hide resolved

Arker123 added 2 commits August 22, 2023 15:07

Updated Design Structure

5a6fdb6

Cleanup

1650f8b

mr-tz reviewed Aug 22, 2023

View reviewed changes

floss/language/rust/extract.py Outdated Show resolved Hide resolved

Rust updates (#7)

6cdccb3

* simplify rdata section extraction * add initial Rust extract tests * simplify rust string extraction

mr-tz reviewed Aug 22, 2023

View reviewed changes

Added push and mov xrefs for i386 arch and test updates

62405fe

mr-tz reviewed Aug 23, 2023

View reviewed changes

Tweaks

c98450d

mr-tz reviewed Aug 23, 2023

View reviewed changes

floss/language/go/coverage.py Outdated Show resolved Hide resolved

Update floss/language/go/coverage.py

57fc902

Co-authored-by: Moritz <[email protected]>

mr-tz reviewed Aug 23, 2023

View reviewed changes

Arker123 added 4 commits August 23, 2023 15:45

Add push and mov for amd64

ef27592

Merge branch 'rust-strings' of https://github.com/Arker123/flare-floss …

1909255

…into rust-strings

Update Comments

6011ea7

Tweaks

890ba55

mr-tz reviewed Aug 23, 2023

View reviewed changes

Arker123 added 2 commits August 23, 2023 17:14

Comment Tweaks

df20ec1

Tweaks

2fdb823

mr-tz merged commit 567fc95 into mandiant:master Aug 23, 2023
19 checks passed

mr-tz mentioned this pull request Aug 25, 2023

extract strings from binaries compiled from Rust #791

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract Rust specific strings from binaries #791 #836

Extract Rust specific strings from binaries #791 #836

Arker123 commented Jul 17, 2023

Arker123 commented Jul 17, 2023

mr-tz commented Jul 18, 2023

Arker123 commented Jul 28, 2023

Arker123 commented Jul 28, 2023

Arker123 commented Aug 5, 2023

Arker123 commented Aug 5, 2023

Arker123 commented Aug 22, 2023

mr-tz left a comment

Arker123 commented Aug 22, 2023

mr-tz Aug 23, 2023

Arker123 Aug 23, 2023

mr-tz Aug 23, 2023

Arker123 Aug 23, 2023

mr-tz Aug 23, 2023

Arker123 Aug 23, 2023

mr-tz Aug 23, 2023

mr-tz Aug 23, 2023

Arker123 Aug 23, 2023

mr-tz Aug 23, 2023

mr-tz Aug 23, 2023

Arker123 Aug 23, 2023

		# It is 0 in case of Rust extraction because we want to extract all strings from binary file
		# while in case of Go extraction, we want to extract only large strings (len > 2800) from binary file

Extract Rust specific strings from binaries #791 #836

Extract Rust specific strings from binaries #791 #836

Conversation

Arker123 commented Jul 17, 2023

Arker123 commented Jul 17, 2023

mr-tz commented Jul 18, 2023

Arker123 commented Jul 28, 2023

Arker123 commented Jul 28, 2023

Arker123 commented Aug 5, 2023

Arker123 commented Aug 5, 2023

Arker123 commented Aug 22, 2023

mr-tz left a comment

Choose a reason for hiding this comment

Arker123 commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment