Metadata-Version: 1.1
Name: rekall-layout-expert
Version: 1.5.0.post4
Summary: Rekall Layout Expert
Home-page: https://www.rekall-forensic.com/
Author: The Rekall team
Author-email: rekall-discuss@googlegroups.com
License: Apache
Description: # Rekall Layout Expert
        
        Live Memory analysis on the Linux platform has traditionally been difficult to
        perform. Memory analysis requires precise knowledge of struct layout information
        in memory, usually obtained through debugging symbols generated at compile
        time. The Linux kernel is however, highly configurable, implying that debugging
        information is rarely applicable to systems other than the ones that generated
        it. For incident response applications, obtaining the relevant debugging
        information is currently a slow and manual process, limiting its usefulness in
        rapid triaging.
        
        ## How do we analyze Linux systems right now?
        
        The current process for generating a Rekall profile for a Linux system is
        tedious:
        
        1) You must find and install the kernel headers package for the same kernel as
        the running kernel (for example `apt-get install
        linux-headers-3.16.0-39-generic`).
        
        2) Then you need to build a kernel module (`rekall/tools/linux/module.c`) on
        that system to generate the debug kernel module `module_dwarf.ko`.
        
        3) Finally on a system with Rekall installed, one needs to convert this to a
        Rekall profile (using `rekall convert_profile 4.2.0-generic.zip
        4.2.0-generic.json` for example).
        
        This is hard to do in an incident response situation. Sometimes servers do not
        have the required compilers, kernel headers etc. This is especially hard if the
        kernel was cusom made. In that case it may be difficult to even find the
        required kernel headers package (it may not have even been built with the custom
        kernel). In all likelyhood you may need to copy the kernel config and System.map
        off the system you want to analyze to another system (with compiler tool chains
        and kernel headers installed) so you can build the profile.
        
        This logistical issue make it difficult to do Linux live memory analysis in
        practice - so you end up taking a memory image of the system for later analysis
        (Then you have to deal with transferring huge images around, smear and lot of
        other fun problems :-).
        
        If you really want to be prepared, you must build a huge library of kernel
        profiles in advance. For each released kernel version, you need to have every
        variation released by every distribution. For example in Ubuntu, there are
        generic and low latency variation (e.g. `linux-headers-3.16.0-39-generic`,
        `linux-headers-3.16.0-39-lowlatency`) for each minor version). You can just
        forget about having custom kernels in your library because you can not predict
        in advance what config parameters someone will change!
        
        ## Is there a better way?
        
        Have you ever found yourself uttering: "I will pay someone $1000 to find a way
        to do Linux Memory forensics without building a *$@#!% profile for every #@$#%@#
        kernel?" - I know I have!
        
        In a perferct world, we would just run Rekall on any Linux system, point it at
        `/proc/kcore` or `/dev/pmem` and just go without worrying about building
        profiles! That would be nice.
        
        We are not quite there, but almost :-). The Layout Expert is the small step
        forward. The process using the Layout Expect is much simpler:
        
        1) On the system you want to analyze, run the Layout Expert which will download
        a single Pre-AST file for every kernel version (regardless of kernel
        configuration, distribution flavour etc.).
        
        2) Then launch the layout expert, providing it the local system's config file and
        System.map:
        
        ```
        $ layout_tool make_profile --config_file_path boot/config-4.2.02.0.smp \
            --system boot/System.map-4.2.02.0.smp pre_ast_4.2.0-22.json profile.json
        
        2016-01-23 09:44:29,416 INFO     LOADING PREPROCESSOR AST FROM: pre_ast_4.2.0-22.json
        2016-01-23 09:44:34,494 INFO     DONE
        2016-01-23 09:44:34,495 INFO     LINKING INCLUDES
        2016-01-23 09:44:34,937 INFO     LINKED
        2016-01-23 09:44:34,937 INFO     EXTRACTING CONFIG FLAGS
        2016-01-23 09:44:34,994 INFO     EXTRACTED
        2016-01-23 09:44:35,108 INFO     PREPROCESSING
        2016-01-23 09:44:50,856 INFO     PREPROCESSED
        2016-01-23 09:44:50,856 INFO     Completed preprocessing pre-ast in 16 Seconds
        2016-01-23 09:44:50,856 INFO     GENERATING PURE C FILE
        2016-01-23 09:44:53,047 INFO     GENERATED
        2016-01-23 09:44:53,048 INFO     Completed generating pure C file in 2 Seconds
        2016-01-23 09:44:53,048 INFO     TRIMMING C FILE
        2016-01-23 09:45:14,340 INFO     Completed trimming C file in 21 Seconds
        2016-01-23 09:45:14,341 INFO     TRIMMED C FILE
        2016-01-23 09:45:14,354 INFO     PARSING STRUCTS
        2016-01-23 09:45:37,853 INFO     Completed parsing struct layouts in 23 Seconds
        2016-01-23 09:45:37,853 INFO     PARSED
        2016-01-23 09:45:37,853 INFO     GENERATING PROFILE
        2016-01-23 09:45:37,949 INFO     Exporting 627 structs
        2016-01-23 09:45:38,763 INFO     GENERATED
        ```
        
        The Layout Expert is able to calculate the memory layout of critical kernel
        structures at runtime on the target system without requiring extra tools, such
        as the compiler tool-chain to be pre-installed.
        
        ## How does it work?
        
        The main problem with memory analysis on Linux is that the Linux kernel is so
        configurable and customizable. For example, in order to properly parse the
        memory layout of `struct task_struct`, we can see the source:
        
        ```
        struct task_struct {
                volatile long state;    /* -1 unrunnable, 0 runnable, >0 stopped */
                void *stack;
                atomic_t usage;
                unsigned int flags;     /* per process flags, defined below */
                unsigned int ptrace;
        
        #ifdef CONFIG_SMP
                struct llist_node wake_entry;
                int on_cpu;
                struct task_struct *last_wakee;
                unsigned long wakee_flips;
                unsigned long wakee_flip_decay_ts;
        
                int wake_cpu;
        #endif
                int on_rq;
        
                int prio, static_prio, normal_prio;
                unsigned int rt_priority;
                const struct sched_class *sched_class;
                struct sched_entity se;
                struct sched_rt_entity rt;
        #ifdef CONFIG_CGROUP_SCHED
                struct task_group *sched_task_group;
        #endif
        
        #ifdef CONFIG_PREEMPT_NOTIFIERS
                /* list of struct preempt_notifier: */
                struct hlist_head preempt_notifiers;
        #endif
        
        ....
        ```
        
        Depending on kernel configuration options there will be different members
        inserted in the middle of the struct - even for the same kernel version. This is
        primarily why you need to compile a debug kernel module for every single kernel
        configuration - even of the same version. Depending on various kernel config
        options the layouts can change dramatically (sometimes if the profiles are very
        close some fields will be parsed correctly by Rekall but others wont - the
        familiar missing data in plugin outputs).
        
        
        The Layout Expert attempts to emulate the GCC compiler chains to the extent of
        being able to predict the struct layout that the compiler might decide
        on. Essentially we simulate the compilation of the kernel debug module.
        
        The GCC compiler, reads the kernel config file and then preprocesses the kernel
        headers to add or remove code depending on these configuration options. In the
        Layout Expert we wish to have a data structure that can be re-used for different
        configurations without needing the kernel config.
        
        Therefore, the Layout Expert first parses all the kernel headers into a
        Preprocessor Abstract Syntax Tree (Pre-AST for short). The Pre-AST includes all
        the possibilities of each `#ifdef` branch. This is the file which the Layout
        Expert operates on.
        
        At runtime (i.e. during system analysis), the Layout Expert combines the system
        configuration with the Pre-AST to produced the Preprocessed C code. In essence,
        the headers `#ifdef` directives are removed, and the different options are
        combined to produce a final C file, free from preprocessing macros. In this
        phase, the Layout Expert acts as a C pre-processor. The result is a huge C file
        with all the code in all the headers stuck together.
        
        Next, the Layout Expert applies trimming to this file. This is essentially a
        quick once over pass to identify only structs, unions, enums and typedef
        instructions. This optimization step means that we have much less code to parse
        in the next step and that the code that we do need to parse is more consistent
        and so easier to parse.
        
        Finally the Layout Expert parses the structs that Rekall is actually interested
        in (i.e. those structs with plugins that look at them). This parsing phase
        emulates a C compiler. We then apply the GCC struct layout model to the parsed
        structs in order to predict the precise memory layout of all fields in the
        structs (considering attributes, e.g. ``__attribute__((packed))`,
        `__attribute__((aligned(8))`).
        
        The last step is to generate a Rekall profile ready for use.
        
        
        ## Preparation.
        
        Before the profile generation can occur in the field, we need to build the
        `Pre-AST` file for the specific kernel version. This is easy since it does not
        require any specific configuration file or special tools (Remember that the
        Pre-AST includes all branches of any `#ifdef` directives so we do not need to
        evaluate any macros at this stage.).
        
        You can use the kernel headers package for the specific kernel, or the full
        kernel source. There is no need to actually compile the source (i.e. `make
        depmod` etc). Note that the kernel header package does not include "private" or
        non exported structs, so these will be missing from the profile, but current
        Rekall does not need those.
        
        ```
        $ layout_tool build_pre_ast --source_file_path ~/rekall/tools/linux/module.c \
          --linux_repository_path /usr/src/linux-headers-4.2.0-22-generic/ pre_ast_4.2.0-22.json
        
        2016-01-23 10:38:00,493 INFO     LOADING AND PARSING HEADERS
        2016-01-23 10:38:58,912 INFO     Completed built pre-ast forest in 58 Seconds
        2016-01-23 10:38:58,913 INFO     LOADED AND PARSED
        ```
        
        ## Bugs and support
        
        The Layout Expert is brought to you by the same people who develop Rekall, but
        it is considered a separate project. It is available under an Apache license
        (Check the LICENSE file). However, there is no official support or warranty; not
        even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
        
        If you think you've found a bug, please report it at:
        
            https://github.com/google/rekall/issues
        
        You can also mail to the list rekall-discuss@googlegroups.com
        
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
