Cythereal MAGIC User Documentation¶
Cythereal MAGIC extracts intelligence from malicious programs (malware) to aid in cyber defense and incidence response. It combines two orthogonal areas – semantic inference and statistical inference – along with big data analytics to create what may be termed as “Google for malware”. Using a patent-pending technology, Cythereal MAGIC can make connections between malware, peering through its protection mechanisms to discover intrinsic relationships that provide strong evidence for classification and attribution.
This documenation provides instructions for accessing the Cythereal MAGIC API and a short tutorial on some of the features provided by this API.
For documentation of all API capabilities see API Reference
Accessing API¶
Request API Key¶
Access to the MAGIC service requires an API key.
To request a key, send an email to support@cythereal.com with the subject “Key Request”. Include your name and email address you want the key to be registered under.
Please note that as part of our minimal verification of identities, we do not provide keys to email addresses pointing to open email services, such as, gmail.com, yahoo.com, aol.com, and many others. Please request a key using your business or university email address.
Client Libraries¶
We provide client libraries in several different languages that can be found at:
Documentation of how to use these libraries can be found at the link given above.
CLI Uploader¶
We additionally provide a stand alone command line utility for uploading files to the Cythereal MAGIC API.
Installation¶
Recommended installation method:
pip install cythereal-magic
This will create the cythereal-magic command.
If you do not want to install via pip, you can alternatively
download a stand alone executable from the table below.
| Version | Download URL |
|---|---|
| Linux | https://bitbucket.org/cythereal/magic-clients/downloads/cythereal-magic |
| Windows | https://bitbucket.org/cythereal/magic-clients/downloads/cythereal-magic.exe |
Usage¶
cythereal-magic [--key=MAGIC_API_KEY] upload [--recursive] UPLOAD_PATH
If the --key option is not provided, the environment variable
MAGIC_API_KEY will be checked for the value of the key. If this
variable is missing or empty, the key will be interactively asked
for.
Note that the --key option MUST come before the upload keyword.
The UPLOAD_PATH can be either a directory or a single file. If
UPLOAD_PATH is a directory, all files in the directory will be uploaded.
By default, directories will not be recursively traversed. This can be
changed by providing the --recursive option.
API Endpoints¶
The MAGIC API can always be directly accessed (using curl or a browser, for example) at:
The endpoints are documented at api-reference
Tutorial¶
This tutorial helps you get started using MAGIC through a walk through of several capabilities MAGIC provides.
If you haven’t yet, first go through Accessing API.
In this tutorial, we will assume that the following variables are available:
| Variable | Description |
|---|---|
| MAGIC_API_KEY | The API key used to access the MAGIC API. |
| BINARY_HASH | The SHA1 of a binary. 2d9035b27ce5c1cce2fb8432c76e315a1a2a8de0 in this tutorial |
Upload file for analysis¶
To start, let’s upload our binary to MAGIC. In addition to individual binaries, archives, such as zip or tar, are also supported.
curl -X POST \
-F "filedata=@./sample.exe" \
-F "filename=sample.exe" \
"https://api.magic.cythereal.com/v1/files/?key=${MAGIC_API_KEY}"
You can also use the cythereal-magic command line utility:
cythereal-magic --key=${MAGIC_API_KEY} upload ./sample.exe
Check status of Analysis¶
In order to check the status of our uploaded file, we need to know the sha1sum of this file. On linux, this can be found using the sha1sum command.
To get the result of analysis:
curl "https://api.magic.cythereal.com/v1/files/${BINARY_HASH}/info/?key=${MAGIC_API_KEY}"
This results in the following (See magic-api-format for description of communication protocol.):
{
"api_version": "1.0.0"
"success": true,
"code": 200,
"data": {
"sha1": "2d9035b27ce5c1cce2fb8432c76e315a1a2a8de0",
"sha256": "338370573c7f275161f7d08cd9a309bcf41c078bf44d4fd50be997e7b83fc856",
"sha512": "7a73805bdb8a6cf15d7b4c07b25edd17fe9fb3af7aa514497c47bdaaafa3526b929c757d26ca859a432c3295ed2ea352479f5b175eb6f72ef57c8c83d45f7f0d",
"md5": "a7fd6503d8bc6d0e61b779d968568067"
"unix_filetype": "PE32 executable (GUI) Intel 80386, for MS Windows",
"filepath": [
"./archive",
],
"parents": [
"fb74c17fad2458f6d8f1f0d3f4dfa7b9ce22a2ec",
],
"uploadDate": "2017-01-10 06:03:52.720000",
"object_class": "binary.pe32",
"children": [
{
"service_name": "srlUnpacker",
"status": "success",
"child": "9bbc14d35aaa53283734826ef54a4e272ae51f01"
},
{
"service_name": "srlJuice",
"status": "sucess",
},
{
"service_name": "srlSimilarity",
"status": "success",
}
],
},
}
The status of the various analyses provided by MAGIC are present in the
children field. A pending analysis will not have any entry in this field, a
failed analysis will have its status field set to “failure” or “failed”,
and a successful analysis will have its status set to success.
This file has been successfully unpacked, the genomic information has been successfully extracted, and the magic report correlations are ready. The unpacked version of this file has a SHA1 of 6efb5c849b400af96b993284f6d39746b6bdb2e6. To get the status of the unpacked file, an additional query would have to be made.
Download unpacked file¶
Note
Downloading files from MAGIC requires additional permissions. Please contact us for these permissions.
To download the unpacked version of a binary using curl, we need to know the SHA1 of the unpacked version. This hash can be located using the status query described last section. For our example of 2d9035b27ce5c1cce2fb8432c76e315a1a2a8de0, the unpacked version has a SHA1 of 9bbc14d35aaa53283734826ef54a4e272ae51f01.
curl "https://api.magic.cythereal.com/v1/files/9bbc14d35aaa53283734826ef54a4e272ae51f01/?key=${MAGIC_API_KEY}" > 9bbc14d35aaa53283734826ef54a4e272ae51f01
This will download the unpacked file to a file named 6efb5c849b400af96b993284f6d39746b6bdb2e6.
Get MAGIC Matches¶
To find binaries that “match”, i.e. are similar to our example binary, we retrieve the MAGIC correlations.
curl "https://api.magic.cythereal.com/v1/reports/${BINARY_HASH}/matches/?key={$MAGIC_API_KEY}"
See the Swagger UI for a description of the returned JSON document.
View Genomic Features¶
To get the genomic features for the entire binary:
curl "https://api.magic.cythereal.com/v1/genomics/${BINARY_HASH}/?key=${MAGIC_API_KEY}"
See the Swagger UI for a description of the returned JSON document.
To filter the output to the features in a single procedure, provide the RVA of the procedure in the path. For example, to get genomics for only the procedure with RVA 0x1000:
curl "https://api.magic.cythereal.com/v1/genomics/${BINARY_HASH}/0x1000/?key=${MAGIC_API_KEY}"
Currently, the only way to get the list of procedures in a binary is by retrieving the genomic features for the entire binary. This will be addressed in a future API version.
Search for similar procedures¶
curl "https://api.magic.cythereal.com/v1/similarities/${BINARY_HASH}/0x1000/?key=${MAGIC_API_KEY}"
See the Swagger UI for a description of the returned JSON document.
Use Cases¶
Malware Intelligence¶
Cythereal MAGIC’s unique analysis method combines deep knowledge of Operating Systems Internals coupled with state-of-the-art programming languages theory for formal program analysis. This allows it to peer through most known obfuscations and easily analyze even the most complex malware and extract a wealth of information about the inner structure and workings of malware. Add Data Mining to mix and you get a very powerful tool to extract Intelligence from large repositories of malware at a scale that was previously un-thought of.
MAGIC can be used to find connections among malware families that were previously never even thought of. Further queries can be made to the system to find out the nature of the connection and also to show the evidence- semantically equivalent procedures that led the system to the conclude the connection.
Below image shows MAGIC identifying a connection between Gamarue Worms and Leechole Trojans. MAGIC found that certain variants of the two families share the same packer. MAGIC also successfully identified the set of procedures that were common to the two families and formed the unpacking stub. This is of immense help to reverse engineers wanting to unpack the malware manually for deeper analysis.
Below two images show two procedures found in several variants of DarkComet and Optima families. Variants of both families use different packers to hide these procedures from static analysis. The procedures were extracted by MAGIC’s unpacker using VM Introspection at runtime.
Malware Signature Generation¶
MAGIC can analyze large collection of labeled malware and generate semantic signatures common to the family. MAGIC analyses are capable of locating and identifying even the smallest set of procedures common to a family and generate obfuscation resistant, semantically meaningful signatures.
Additionally, MAGIC can also perform probabilistic analysis to calculate a confidence value with which it assigns a new malware variant to a known family.
Below graph show number of procedures (y-axis) vs percentage of nitol binaries they are found in (x-axis) as identified by MAGIC. The graph shows that MAGIC is capable of finding the needle in haystack! It successfully generated juice based signatures for the set of 5 procedures that were present in more than 95% of nitol executables.
Reverse Engineering¶
MAGIC uses VM introspection to observe malware execution at a level below ring 0. The intricate knowledge of Windows Internals is in-built the system to monitor the malware’s interaction with the Operating System as it is executing. This is followed by a rigorous static analysis of the original code, as well as, that of runtime generated code extracted during the execution.
MAGIC’s static analysis engine performs a variety of analyses. The most important to reverse engineers being the BinJuice analysis. Juice is an abstraction over semantics that can be computed and compared in a fast and scalable fashion.
Given a binary executable, in about a minute, MAGIC can calculate juice of all procedures in the binary and find out known procedures in the database which are semantically equivalent to procedures in the given binary. Users then have access to all the information and notes of malware analysts who have worked on the procedure before, leaving only the unique, never-seen-before, procedures to be reversed. This reduces the workload by orders of magnitude.
Propagating information from procedure to another juice equivalent procedure has interesting advantages. For instance, IDA more often than not, misses to identify library procedures. Reverse engineers thus often end up spending time reversing a library procedure which can be avoided.
Below image shows percentage of library procedures as identified by IDA followed by those identified by MAGIC by just propagating IDA isLibrary tag information across juice-equivalent procedures.
The above is just a glimpse of what can be achieved by propagating information across equivalent procedures. One can also throw in labeled open source code and propagate information from them to similar equivalent procedures in malware and use the labels to guide a reverse engineer trying to understand the malware behaviors.
To better aid in understanding new malware, MAGIC also reports on the ControlFlow Graph of the malware. Additionally, MAGIC also generates an APIFlow Graph. Since API calls are the most common way to interact with the OS, they can be used to understand malware behavior. APIFlow Graph thus may be understood as an abstraction of the ControlFlow Graph where each path describes the behavior of the program as it executed that path on the ControlFlow Graph.
Legalese¶
Privacy and Access Control¶
Though access to MAGIC is restricted using an API Key, anyone with a valid API Key may have access to all the uploaded files, unpacked binaries, and their analyses. This includes even the files that are not uploaded by the user.
Which means any data you upload and all the analyses we perform is available to everyone who has access to MAGIC. Please be aware of this before uploading any data.
More fine-grained access control is planned for future versions.
Disclaimers¶
The service is provided as is, without any claims of fitness for use. We offer no guaranties or warranties. There is also no assurance for any level of quality of service.
IP Rights¶
Access to use this service does not afford a user any rights on the SDK, the analyses results computed by the service, the underlying algorithms, or the implementation.
Contact Cythereal¶
Contact via Email: support@cythereal.com