Memex Explorer¶

Registering a Crawl¶

To register a new crawl, click the “Add Crawl” button above the Crawls table. This will open a popup for adding crawls. If necessary, you can also create seeds list objects and crawl models from the same form.

For both crawls, you have to supply a seeds list object, which contains the list of urls to be crawled. The seeds list object can be created from the Add Crawl form.

For ACHE crawls, you have to specify a crawl model for the crawl, which can also be added from the Add Crawl form.

Registering a Crawl Model¶

ACHE crawls require a Crawl Model to power the page classifier. The model consists of two elements: a “model” file and “features” file. These can be generated by following the instructions on the ACHE GitHub page.

To register a new crawl model, click on the “Add Crawl Model” button in the Crawl Models header. This will bring up the crawl model creation popup. Models can also be added from the Add Crawl form by selecting “ache” as a crawler.

Uploading Files and Dataset Creation¶

With Memex Explorer you can create indices by uploading zipfiles of important documents. Memex Explorer will analyze these documents with Tika. You can then easily access the documents from the local Elasticsearch index, and incorporate them into other data analysis tools. You can create the dataset by clicking “Add Dataset” on the project page.

The add dataset page has a progress bar, and when your dataset has been successfully uploaded, you will get a success message and an alert to close the page. If you attempt to close the page before the files have been successfully uploaded, you will get an alert warning you to wait until the page is done uploading.

Registering a Seeds List¶

Each crawl requires a seeds list object. Ache requires the seed list in a textfile, whereas Nutch requires a seeds list injection. The seeds list object handles both of these requirements. It creates a file for Ache and contains fields for injecting seeds through the Nutch REST Api. All seeds objects can be added on the “Add Crawl” popup. This is the seeds list form.

Seeds require a valid name, and either a file or URLs placed in the textarea below. If any of your seeds are invalid, you will get a form error, and all the invalid urls will be highlighted.

Creating a Seeds List from a DataWake Trail¶

If you are using DataWake, and Memex Explorer has access to the index used by DataWake, you will be able to create seeds lists from DataWake trails. To create a seeds list, all that is required is a valid name. After you create the seeds list, you can edit it just like any other seeds list.

Editing a Seeds List¶

Once you have created your seeds list, you can edit through our built in editor. This editor allow you to change the content of your seeds list, by adding or removing seeds. It will also validate all of the URLs and display the ones which contain errors.

Creating a Seeds List¶

The common point between the two crawlers is that they both use the same kind of seeds list for their crawling. The seeds list is comprised of a list of urls separated by line breaks. Both Nutch and Ache use them in different ways, and the result you get directly from the crawlers is different for each of them. Here is a sample seeds list:

http://www.reddit.com/r/aww
http://gizmodo.com/of-course-japan-has-an-island-where-cats-outnumber-peop-1695365964
http://en.wikipedia.org/wiki/Cat
http://www.catchannel.com/
http://mashable.com/category/cats/
http://www.huffingtonpost.com/news/cats/
http://www.lolcats.com/

Simply put, the seeds list should contain pages that are relevant to the topics you are searching. Both Nutch and Ache provide insight into the relevance of your seeds list, but in different ways.

For the purposes of memex-explorer, the extension and name of your seeds list does not matter. It will be automatically renamed and stored according to the specifications of the crawler.

Seeds lists are created on the seeds page, and seeds lists can be created from the add crawl page.

Crawler Control Buttons¶

Here we have an overview of the buttons available to each crawler for controlling the crawlers. The buttons behave differently depending on which one you are using.

These are the buttons available for Ache:

These are the buttons available for Nutch:

Crawl Settings¶

The crawl settings page allows you to delete the crawl, as well as change the name or description of the crawl. It is accessed by clicking the “pencil” icon next to the name of the crawl.

Here you can change the name or description of the crawl. You can also delete the crawl.

Nutch Dashboard¶

Memex explorer recently added features for monitoring the status of Nutch crawls. You can now get real-time information about which pages Nutch is currently crawling, and information about the duration of the crawl.

Statistics¶

Nutch will tell you how many pages have been crawled after the current round has finished.

Ache Dashboard¶

Application Setup¶

To set up a developer’s environment, clone the repository, then run the app_setup.sh script:

$ git clone https://github.com/memex-explorer/memex-explorer.git
$ cd memex-explorer/source
$ ./app_setup.sh

You can then start the application from this directory:

$ source activate memex
$ supervisord

Memex Explorer will now be running locally at http://localhost:8000.

Tests¶

To run the tests, return to the root directory and run:

$ py.test

The Database Model¶

The current entity relation diagram:

Enabling Non-Default Services¶

Project Creation¶

1. When you start up the application, you should see a landing page with a button for adding a new project.

   1. Click the new project button.
   2. Provide a name and a description for the project on the next page, and press submit.
   3. Verify that your new project shows up on the project page list.
   4. Click on the new project and go to the project page. Verify that there are no crawls, models, or datasets yet.

Project Settings¶

1. Click the “pencil” icon next to the name of the project on the project overview page.

   1. Supply a different name and description for the project, and hit “submit”.
   2. Verify that the project was edited successfully by checking the success message at the top of the page.

2. Go back to the settings page.

   1. Click on the “trashcan” icon. Verify that there is a popup asking you whether you want to delete the project.
   2. Click on the trash icon and click yes.
   3. Verify that you are taken to the landing page, and that there are no projects listed on the landing page.

Index Creation¶

1. Create a new project.

2. Click on the “Add Index” button either in the sidebar or under the list of indices on the project page.

   1. Add an index. Give the index a name and a zip file. There are two zipfiles in the repository to use, located at “source/resources/test_resources”. Click submit.
   2. Verify that the index was added successfully by checking for the success message at the top of the page.
   3. Verify that the index was successfully created by checking the status next to the name of the index.

   

Index Settings¶

1. Click on the link to the index on the project overview page. This will take you to the index settings page.

   

   1. Supply a new zipfile for the index creation. Use the zipfile that you did not use earlier – “sample2.zip” if you earlier used “sample.zip”.
   2. Verify that the index was updated successfully by checking the indices list.
   3. Verify that the new files were added to the newly created index.

2. Return to the index settings page and click the “trashcan” icon. As before, confirm that the cancel button works, and then delete the index.

   1. Confirm that the index was deleted successfully by looking at the list of indices on the project overview page.

Testing Nutch Crawls¶

Included with the repository is a test seeds file. You can use this file to testing of nutch and ache crawls. The seeds file is located at “source/test_resources/test_crawl_data/cats.seeds”.

1. From the project overview page, click the Add Crawl button on the list of crawls or in the sidebar dropdown.

2. At the add crawl page, supply a name and description.

   1. Make sure that the “nutch” option is selected.

   

   2. Select one of the previously created seed lists and create the crawl.

3. Verify that the crawl has been added successfully to the crawls list table.

4. Go to the crawl page by following the link in the crawls list table.

   1. Verify that the crawl status and available buttons are the same as in this image.
   
   2. The following buttons should be available: “Start Crawl”, “Get Seeds List”. All other buttons should be greyed-out.
   3. The crawl status should be set to NOT STARTED with 0 rounds left to crawl.

5. Start a crawl and verify that the crawl completes successfully.

   1. When you start the crawl, there should be two rounds left.
   2. At the end of the first round, summary statistics should list total pages crawled as between 6 and 9.
   3. After the first round is done, the status should show “SUCCESS” before going onto the next round.
   4. On the start of the next round, the crawl status should change to “STARTED”
   5. At the end of the second round, the rounds left should be zero.
   6. The pages crawled should be between 300 and 400.

Test Crawl Settings¶

1. On the crawl page, click the “gears” icon to access the settings.

   1. Change the name and description of the crawl, and submit.
   2. Click the “trashcan” icon to delete the crawl.
   3. Hit cancel on the popup first, and then delete the crawl.
   4. Verify that you are brought to the project overview page.