Project

This repository is to outline, project manage, and issue track all the various tools TransparencyToolkit is creating.

Our tools are primarily made for investigative journalists and researching various open source intelligence sources. All of our tools are open source. We could love to collaborate and receiving contributions. We also conduct our own original research, mostly on the international surveillance industry.

Interested in collaborating or contributing contact us

LookingGlass

A highly configurable web application for viewing JSON datasets in a themeable faceted search interface.

Examples

• ICWATCH
• Snowden Doc Search

Technical

• Ruby, Rails, ElasticSearch, JS, SASS, Bootstrap
• Browse Code

---

Harvester

A rails application that provides a web interface to run all of our scrapping tools and integrates with LookingGlass

Technical

• Technologies: Ruby, Rails, neo4j, JS, SASS, Bourbon
• Browse Code

---

Components

• CrawlerManager - an API for running and managing crawlers
• KeepGrabbing - a collection of scripts and CLIs to manage and run scrappers and crawlers

---

Gems

• RequestManager
• DirCrawl
• ParseFile
• EmailParser
• LinkedInCrawler
• IndeedCrawler
• PiplRequest
• PiplCollector

---

Misc

• Twiddler - an experimental tool to help with processing, refining, and extracting meaning from text and file dumps
• ArchivePile - a TT inspired read-only theme for browsing email archives that runs on Mailpile