# Deduplicate

Some spiders may crawl duplicate data, such as *transaction subgraph spiders*. The built-in extractor of this project realizes the function of deduplicating transaction data. You can run on this command in the console:

```
python extract.py deduplicate\
-i /path/of/raw/data \
-o /path/of/output/data
```

Parameter Description:

* `-i`: the input directory of raw data.
* `-o`: the output directory of deduplicated data.
