82 lines
2.5 KiB
Text
82 lines
2.5 KiB
Text
|
# Scraping Alpha
|
||
|
|
|
||
|
|
### Author
|
||
|
|
|
||
|
|
Ben Goldsworthy
|
||
|
|
<[email](mailto:b.goldsworthy@lancaster.ac.uk)>
|
||
|
|
<[website](http://www.bengoldsworthy.uk/)>
|
||
|
|
|
||
|
|
### Version
|
||
|
|
|
||
|
|
1.0
|
||
|
|
|
||
|
|
### Abstract
|
||
|
|
|
||
|
|
Scraping Alpha is a series of Python scripts used to scrape
|
||
|
|
[Seeking Alpha](http://seekingalpha.com/) earnings call transcripts and produce
|
||
|
|
SQL from them.
|
||
|
|
|
||
|
|
It was created for Dr Lars Hass of the Lancaster University Management School.
|
||
|
|
|
||
|
|
### Usage
|
||
|
|
|
||
|
|
The instructions for each step of the process can be found at the beginning of
|
||
|
|
each of the files involved: `transcript_spider.py`, `JSONtoSQL.py` and
|
||
|
|
`execsAndAnalysts.py`. The are repeated here for brevity.
|
||
|
|
|
||
|
|
#### `transcript_spider.py`
|
||
|
|
|
||
|
|
This file is the webspider that Scrapy uses to retrieve the information from the
|
||
|
|
website. Left unattended, it will scrape all 4,000+ pages of results.
|
||
|
|
To interrupt this behaviour and still be able to proceed with the other steps,
|
||
|
|
cancel the script with `CTRL+Z`. This will likely leave an unfinished JSON item
|
||
|
|
at the end of the output file. To clear this up, open the file in `vim` and type
|
||
|
|
the following keys:
|
||
|
|
```vim
|
||
|
|
G
|
||
|
|
V
|
||
|
|
d
|
||
|
|
$
|
||
|
|
i
|
||
|
|
BACKSPACE
|
||
|
|
ENTER
|
||
|
|
]
|
||
|
|
ESC
|
||
|
|
:wp
|
||
|
|
ENTER
|
||
|
|
```
|
||
|
|
|
||
|
|
This will truncate the file at the last complete record and seal it off.
|
||
|
|
|
||
|
|
For installation instructions for Scrapy, see
|
||
|
|
[here](https://doc.scrapy.org/en/latest/intro/install.html). This file should be
|
||
|
|
in the `spiders` directory of the project, and is run via `scrapy crawl
|
||
|
|
transcripts -o transcripts.json` at the command line (the output file will be
|
||
|
|
placed in the directory the Terminal is currently pointing to).
|
||
|
|
|
||
|
|
#### `JSONtoSQL.py`
|
||
|
|
|
||
|
|
This file takes the `transcripts.json` file output of `transcript_spider.py` and
|
||
|
|
converts it into SQL.
|
||
|
|
|
||
|
|
This file should be located in the same directory as `transcripts.json`, and is
|
||
|
|
run via `python JSONtoSQL.py > [FILE].sql`, where `[FILE]` is the desired name
|
||
|
|
of the output file.
|
||
|
|
|
||
|
|
#### `execsAndAnalysts.py`
|
||
|
|
|
||
|
|
First, import the output file of `JSONtoSQL.py` to your chosen DBMS (I've tested
|
||
|
|
it with phpMyAdmin). Then, run the following query:
|
||
|
|
```SQL
|
||
|
|
SELECT `id`, `execs`, `analysts` FROM `transcripts`
|
||
|
|
```
|
||
|
|
|
||
|
|
Export the resulting table ([instructions](http://serverfault.com/a/435443)) to
|
||
|
|
`transcripts.sql`, and place the file in the same directory as
|
||
|
|
`execsAndAnalysts.py`. Run it with 'python execsAndAnalysts'.
|
||
|
|
|
||
|
|
It creates from this two files (`execs.sql` and `analysts.sql`). Import them
|
||
|
|
into your DBMS to create two linking tables. The final instruction of
|
||
|
|
`analysts.sql` then deletes the superfluous `execs` and `analysts` columns from
|
||
|
|
the `transcripts` table.
|