https://github.com/uconnstamford/datastore
#program1.py:
this program loops into a csv file that has column format. As you initialize each column in your csv file, the program will then use it as entities in datastore. Then, after the program runs, each line of text in your csv will be put into datastore, separated by entities, and will be given a key value.
https://docs.python.org/3/library/csv.html
1) create entities in datastore with a kind-name
2) initialize your kind-name
3) open your csv file instead of the example
4) initialize the rows/entities in order, with the rows of your csv file
5) check datastore to see the data
---------------------
#program2.py
this program will also loop into a csv file, but does not need column format.However, there is a size cap. This program is meant to send data into datastore in small amounts, with one entity which is 'paragraph'.
1) create entity in datastore under a new kind-name titled 'paragraph'
2) initialize our kind-name
2) open your csv file instead of example
3) check datastore to see the data
from google.cloud import datastore
import csv
from datetime import datetime
client = datastore.Client()
kind = 'twitter-data'
with open('WSJ_Apple.csv', 'r') as csv_file:
csv_reader = csv.reader(csv_file)
next(csv_reader)
#loop over the rows
for row in csv_reader:
created_at = datetime.strptime(row[1], '%Y-%m-%d %H:%M:%S%z')
text = row[2]
username = row[3]
retweet_count = int(row[4])
like_count = int(row[5])
reply_count = int(row[6])
entity = datastore.Entity(client.key(kind))
entity.update({
"created_at":created_at,
"text": text,
"username": username,
'retweet_count': retweet_count,
"like_count": like_count,
"reply_count": reply_count
})
client.put(entity)
from google.cloud import datastore
import csv
from datetime import datetime
client = datastore.Client()
kind = 'fec-data'
with open('sec_data.csv', 'r') as csv_file:
csv_reader = csv.reader(csv_file)
# loop over the rows
for row in csv_reader:
for paragraph in row:
entity = datastore.Entity(key=client.key(kind))
entity.update({
'text': paragraph
})
client.put(entity)
,Date/Time,Text,Username,Retweet_Count,Like_Count,Reply_Count
0,2019-09-11 11:25:50+00:00,Adding to the post: The Wall Street Journal has a commercial agreement to supply news through Apple News.,WSJ,6,29,6
1,2012-03-29 20:24:39+00:00,"BREAKING: Outside audit of Apple's supply chain found excessive working hours, health & safety issues at its largest manufacturer.",WSJ,174,13,7
2,2011-08-24 22:41:50+00:00,BREAKING: Steve Jobs Resigns as CEO of Apple,WSJ,1285,17,21
3,2010-10-06 19:47:43+00:00,"@LuluADH Well, it's people who have been briefed by Apple about the plans.",WSJ,0,0,0
4,2010-08-27 18:36:47+00:00,"Companies named in Paul Allen's lawsuit: AOL, Apple, eBay, Facebook, Google, Netflix, Office Depot, OfficeMax, Staples, Yahoo and YouTube.",WSJ,39,7,0
5,2010-04-20 20:40:55+00:00,"Earnings: Apple beats expectations by a wide margin, netting $3.07 billion last quarter on $13.5 billion in revenue.",WSJ,41,5,0
6,2010-04-14 12:40:09+00:00,"Apple will postpone the international launch of the iPad by one month, until the end of May, citing heavy U.S. demand.",WSJ,67,4,0
7,2009-10-19 21:00:44+00:00,"BREAKING NEWS: Apple fourth-quarter profit jumped 47%, to $1.67 billion, as the company sold more Macintosh computers and i..",WSJ,0,5,0
8,2009-10-19 20:31:27+00:00,BREAKING NEWS: Apple reported net quarterly profit of $1.67 billion.,WSJ,0,1,0
9,2009-07-21 20:52:50+00:00,BREAKING NEWS: Apple has third-quarter profit of $1.23 billion on $8.34 billion in revenue. Yahoo's second-quarter net rise..,WSJ,0,2,0
10,2009-04-15 21:57:59+00:00,"The Journal now has an #iPhone #app. iPhone users can download it from the Apple App Store by searching for ""WSJ"".",WSJ,0,9,0
admin@instance-2:/home/public/datastore$
No comments:
Post a Comment