Generate Dummy Data
Generate Dummy Data
In this step, we will configure the Kinesis Data Generator to generate dummy data and feed it to Kinesis Firehose.
Configure Amazon Cognito for Kinesis Data Generator - In this step, we will launch a stack cloud formation to configure Cognito. These cloudformation script files will be launched in the N.Virginia region (No need to change this region).
-
Go to CloudFormation or Github
- Click Next
- Specify the details:
- Username: admin
- Password: choose a password consisting of letters and numbers
- Click on Next
- Options:
- Option to add Tags (labels) if desired, for example: workshop: AnalyticsOnAWS
- Click on Next
- I acknowledge that AWS CloudFormation might create IAM resources: choose
- Click Create stack
- Refresh your AWS CloudFormation console
- Wait until the status of the stack changes to Create_Complete
- Select Kinesis-Data-Generator-Cognito-User stack Go to the outputs tab: Click on the link called: KinesisDataGeneratorUrl - This will open your Kinesis Data Generator tool.
- On the Amazon Kinesis Data Generator homepage
Log in with the username and password from the previous step Region: us-east-1 Stream/delivery stream: analytics-workshop-stream Records per second: 2000 Record Template (Sample 1): In the large text area, insert the following json template:
{
"uuid": "{{random.uuid}}",
"device_ts": "{{date.utc("YYYY-MM-DD HH:mm:ss.SSS")}}",
"device_id": {{random.number(50)}},
"device_temp": {{random.weightedArrayElement(
{"weights":[0.30, 0.30, 0.20, 0.20],"data":[32, 34, 28, 40]}
)}},
"track_id": {{random.number(30)}},
"activity_type": {{random.weightedArrayElement(
{
"weights": [0.1, 0.2, 0.2, 0.3, 0.2],
"data": ["\"Running\"", "\"Working\"", "\"Walking\"", "\"Traveling\"", "\"Sitting\""]
}
)}}
}
- Click - Send Data. Once the tool has sent about ~10,000 messages, you can Select - Stop sending data to Kinesis.
- Verify that data has been sent to S3
- After some time, go to the S3 . console
- Go to: yourname-analytics-workshop-bucket > data
- There should be a folder called “raw” > Open it and keep navigating, you will notice that firehose dumped the data into S3 using the partition in the format yyyy/mm/dd/hh
- If you have received dummy data in your S3 buckets, we can continue to the Next step!
