A little assignment to practice importing and analyzing data within a MongoDB database.
The goals of this assignment are to:
An exam covering:
There two parts to this exam. Please follow the instructions for each:
Update the solution contents of this file according to the instructions.
The following sections contain a report on the solutions to each of the required components of this exam.
The code in the Python program, solution.py, contains the solutions to the data munging part of this exam.
The following table, representing students' grades in courses at a university, is already in first normal form (1NF) - all records have the same number of fields, and there is only one value per field.
This assignment helps prepare you for the midterm exam by making you (1) manipulate bits of variables, (2) understand how integer multiplication and floating point addition is performed, and (3) write some simple assembly code.
I have provided some of the code and step-by-step instructions in the comments.
Due Tuesday, October 10 at 11:55pm
This assignment gives you practice working with pointers, trees, and dynamic allocation. The program will be to implement the “Animal Guessing Game”. This game is a version of “20 Questions” where the only category is animals. You think of an animal and your program will try to guess the animal by asking you a series of “yes or no” questions.
A little assignment to practice finding data, munging it, and analyzing it in a spreadsheet program.
Replace the contents of this file with a report, as described in the instructions.
A little assignment to practice finding data, munging it, and analyzing it in a spreadsheet program.
Which of the following are spreadsheet programs?「A、B、C、D」
A. Apple Numbers
B. Google Sheets
C. LibreOffice Calc
D. Microsoft Excel
E. Visual Studio Code
Which spreadsheet program would be best for a very large data set with a million records?「B」
A. Google Sheets
B. Microsoft Excel
Do spreadsheet applications generally allow users to write custom functions to analyze or modify the data in the spreadsheet?「A」
A. Yes
B. No
Data in a spreadsheet generally has a fixed schema.「B」
在这次作业中，你将尝试使用一个真实的数据源：NASA 的地球表面温度的历史测量数据。为了分析这些数据，一些预备工作是必要的。特别是，你将：
团队可以选择几种不同的工作流程。强烈建议实时合作，通常比异步合作更有效。但是，在实时合作不可能的情况下，有一些特定的工作流程可以帮助异步团队工作。
Database Design
在这个作业中，你将：
下面详细说明了你应该如何进行这项作业的其他要求。
这个作业是以一个在GitHub.com的仓库的形式给你的，该网站用于分享代码。仓库就是项目的一个高级别的名字。
Department of Computer Science
New York University
| Date | Class | Topics | Notes | Helpful resources | Quiz due | Exercise due |
|---|---|---|---|---|---|---|
| 9/5 | 1 | Welcome | Course intro | Syllabus | ||
| 9/7 | 2 | Python | - Discord for Education - Python | Videos: - Python crash course - Set up Visual Studio Code for Python programming Python for Everyone (py4e): - Variables - Conditionals - Functions - Iterations - Strings - Lists - Dictionaries - Tuples | Intake | - Complete consent form - Join Discord - Create GitHub account |
| 9/12 | 3 | Plain text data formats | Plain text data formats | - Video: Plain-text data formats - py4e: - Files - Text files code examples | Python | - Set up Python - Contemporary issues exercise |
| 9/14 | 4 | Data munging | - Text files in Python - Example Python code | Accessing O’Reilly Safari e-books Big Data Handbook (bdh): - Does This Data Smell Funny? - Data Lurking in Plain Sight | Text files | |
| 9/19 | 5 | Workshop - raw data munging | Workshop - raw data munging | Data munging | ||
| 9/21 | 6 | Spreadsheets | Spreadsheets | |||
| 9/26 | 7 | Workshop - spreadsheets | Workshop - spreadsheets | Vicky Rampin’s slides | Workshop - raw data munging | |
| 9/28 | 8 | SQL CRUD | SQLite intro | Using SQLite: - What is SQLite? - Uses of SQLite - Building and Installing SQLite (only Precompiled Distributions and An sqlite3 Primer) - The SQL Language - The SELECT Command | Spreadsheets | |
| 10/3 | 9 | SQL joins | - SQL joins - SQL inner joins - SQL left, right, and reflexive joins | SQL joins example code | SQL CRUD | Workshop - spreadsheets |
| 10/5 | 10 | Workshop - SQL | Workshop - SQL | SQL joins | ||
| 10/10 | n/a | no class (Legislative Day) | ||||
| 10/12 | 11 | Normalization | Normalization | - A Simple Guide to Five Normal Forms in Relational Database Theory - Database Design, Chapter 12 | ||
| 10/17 | 12 | Entity-relationship diagrams | Entity-Relationship Diagrams | Relational Database Design and Implementation: - Chapter 3 - Why Good Design Matters - Chapter 4 - Entities and Relationships - Chapter 5 - The Relational Data Model | Normalization | Workshop - SQL |
| 10/19 | n/a | no class (Prof. out) | ||||
| 10/24 | 13 | Workshop - database design | Workshop - database design | Entity-Relationship Diagrams | ||
| 10/26 | 14 | Review | Exam 1 review | Exam Practice: - Quiz - GitHub Repository | ||
| 10/31 | 15 | Exam 1 | Workshop - database design | |||
| 11/2 | 16 | MongoDB CRUD | - MongoDB setup - MongoDB CRUD | Video: MongoDB setup NYU CIMS documentation: - MongoDB Database Management on CIMS - Accessing MongoDB on CIMS MongoDB documentation: - MongoDB CRUD Operations - MongoDB db.collection.find() | ||
| 11/7 | 17 | MongoDB CRUD | MongoDB aggregation | MongoDB documentation: - MongoDB Data Model Design - MongoDB Data Model Examples and Patterns | ||
| 11/9 | 18 | Workshop - MongoDB | Workshop - MongoDB | MongoDB CRUD | ||
| 11/14 | 19 | Web apps | - The Web - Intro to HTML | |||
| 11/16 | 20 | Web apps | Pymongo & flask | Flask & pymongo documentation: - Flask Quickstart - Pymongo tutorial | Workshop - MongoDB | |
| 11/21 | 21 | Workshop - web apps | Workshop - Web app | Web apps | ||
| 11/23 | n/a | no class (Thanksgiving) | ||||
| 11/28 | 22 | Jupyter notebooks | Jupyter Notebooks | |||
| 11/30 | 23 | Pandas | - Pandas | - 10 minutes to Pandas - Example Jupyter notebooks | ||
| 12/5 | 24 | Data visualization | Workshop - web app | |||
| 12/7 | 25 | Workshop - pandas | Workshop - pandas | Pandas | ||
| 12/12 | 26 | Review | Exam 2 review | Workshop - pandas (No late work accepted beyond this date.) | ||
| 12/14 | n/a | no class (Prof out) | ||||
| TBD | 27 | Exam 2 |
For our last macro assignment you will be building a custom Node.js website using the express framework. The actual content and design for this site are very flexible, so be creative! Your site should incorporate the following technical features:
对于我们的最后一项宏观任务，您将使用
express框架构建一个自定义的Node.js网站。实际的内容和设计非常灵活，因此要有创意！您的网站应包含以下技术特性：
"messages.db" (database)
CREATE TABLE chats (id integer primary key autoincrement, nickname text, message text);
"index.html"
<!DOCTYPE html>
<html>
<head>
<title>Let's Chat!</title>
<style>
#previous_messages {
width: 100%;
height: 300px;
resize: none;
}
.hidden {
display: none;
}
</style>
<script src="helpers.js"></script>
</head>
<body>
<h1>Let's Chat!</h1>
<div id="panel_nickname">
<input type="text" id="nickname">
<button id="button_savenickname">Save Nickname & Chat</button>
</div>
<div id="panel_chat" class="hidden">
<textarea id="previous_messages" readonly></textarea>
<input type="text" id="message">
<button id="button_sendmessage">Send Message</button>
</div>
<script>
// global variables
let userNickname;
// figure out when the user saves their nickname
document.querySelector('#button_savenickname').onclick = function(e) {
// store the nickname for future use
userNickname = document.querySelector('#nickname').value;
// hide the nickname panel
document.querySelector('#panel_nickname').classList.add('hidden');
// show the chat panel
document.querySelector('#panel_chat').classList.remove('hidden');
}
// when the user types in a new chat message
document.querySelector('#button_sendmessage').onclick = function(e) {
// contact the server with our message AND our nickname
performFetch({
url: 'api.php?command=save',
method: 'post',
data: {
nickname: userNickname,
message: document.querySelector('#message').value
},
success: function(data) {
console.log("SUCCESS");
console.log(data);
if (data != "MISSINGDATA") {
document.querySelector('#previous_messages').value += data + "\n";
}
},
error: function(error) {
console.log("ERROR");
}
})
}
function getAllMessages() {
performFetch({
url: 'api.php',
method: 'get',
data: {
command: 'get_all_messages'
},
success: function(data) {
console.log(data);
// take what the server gave us and turn it into a JS object
data = JSON.parse( data );
console.log(data);
document.querySelector('#previous_messages').value = '';
for (let i = 0; i < data.length; i++) {
document.querySelector('#previous_messages').value += data[i] + "\n";
}
},
error: function(error) {
console.log(error);
}
})
}
setInterval(
getAllMessages,
3000
);
</script>
</body>
</html>
"helpers.js"
function performFetch(args) {
/* args is an object that is formatted as follows:
{
// the URL to contact on the server
url: url_to_contact
// request method ('get' or 'post')
method: 'get',
// object of variables to send to the server
data: {
var1: value1,
var2: value2,
var3: value3 // ... etc
},
// function to run if request succeeds, should accept a single argument which is the data returned by the server
success: function(data),
// function to run if request fails, should accept a single argument which is the error message
error: function(error)
}
*/
// GET requests
if (args.method && args.method.toLowerCase() == 'get') {
// package up the data to send to the server
const params = new URLSearchParams();
for (const varName in args.data) {
params.append(varName, args.data[varName]);
}
// append variables to URL
args.url += '?' + params.toString();
// perform the fetch request
fetch(args.url)
.then(function(response) {
if (response.ok) {
return response.text();
}
else {
let error = new Error("server error");
throw error;
}
})
// call the provided success callback function
.then(function(text) {
args.success(text);
})
// call the provided error callback function
.catch(function(error) {
args.error(error);
});
} // end GET request
// POST requests
else if (args.method && args.method.toLowerCase() == 'post') {
// package up the data to send to the server
// note that this is designed specifically to contact a PHP script
// we will use a slightly different approach when we contact
// node.js scripts in the next unit
const formData = new FormData();
for (const key in args.data) {
if (args.data.hasOwnProperty(key)) {
formData.append(key, args.data[key]);
}
}
// perform the fetch request
fetch(args.url, {
method: "POST",
body: formData,
})
.then(function(response) {
if (response.ok) {
return response.text();
}
else {
let error = new Error("server error");
throw error;
}
})
// call the provided success callback function
.then(function(text) {
args.success(text);
})
// call the provided error callback function
.catch(function(error) {
args.error(error);
});
} // end POST request
}
"api.php"
<?php
// get command
$command = $_GET['command'];
// command: save
// inputs: a nickname and a message
// outputs: a copy of the message, or 'error'
if ($command == 'save') {
$nickname = $_POST['nickname'];
$message = $_POST['message'];
// basic validation
if ($nickname && $message) {
// save this message into our database
$db = new SQLite3( getcwd() . '/database/messages.db' );
$sql = "INSERT INTO messages (nickname, message) VALUES (:nick, :msg)";
$statement = $db->prepare($sql);
$statement->bindParam(':nick', $nickname);
$statement->bindParam(':msg', $message);
$statement->execute();
print $nickname . ": " . $message;
}
else {
print "MISSINGDATA";
}
$db->close();
unset($db);
}
// command: get_all_messages
// inputs: none
// output: all previous messages, formatted as a JSON array
if ($command == 'get_all_messages') {
$db = new SQLite3( getcwd() . '/database/messages.db' );
$sql = "SELECT nickname, message FROM messages";
$statement = $db->prepare($sql);
$result = $statement->execute();
$return_array = array();
while ($temp_array = $result->fetchArray()) {
array_push( $return_array, $temp_array['nickname'] . ": " . $temp_array['message']);
}
$db->close();
unset($db);
print json_encode($return_array);
}
?>Note: The majority of this assignment uses only HTML, CSS and PHP - very little (if any) JavaScript is necessary to solve this problem. I've outlined the technologies needed for each feature below.
Micro assignment 06 focuses on the following concepts:
This micro assignment contains a series of videos along with a programming challenge, which you should attempt after watching the videos. The challenge is set up with a "Mark" button that will let you immediately see if you solved it correctly. You will get credit for the micro assignment by attempting and solving this challenge, and you are allowed to attempt the challenge as many times as you'd like.
Micro assignment 06 focuses on the following concepts:
微任务06主要关注以下概念：
This micro assignment contains a series of videos along with a programming challenge, which you should attempt after watching the videos. The challenge is set up with a "Mark" button that will let you immediately see if you solved it correctly. You will get credit for the micro assignment by attempting and solving this challenge, and you are allowed to attempt the challenge as many times as you'd like.