In this article, I will provide detailed instructions on how to create a project with Scrapy and use it to analyze and extract housing data from the alonhadat website. If your machine doesn’t have ...
Why don't website creators completely want to protect their websites from being crawled?
Websites nowadays are no longer as easy to extract data from as before because the structure of websites now is very different from the past - they don’t have clearly defined sections for quick ana...
Crawler, some things I want to share about crawlers and upcoming crawler article series?
Crawler, Web Scrape, Web Scraping, data collection, data scraping,… are probably the terms we use most to talk about work that creates programs to analyze and extract data from websites. There’s ce...
Project Socket Stream with Spark Streaming
In this post, we consider a small example with Spark Streaming. My work is creating a project with Spark Streaming listen in port 7777 and filter line contain “error” word and print it to console. ...
Summary of questions about Apache Hadoop
The main goal of Apache Hadoop Open data storage and powerful data processing. Save costs when storing and processing large amounts of data. You can see more details about Hadoop’s goals HERE Ha...
Hadoop MapReduce and basic WordCount program with MapReduce
MapReduce is a processing technique and a programming model for distributed computing to deploy and process big data. Hadoop MapReduce is a data processing framework of Hadoop built on the idea of ...
Docker Basics and Practice
This article is referenced and noted from the practical Docker series on Viblo. Series link here: Practical Docker from Basics I. Installing Docker and Some Basic Concepts Installation can be re...
Understanding Apache NiFi
Apache NiFi is used to automate and control data flows between systems. It provides us with a web-based interface that can collect, process, and analyze data. NiFi is known for its ability to buil...
Kafka In Depth
During the process of working on a major project for the Data Storage and Big Data Processing course at university, I learned about Kafka and used it in my project. However, at that time I only sim...
Commands for manipulating files and directories on HDFS
The commands on HDFS are generally quite similar to the commands on Linux, both in terms of their functions and names, if you are familiar with Linux/Ubuntu then you probably don’t need to learn mu...