MCA-227
Web Intelligence and Big Data
Finding things on the web and coping with its scale — information retrieval and search engine architecture, then NoSQL, Hadoop and MapReduce.
- Semester
- Semester 3
- Credits
- 3
- Scheme
- 3 lectures a week
- Contact hours
- 40 across four units
- Category
- Core Elective - III
What the paper is
Web Intelligence and Big Data joins two problems that turn out to be the same problem at different scales: how do you find the one document you want, and what do you do when there are too many documents to fit anywhere. The first half is information retrieval, the second is the infrastructure that makes retrieval possible at web scale.
Unit I is classical IR and the most mathematically precise part of the paper — the term-document matrix, the Boolean, vector space and probabilistic retrieval models, and precision and recall as the measures of whether any of it worked.
Units III and IV are the big data half: why relational databases stop being the answer, what the NoSQL categories are, and then Hadoop, HDFS and MapReduce with the surrounding ecosystem — HBase, Sqoop, Flume, Pig, Hive and Spark.
What to concentrate on
- Precision and recall, computed on a small result set and traded off against each other, is the most certain question in the paper. Know the F-measure too, and be able to say why optimising one alone is meaningless.
- The vector space model with TF-IDF weighting and cosine similarity is the other reliable computation. Work one example with three documents and two query terms all the way through.
- For MapReduce, be able to write the map and reduce steps for word count and for one variation — the exam almost always asks for a variation rather than the canonical example.
Syllabus, unit by unit
Summarised from the approved scheme. Hours are the contact hours the scheme allots, which is a fair guide to where the marks sit.
Unit I · Web intelligence and information retrieval
10 hours- Characteristics of the web and web structure, retrieval versus browsing, the long tail in social networks
- What web intelligence is, its benefits, ingredients, related technology and applications
- Information retrieval: document representation, stemming, the term-document matrix, standard document collections
- Retrieval models: Boolean, vector space and probabilistic
- Evaluation criteria: precision and recall, the confusion matrix
Unit II · Search engines and mining
10 hours- Architecture of a web search engine: the crawler, indexing systems, queries and ranking, scalability, ranking through link analysis
- Multimedia search: images, audio and video
- Text pre-processing: cleaning, integration, transformation, segmentation
- Image pre-processing: histogram analysis, noise cleaning, segmentation
- Classification algorithms: linear regression, decision tree, k-means, naive Bayes
Unit III · Big data and NoSQL
10 hours- What big data is and why it matters; applications, analytics and challenges
- Introduction to NoSQL: aggregate data models, key-value and document data models
- Graph databases and schema-less databases; big data solutions
- Case studies on big data
Unit IV · Hadoop and MapReduce
10 hours- Hadoop architecture and working, its advantages
- HDFS: overview, features and architecture
- MapReduce workflows and Hadoop streaming
- Components of the Hadoop ecosystem: HBase, Sqoop, Flume, Pig Latin scripts, Hive
- Apache Spark
How it is examined
- Question 1 is compulsory, covers the whole syllabus, and is made of objective or short-answer questions. It carries 12 marks.
- The rest of the paper is four units matching the four above. Each unit sets two questions and you attempt one, so there are four more answers at 12 marks each.
- Examiners are asked to map every question to a course outcome using Bloom's taxonomy, which is why questions are phrased as “apply”, “analyse” or “evaluate” rather than only “explain”.
The pass mark is 40 out of 100 in every paper.
Learning objectives
- Concepts of web intelligence and big data technologies
- Applications of web intelligence capabilities and big data analytics tools
Assumed beforehand
- Concepts of web technology
- Understanding of database management systems
Paired in the scheme with MCA-267 Lab based on Core Elective III.
Books it is taught from
- TB1Rajendra Akerkar and Pawan Lingras, “Building an Intelligent Web: Theory and Practice”, Jones & Bartlett Learning, 2010
- TB2“Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data”, Wiley
- TB3Nathan Marz and James Warren, “Big Data: Principles and Best Practices of Scalable Real-Time Data Systems”, Manning, 2015
- TB4Dirk deRoos et al., “Hadoop For Dummies”, John Wiley & Sons, 1st edition, 2014
Notes for MCA-227
Lecture notes, question papers and assignments filed against MCA-227 are in the library. Everything there has been checked by a moderator before it appears, and an account is free — new ones are approved by hand, which is what keeps the shelf clean.