Entropy, relative entropy and mutual information, asymptotic equipartition property, entropy rate of a stochastic process, data compression, gambling, channel capacity, differential entropy, Gaussian channel, rate distortion theory, information theory and statistics, maximum entropy, universal source coding, Kolmogorov complexity, network information theory, application of information theory on portfolio theory, inequalities in information theory.
- Teacher: Μιλτιάδης Αναγνώστου
Description : The course covers key concepts and areas in information theory, including entropy, data compression, channel capacity, and their applications in various fields like gambling, statistics, and portfolio theory. It also addresses topics such as Gaussian channels, rate distortion theory, Kolmogorov complexity, and network information theory.
Credits/ECTS : 4
Instructors : Miltiades E. Anagnostou
Bibliography :
Cover, Thomas M.., Thomas, Joy A.. Elements of Information Theory. Wiley, 2012.
Course Outcome :
- Knowledge of basic concepts of information theory and in particular the entropy of a random phenomenon, the mutual information between random variables, etc.
- Ability in performing small and large-scale numerical calculations with appropriate mathematical tools.
- Knowledge of fundamental properties of communication and how they are related to the properties of the source of information and the channel.
- Basic understanding of how the capacity of a channel is connected on the one hand with the mutual information, on the other hand with its functional characteristics and mainly with the transmission errors.
- Understanding of the way a transmission through a channel can be optimized.
- Understanding of the effect of noise on the capacity of a channel.,
- Understanding of data compression.
- Understanding of the gambling systems, and how they can be analyzed by using information theory concepts.
- Basic understanding of the usage of information theory in portfolio theory.
- Basic understanding of alternative considerations of entropy, such as Kolmogorov complexity.
Program it belongs to : MS level course of the integrated Master Program of ECE School NTUA
Course Type: Compulsory or Selective : Selective
Course Type: Core or Specialization : Specialization
StudyLoad : 3 theory; 9 homestudy
Evaluation method : Written exam 100%, homework exercises bonus 15%
Distribution of teaching and learning materials : Lecture Hours: 39, Homework hours: 16, Study: 65
Type : Course
Audience : students only
Parts Or Units :
UNIT 1: Unit 1 Title
- Presentation link
- Book link
- Exercise link
Language : el