ICLR 2025 oral notes P.S. The paper descriptions are based on my personal understanding. Some text were extracted from the abstracts and reviews. Domain LLM Transfusion ChuntingZhou2025ICLR combines next token prediction for text and diffusion based learning for images within a single transformer architecture, bridging the modality gap without quantizing images into discrete tokens Embedding AlexIacob2025ICLR introduces a pre training framework that decouples embedding layers from the transformer body, enabling robust training on heterogeneous data avoiding the curse of multi linguality , improving generalization, and reducing memory footprint ZiyueLi2025ICLR investigates the limitation of using decoder only models for embedding and finds that a good embedding can be acquired from the MoE layer, by combining rou...

How Computers Really Work Matthew Justic P.S. Part names are added by me. Some chapters are merged and renamed. Part I Basics Electrical circuits Term Explanation Measured in Water analogy Electric charge Causes matter to experience a force Coulombs Water Electric current The flow of charge Amps Flow of water through a pipe Voltage The difference in electric potential between two points Volts Water pressure Resistance A measure of the difficulty for current to flow through a material Ohms The width of a pipe Ohm s Law I frac V R Kirchhoff s voltage law The sum of the voltages around a circuit is zero. Binary Analogy v.s. Digital Decimal v.s. Binary v.s. Hexadecimal Binary data Text, image, etc. 8 Bits b 1 Bytes B 1 KB 1024 Bytes 2 10 Byte 1 MB 1024 2 Bytes 2 20 Bytes 1 GB 1024 3 Bytes 2 30 Bytes 1 ...

Straight line Leadership Dusan Djukich Sam Matla described this book as the one that changed his life in this youtube video. After reading it through, I had to say that this book does have a transformative power in it. The distinctions delivered in this book forces me to face the reality and, definitively, helped me jump out of the vicious bubble I created for myself to keep procrastinate. Indeed, usually it s ourselves who stumble our productivity and refuse to see the straight line to our goal, because facing the straight real solution always expose ourselves to risks and uncertainty, which is very uncomfortable. We may even invent things to do to insulate us from it Waiting for the best time to do something Endless information collection without taking any action Trying to find the best productivity note taking app that solve all of your problem once and for all ... You nam...

NeurIPS 2024 oral notes P.S. The paper descriptions are based on my personal understanding. Some text were extracted from the abstracts and reviews. Domain LLM ZhenghaoLin2024NeurIPS scores training tokens using a reference model and then training the language model with a focused loss on tokens with higher scores selectively, proving not all tokens are what you need Quantization HaokunLin2024NeurIPS utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers when quantizing LLMs VladimirMalinovskii2024NeurIPS proposes a quantization aware strategies for fine tuning the LLMs after quantization, improving its performance especially in extreme compression Alignment JiamingJi2024NeurIPS proposes Aligner, a small model that l...

The Road to Serfdom Friedrich Hayek Nor am I arguing that these developments are inevitable. If they were, there would be no point in writing this. Friedrich Hayek The ideas of economists and political philosophers, both when they are right and when they are wrong, are more powerful than commonly understood... Madmen in authority, who hear voices in the air, are distilling their frenzy from some academic scribbler of a few years back. John Maynard Keynes P.S. Part name was added by me. Introduction The supreme tragedy is still not seen that in Germany it was largely people of good will, men who were admired and held up as models in the democratic countries, who prepared the way for, if they did not actually create, the forces which now stand for everything they detest. Mere hatred of everything German instead of the particular ideas which now dominate the Germans is, moreover, very dangerous, because it blinds those who indulge in it against a real threat....

Mastering Your PhD Patricia Gosling Bart Noordam P.S. Part name was added by me. Some chapters are reordered. Part I Getting Started Introduction As a PhD student, it s important to understand the larger scene of the academic system and look from your supervisor s perspective. The author first introduce 3 typical stages of professorship, corresponding to 3 types of research teams The start up group Assistant Professor Your supervisor is in his her junior years, whose career depends on your outcome. You will have frequent intense interactions will your supervisor. The up and running group Associate Professor Your supervisor is in his her mid career, who has a lot of responsibilities and less time for you. Your fellow graduate students are your main support. The mature group Full Professor The small but established group The large and established group the empire Getting Started At the beginning of your PhD Become familiar ...

Computing and Processing Correspondences with Functional Maps Maks Ovsjanikov et. al Computing and processing correspondences with functional maps ACM SIGGRAPH 2017 Courses Part I Functional Maps Basics P.S. Part name was added by me. What are Functional Maps Shape correspondence can be represented as the transportation T F of functions between shapes. If such functions are represented with bases phi i on different shapes, the formulation becomes T F f T F sum i a i phi i M sum i a i T F phi i M Here the core is T F phi i M sum j c ji phi j N T F f sum j sum i a i c ji phi j N Noted that here the matrix C or c ji can fully encode the shape correspondence, in the form of transportation of basis functions between shapes. If only consider the spectral embeddings a i T F a Ca If C is correct, the point to point map can be...

Spectral Mesh Processing Bruno L vy Hao Zhang Spectral mesh processing ACM SIGGRAPH 2010 Courses A gentle introduction Setting the stage Laplacian, Laplacian smoothing and its relationship with midpoint smoothing Spectral transform Use eigenvectors of the laplacian as the basis to represent coordinates functions x i , y i . From linear algebra, we know that since L is symmetric, it has real eigenvalues and a set of real and orthogonal set of eigenvectors which form a basis. Any vector of size n can be expressed as a linear sum of these basis vectors. Signal reconstruction and compression with this basis As well as filtering and smoothing Connection with discrete Fourier transform The connection we seek, between DFT and spectral analysis with respect to the Laplace operator, is that the DFT basis functions, the g k s, form a set of eigenvectors of the 1D discrete Laplac...

Deep Learning Tuning Playbook Google Research GitHub google research tuning playbook A playbook for systematically maximizing the performance of deep learning models. Guide for starting a new project Guides on choosing the model architecture, the optimizer, and the batch size. The goal of tuning batch size is to saturate the GPUs, which can be monitored by training throughput examples processed per second time per step batch size training throughput P.S. When batch size is tuned, most hyper parameters need to be tuned. Among them, the learning rate and the regularization term are the most important. A scientific approach to improving model performance Design the experiment Scientific hyper parameters The experiment is aimed at explore the effect of the scientific hyper parameters. Nuisance hyper parameters To compare different approach, nu...

Deep Work Cal Newport Part I The Idea Deep Work Is Valuable 3 types who thrived in the new economy high skilled workers super stars owners. How to thrive in the new economy quickly master hard things produce at elite level. Deep work deliberate practice intense concentration feedbacks towards productivity. Reasoning myelin attention residue law of productivity. Exception Jack Dorsey s case. Deep Work Is Rare Why hard works is rare Deep work is hard and shallow work is easier. The metric black hole hard to measure different knowledge work behavior s actual impact on the bottom line outcome The Internet technopoly culture that believes whatever the IT revolution provides are revolutionarily good. Deep Work Is Meaningful Neurologically, as Winifred Gallagher discovered, our brain construct our worldview based on what we pay attention to. Deep work requires concentration and thus it both focuses your worldview to you...

Clean Code Handbook LeetCode 50 Common Interview Questions LeetCode Data structure Array string Transform simplify combinatorial iterations Representation 13 Longest Palindromic Substring Instead of representing the substring with starting and ending indices, represent it as center index and length is much more efficient, improving runtime from O n 3 to O n 2 . Proxy items 2 Two Sums II Input Array Is Sorted If the array is sorted, then a i a j leq a i a j leq a i a j . Thus a i a j can be used as a proxy of a bunch of other items in the iterations, improving the runtime to O n . Iteration folding 1 Two Sums is commutative, thus hal...

Computer Vision Algorithms and Applications Richard Szelisky Free official electronic version Computer Vision Algorithms and Applications, 2nd ed. P.S. Part name was added by me. Introduction Sample Editor Command Very difficult due to imperfect information. A brief history of computer vision. Part I The Basics Image formation Sample Editor Command Homogeneous coordinates and how to use it to describe 2D 3D transformation compactly. Photometric image formation Models of lighting, reflectance, shading, and optics. Digital camera Sampling and aliasing when digitizing image. Demosaicing color filter arrays and describe colors in different color space. Gamma correction and white balance. Image compression technique used by JPEG and other image formats. Image processing Operations Point level operators Convolutional operators linear filtering Neig...

So Good They Can t Ignore You Cal Newport So Good They Can t Ignore You Cal Newport Rule 1 Don t Follow Your Passion x The Passion of Steve Jobs x Passion Is Rare x Passion Is Dangerous Rule 2 Be So Good They Can t Ignore You x The Clarity of the Craftsman x The Power of Career Capital x The Career Capitalists x Becoming a Craftsman 5 steps to start becoming a craftsman who accumulates career capital with deliberate practice Decide what capital market you re in . I m in academia. To win good positions, the only capital is good publications . Therefore, this is a winner take all marker. Identify your capital type . Good publication is the major capital in the academic market. To gain this kind of capital, critical skills to master include grand vision of the field e.g. keeping tracking of up to date papers , ability to hunt down important impactful manageable problems,...

How to Become A Straight A Student Cal Newport How to Become a Straight A Student Cal Newport Part I Study Basics Step 1 Manage Your Time in Five Minutes a Day See Cal Newport s time management system Two column todo list. Step 2 Declare War on Procrastination Step 3 Choose When, Where, and How Long Part II Quizzes and Exams Step 1 Take Smart Notes Step 2 Demote Your Assignments Step 3 Marshal Your Resources Step 4 Conquer the Material Step 5 Invest in Academic Disaster Insurance Step 6 Provide A Answers The Plan in Action Part III Essays and Papers In part 3, Cal devises a detail scheme to approach thesis essay assignments. I got 3 takeaways that not only could help my academic writing but may also be applicable to research works Split the mentally demanding tasks into steps. Intertwining thesis ...

The Time Block Planner Cal Newport The Time Block Planner Cal Newport Cal Newport s time management system is summarized in this book. Combining the techniques discussed in this tiny pamphlet and his other books podcasts, I compiled this blog post Cal Newport s time management system.